Skip to content
David Cumps
The Repo Now Lives on the Server It Builds

The Repo Now Lives on the Server It Builds

Every service in this series so far was for using the platform. This one changed what the platform is. Forgejo, the community fork of Gitea, went live on apps1 in the spring: git hosting with a web UI, SSH access, OIDC login through Authelia, and eventually its own CI runner. Then came the move that still makes me grin: the selfhost repository, the Terraform and Ansible that define this entire platform, migrated off GitHub and onto the Forgejo that the repository itself had provisioned.

A git push --mirror carried every branch and tag across, reference parity got verified rather than assumed, and the README’s clone URL flipped to the new home. From that day, every change to the infrastructure is committed to infrastructure it manages. The project’s retrospective called it the strongest possible end-to-end test: the platform can host itself. My own note from that week was less formal: it was the first time this felt like a system running itself, instead of a system being run.

CI on the RAM that was left

Self-hosted git invites self-hosted CI, so a Forgejo Actions runner followed, with Docker-in-Docker for job isolation. The story draft cheerfully assumed 4 GB of headroom; free -h on apps1 reported 598 MB actually available. Instead of resizing the instance, the runner got budgeted like it was 2005: 256 MB for the runner, 384 MB for DinD, with the risk that a heavy job hits the cgroup ceiling written down explicitly instead of wished away. The first workflow ran green in 30 seconds. There’s a quiet lesson in that pairing: capacity assumptions belong in the same review as everything else, because drafts inherit defaults from a world with more RAM than yours.

Getting the runner registered surfaced a genre of bug this project keeps meeting: documentation describing software that doesn’t quite exist. The CLI subcommand the docs named was wrong, per-repository Actions turned out to be off even with the global flag on (flippable only through the API, no UI in that version), and an admin endpoint the verification plan relied on wasn’t in the API at all. Three acceptance criteria got amended mid-story, each amendment logged. Upstream reality outranks upstream documentation, and a story process that can absorb that without either lying or collapsing is worth the ceremony it costs.

Upgrading the thing that hosts the thing

The season finale was a major-version upgrade, v14 to v15, of the forge holding the project’s source of truth. That circularity concentrates the mind: if the upgrade eats the data, the repository describing how to rebuild everything is inside the thing that just ate it (GitHub’s copy notwithstanding).

The gate I set: no upgrade until the backup proved itself, and the backup tool’s own integrity check doesn’t count. The pre-upgrade snapshot got restored to scratch and diffed bit-for-bit against the live tree, and the database dump got round-tripped through an actual restore. The diff came back empty; only then did the version bump run. Thirteen database migrations completed in about a second, five more documentation-versus-reality drifts got reconciled in flight, and a written rollback procedure sat ready and unused. That restore-to-scratch-plus-diff gate is now my standing rule for any major upgrade, and I’d recommend it to anyone whose backups have only ever been checked, never exercised.

The upgrade also delivered the sharpest infrastructure lesson of the epic, when the freshly recreated Tailscale sidecar left Forgejo bound to a network namespace that no longer existed. The known workaround, a container restart, turned out to be structurally useless: Docker’s restart API does not re-resolve network_mode to the new sidecar. It rebinds you to the corpse. That discovery rippled into a fleet-wide fix across every sidecar-consuming role, and the full story belongs to the sidecar saga post in a few weeks. The takeaway that generalizes: we had verified the workaround was present in all the roles, not that it worked. Verifying the goal instead of the task is the whole job, and it took a namespace inode to teach me properly.

Next week: what happens when the software you’re deploying is younger than your deployment process.

You May Also Like