This is the first of the foundation retrospectives I promised in the catch-up post. We’re going back to December, before a single server existed, when the first real decision of the project wasn’t about compute or DNS. It was: where do the secrets live?

Every tutorial answers this with “never commit secrets to git”, which is correct and useless. The passwords have to live somewhere. A password manager means copy-pasting values into servers by hand, and hand-managed state drifts. HashiCorp Vault means running a secrets server before I have any servers. Environment files outside the repo break the one promise I made myself when starting this: if I lose every server tomorrow, I rebuild from the git repository and nothing else. For that promise to hold, the secrets file has to be in the repository.

SOPS

SOPS squares the circle. It encrypts YAML at the value level: the keys stay readable, the values don’t.

stalwart_admin_password: ENC[AES256_GCM,data:kX9f...,tag:Rq2w...,type:str]
luks_passphrase: ENC[AES256_GCM,data:8jNM...,tag:pL0v...,type:str]

This structure matters more than it looks. A git diff on the encrypted file shows exactly which secret changed and when, without revealing anything. Reviewing “the LUKS passphrase was rotated on this date” is possible; reading the passphrase is not. The whole file, currently over a hundred credentials for databases, DKIM keys, API tokens, and monitoring push URLs, is one production.enc.yaml committed like any other file.

Under the hood SOPS encrypts the data with AES and then wraps that data key for each configured recipient. Recipients, in my setup, are two GPG keys.

Two keys, two jobs

The .sops.yaml at the repository root pins the rules:

creation_rules:
  - path_regex: secrets/.*\.enc\.yaml$
    pgp: >-
      96C52216505164158B0FC2A589A9B2657094C379,
      7387AF3C3410F87EA63DF73317FF21C26659B4D1

The first fingerprint is the human key: passphrase-protected, lives on my workstation, used when I run sops secrets/production.enc.yaml to edit values in my editor. The second is the automation key: no passphrase, used by Ansible and Terraform wrappers that need to decrypt at run time without me typing anything. Both keys can decrypt every secret, so losing my workstation doesn’t lock out automation and vice versa.

Those two private keys are the crown jewels of the entire platform. The human key has never touched a server. The automation key travels to machines that need it inside a symmetrically encrypted wrapper, gets imported, and the transfer file gets shredded. If both keys vanish, the repository full of ENC[...] blobs is confetti. That’s the deal you accept.

The Ansible side

Playbooks decrypt through the community.sops lookup plugin:

luks_secrets: "{{ lookup('community.sops.sops',
                  playbook_dir + '/../secrets/production.enc.yaml') }}"

That playbook_dir is there because of the one real fight this setup gave me. With a relative path, the lookup resolves differently depending on whether it runs from a playbook or from inside a role, and when it goes wrong it goes wrong quietly: no error, just a lookup pointed at a file that isn’t there. I spent an evening staring at a playbook that behaved as if the secrets file were empty. Absolute paths anchored to playbook_dir ended it, and the pattern is now law in this repository.

What I’d tell December me

Two things weren’t obvious at the start. First, git never forgets: every secret ever committed is in the history permanently, encrypted but present. So the response to any doubt is rotation, not deletion. Changing the value is easy; pretending the old ciphertext was never there is impossible.

Second, this setup gets more valuable with every service, not less. Eight months and about twenty services later, onboarding a new application is still “add three lines to production.enc.yaml, reference them in the role”. No secrets sprawl, no “which .env file is current”. The most boring part of the platform, exactly as it should be.

Next week: encrypting the disks those secrets get deployed to, and why unlocking them still requires a human.