The threat model for a rented server is simple to state: my data sits on disks I will never see, in a data center I will never visit, handled by people I will never meet. Disks get decommissioned, snapshots get copied, hardware gets recycled. I can’t control any of that. What I can control is whether the bytes on those disks mean anything without a passphrase. Last week covered the secrets that never leave the git repository; this week is about the disks those secrets unlock.

The layout

Every node in the fleet gets two volumes. The OS volume is unencrypted and boring: it boots, runs sshd, and holds nothing I’d miss. The data volume is LUKS2, opened as data_crypt and mounted at /mnt/data. Everything of value lives under that mount point, including Docker’s entire data root, so a container can’t quietly write state to an unencrypted disk even if I misconfigure it.

The part that took actual thought is what happens when the volume is not unlocked. Docker’s systemd unit carries Requires=mnt-data.mount, which means no unlocked volume, no Docker, no services. Without that dependency, Docker would happily start against an empty /mnt/data directory and every container would initialize fresh state on the wrong disk. Services refusing to start is annoying; services starting wrong is dangerous.

Manual unlock is a feature

There are ways to unlock LUKS automatically: a key file fetched at boot, a network-bound scheme like tang/clevis, a passphrase baked into cloud-init. I skipped all of them. Automatic unlock means the passphrase, in some form, is reachable from the machine, and the machine is exactly what I don’t trust.

So a reboot requires me. The procedure is deliberately short: decrypt the passphrase with SOPS on my workstation, SSH in, run /usr/local/bin/unlock-luks.sh, type the passphrase, watch Docker come up. Two minutes per host.

The side effect turned out to be the actual feature: I notice every reboot. There is no scenario where a host silently bounced overnight and I find out weeks later. On a five-node personal fleet, reboots are rare enough that the ritual costs me maybe ten minutes a year. The honest downside: if a host reboots while I’m on a beach without a laptop, it stays down until I’m back. I’ve accepted that trade. Email, the one service that can’t wait, runs on two nodes in two data centers precisely so one locked volume doesn’t take it down.

The header is the real key

A detail that’s easy to learn too late: the LUKS header at the start of the volume holds the key slots. If those few megabytes get corrupted, the passphrase becomes worthless and the volume is gone, passphrase or not. So every host’s header is backed up twice. Once GPG-encrypted inside the repository (secrets/luks-header-mx1.img.gpg and friends, same commit-the-crown-jewels logic as SOPS), and once on the host itself in /root/luks-backup. Restore is one cryptsetup luksHeaderRestore away, and the exact commands live in a runbook in the repo, written while calm for reading while not.

One small build-time gotcha for anyone scripting this: lsblk prints multiple lines for a device once its LUKS mapping is open, which broke my idempotency checks until I found the -d flag. The Ansible role that manages all of this has been reapplied dozens of times since December without incident.

The landmine I didn’t know about

Full honesty, as promised: this design shipped with a bug that stayed invisible for seven months. The fstab entry for /mnt/data lacked the noauto flag, which doesn’t matter at all until a host actually reboots, tries to mount a volume that’s still locked, fails, and drops into systemd emergency mode instead of booting cleanly and waiting for me. Every host in the fleet carried that trap. It finally fired in July during a planned resize, and the fix is now part of the role. The full story belongs to a later post about that month, but consider this the disclosure: the manual-unlock design was right, and my first implementation of it was wrong in a way no amount of code review caught. Only a real reboot did.

For email there’s one more layer on top of LUKS: Stalwart encrypts mail content at rest with per-user OpenPGP keys, so even a database-level compromise reads ciphertext. Which brings us to next week: deploying the mail server itself.