048 — Codify ops-VPS Caddy site configs in server-bootstrap

Date: 2026-04-22 Status: Accepted Supersedes: Part of #267 (which is now closed as resolved by this ADR)

Context

The ops VPS (dark-factory-ops) runs Caddy as the public ingress for the four observability surfaces: grafana.gyrum.ai, alertmanager.gyrum.ai, ntfy.gyrum.ai, loki.gyrum.ai. Each is a separate site config in /deploy/caddy/sites/, imported by /deploy/caddy/Caddyfile.

Three problems came to light during the 2026-04-22 end-to-end wiring:

The site configs only existed on the VPS. No copy in git. A fresh ops VPS bootstrap would have nothing to import — Caddy would serve 404 for every route. The only "documentation" was a knowledge-base page documenting what should be there.
The Grafana IP allow-list was stale. The site file restricted access to 86.12.60.129 — an old operator IP. The live operator IP was 81.105.99.155. Nobody noticed because earlier reload attempts silently failed (see #3) and Caddy was serving an in-memory config from before the file was last edited.
A stray .bak file in sites/ broke caddy reload. The Caddyfile imports sites/* — that glob matched grafana-gyrum-ai.bak.<ts> too, producing "ambiguous site definition: grafana.gyrum.ai" and aborting the reload. The OLD config kept serving from memory, so the failure was invisible until the next full restart.

The compounding effect: the ops VPS was running an undocumented config from memory, the on-disk files were stale, and any attempt to fix things with systemctl reload caddy silently no-op'd. The first hint anything was wrong was when the operator's IP rotated and they got 403.

Decision

Codify all four ops-VPS Caddy site configs as templates in dark-factory:

infrastructure/server-bootstrap/config/caddy-sites/
├── README.md
├── grafana-gyrum-ai.tmpl       # __OWNER_IP__ + __APP_VPS_IP__ allow-list
├── alertmanager-gyrum-ai.tmpl  # currently no auth — issue #268
├── ntfy-gyrum-ai.tmpl          # currently no auth — issue #268
└── loki-gyrum-ai.tmpl          # __LOKI_BEARER_TOKEN__ header gate

A new bootstrap stage 12-ops-vps-caddy-sites.sh (role-gated to SERVER_ROLE=ops) installs them with sed-based token substitution from vars/<host>.env, and — critically — archives any file in sites/ not declared as a template to /deploy/caddy/backups/ (outside the import glob). That kills the .bak reload-trap class of bug structurally.

A separate single-purpose script scripts/rotate-grafana-allowed-ip.sh handles the most common operational change: operator IP rotation. It auto-detects the new IP, updates the live Caddy config, reloads, verifies the new IP can reach Grafana, and auto-rollbacks if not.

Consequences

Good:

Fresh ops VPS bootstrap now produces a complete, functional Caddy config — no manual recovery from memory.
IP rotation is a one-command operation with built-in safety (auto-rollback if the new IP can't reach Grafana).
Stage 12 enforces "the import glob only contains declared files" — the stale-.bak-in-glob class of bug can't recur.
Site configs are reviewable in git diff.
Bearer tokens stay out of git (templated from per-host vars).

Trade-offs:

Adds a sixth stage to the ops bootstrap. Not significant — it's idempotent and runs in <2s.
Changing a site config now means a PR + bootstrap re-run, not a quick SSH edit. We accept this because the SSH-edit path is exactly what got us into this state. The rotation script preserves the fast path for the one case (IP rotation) that actually justifies it.

What this does NOT solve:

Issue #268 — alertmanager + ntfy still publicly exposed without auth. This ADR codifies the current state, including the no-auth templates. A separate ADR + change is needed to add auth.
Issue #266 — loki.gyrum.ai DNS still points at a decommissioned IP (116.203.167.0). The Caddy template is correct; the DNS record needs updating in Cloudflare. Out of scope here.
/etc/caddy/Caddyfile vs /deploy/caddy/Caddyfile confusion — the systemd drop-in unit overrides ExecStart to use the latter. We don't change this; we document it loudly in the topology doc and the KB so the next person doesn't waste an hour editing the wrong file like I did.

Alternatives considered

Manage Caddy config via the admin API only. Live-only, no on-disk file. Rejected: opaque to git, no review trail, dies on Caddy restart unless we save+restore the JSON each time. The existing approach (Caddyfile + sites/) is more debuggable.
One big Caddyfile, no import glob. Avoids the .bak-in-glob trap. Rejected: per-site reloads become per-file diffs in a monolithic file (worse review surface), and adding a new site requires editing the central file (more conflict-prone for multi-agent or multi-operator changes).
Move all the IP allow-listing into Cloudflare Access (the medium- term option I floated to the user when discussing MFA). Rejected for THIS ADR because it's a larger change requiring Cloudflare Zero Trust setup and replaces the remote_ip matcher entirely. Tracked separately as a follow-up; this ADR makes the current model correct and reproducible while we evaluate Cloudflare Access.
Pull the existing site files into a caddy/ repo of their own. Rejected: tight coupling to bootstrap stages keeps the install path one-step (bootstrap.sh --only 12); a separate repo would need its own bootstrap-discovery wiring. The dark-factory infra already owns the ops VPS configuration story.

Verification

After applying this ADR's changes:

# Stage 12 installs all four site files cleanly:
SERVER_ROLE=ops sudo bash infrastructure/server-bootstrap/stages/12-ops-vps-caddy-sites.sh

# All four routes serve as expected from the operator's allow-listed IP:
curl -s -o /dev/null -w "grafana: %{http_code}\n" https://grafana.gyrum.ai/         # 302 (login redirect)
curl -s -o /dev/null -w "alertmanager: %{http_code}\n" https://alertmanager.gyrum.ai/  # 200
curl -s -o /dev/null -w "ntfy: %{http_code}\n" https://ntfy.gyrum.ai/                  # 200

# Rotation flow works end-to-end (with --dry-run):
bash scripts/rotate-grafana-allowed-ip.sh --dry-run