ADR-057: Codify the Prometheus scrape config in git, drift-check with a reconcile script

Status: Accepted Date: 2026-04-22

Context

On 2026-04-22 the fleet observability rollout added 17 per-product scrape jobs to Prometheus one at a time, as each service was wired through gyrum-observ-integrate (ADR-045). Every addition was a hand-edit to /deploy/observability/prometheus/prometheus.yml on dark-factory-ops, followed by a POST /-/reload. The end state of that file — 7 infra jobs plus control-plane plus 17 service jobs, 24 total — was never checked in.

Two things became true at the same time:

The git-tracked scrape template at infrastructure/observability/prometheus/prometheus.yml lists only distill and the infra jobs. A fresh ops bootstrap run would have deployed that file and silently dropped 17 scrape targets.
A second, stale template at infrastructure/server-bootstrap/config/prometheus.yml was what stages/08-observability-stack.sh actually copied onto the VPS. It listed only three jobs (self, node-exporter, cadvisor) and pointed at the wrong container names (host.docker.internal instead of the private-net IPs used on live). Neither tracked file matched reality.

The 17 jobs were added during a single evening under time pressure — the live file is the only place they exist, and losing dark-factory-ops (or re-running stage 08 on it) would silently lose them. That is the class of mistake that justifies a decision record: the system has entered a state where its production config only exists in one place that can be destroyed, and nobody would know from reading the repo.

Related in flight:

ADR-045 — defines the CLI that adds a service's /metrics scrape target; wiring to Prometheus was left as "operator step" in that ADR.
ADR-050 — the bearer-auth story for /metrics. Fixed the credentials_file pattern we codify here; also flagged that dark-factory-control-plane still uses an inline bearer token as a legacy migration case.
ADR-052, ADR-053 — each of these added scrape-config requirements on the operator side that were absorbed ad-hoc into the live file; we are consolidating their tail-of-rollout work here.
docs/observability-stack-topology.md and docs/observability-runbook.md already describe the live layout (ops VPS at 178.104.63.31, scrape config at /deploy/observability/prometheus/prometheus.yml). Those docs were already accurate; the code lagged.

Decision

infrastructure/observability/prometheus/prometheus.yml is the canonical source of truth for the ops-VPS scrape config. It lists all 25 jobs that run on live today (7 infra + control-plane + 17 per-product) and is the file that stages/08-observability-stack.sh syncs to the VPS. Every new scrape target lands via a PR against this file. The stale infrastructure/server-bootstrap/config/prometheus.yml is deleted.

Bootstrap stage 08 is updated to sync the canonical template into /deploy/observability/prometheus/prometheus.yml (the subdir path live already uses), plus the matching alert-rules.yml. A back-compat symlink keeps the flat-path mount in the bootstrap compose working so this PR does not have to rationalise the bootstrap-vs-live compose drift (that is a separate ADR-055/ADR-056 work item, and this PR explicitly stays in its lane).

No secrets go in the template. The only auth on a scrape job is via authorization.credentials_file: /etc/prometheus/<slug>-bearer.token, following ADR-050. The inline bearer that today sits in the live config for dark-factory-control-plane is replaced in the template with a credentials_file path; reconciling that onto the live box requires the operator to drop the token file first, then copy the new config (see Deploy sequencing below).

A new tool, scripts/reconcile-prometheus-live.sh, pulls the live prometheus.yml over SSH and diffs it against the canonical template. It reports drift on job names, targets, labels, and auth shape (inline creds vs credentials_file) and exits non-zero if anything differs. It intentionally does not compare token values — those are not in git. It is a run-once operator tool, not a cron job, and is explicitly not allowed to push to live; reconciliation is a supervised step.

Deploy sequencing for this PR

This PR lands git state only. Reconciliation onto the live VPS is a separate supervised step. The sequence:

Merge this PR. Live is unchanged; a fresh ops bootstrap would now deploy all 25 jobs via stage 08, so the "bootstrap drops jobs" risk is closed as of this merge.
Run scripts/reconcile-prometheus-live.sh. Expected output today: one drift — dark-factory-control-plane has credentials_INLINE on live vs credentials_file:/etc/prometheus/control-plane-bearer.token in the template. This is expected and documented.
Drop the control-plane bearer token on dark-factory-ops at /deploy/observability/prometheus/control-plane-bearer.token (mode 0400, chown 65534:65534 — the prom/prometheus container's nobody uid, the gotcha documented in ADR-050). Copy the token value from the current live prometheus.yml credentials: field.

Copy the template onto live and reload Prometheus:

scp infrastructure/observability/prometheus/prometheus.yml \
    deploy@178.104.63.31:/tmp/prometheus.yml
ssh deploy@178.104.63.31 \
    "sudo cp /tmp/prometheus.yml /deploy/observability/prometheus/prometheus.yml \
     && curl -sf -X POST http://10.0.0.4:9090/-/reload"

Re-run the reconcile script. Exit code 0 ⇒ live and git are in lockstep. From this point onward every new scrape target must land in a PR first.

Consequences

Good:

A fresh ops bootstrap is now recoverable. Losing dark-factory-ops or re-running stage 08 repaves the full 25-job scrape set instead of silently dropping 17 of them.
Every future scrape target requires a PR touching infrastructure/observability/prometheus/prometheus.yml — the diff is visible to reviewers, the history is in git blame, and the new job is covered by the same review gates as any other PR.
scripts/reconcile-prometheus-live.sh makes drift a one-command check. The next operator who changes live before PRing is caught on the next scheduled reconcile run (even without automation — any operator who reads the runbook hits it).
The inline-creds pattern that survived on dark-factory-control-plane is flagged as drift and pushed toward the ADR-050 compliant credentials_file pattern with an explicit deploy sequence.
Cross-ADR coupling is now explicit: every ADR that added a scrape target (ADR-045 integrate CLI, ADR-050 bearer-auth, ADR-052 operate proxy, ADR-053 frontend-observability) is referenced from one place.

Trade-offs:

Adding a scrape target is now a PR, not an ssh-and-nano. For a team of one doing a late-night fleet rollout that is mild friction. For a team of more than one, or for the same operator at a later date, it is a strict improvement in recoverability.
The back-compat symlink from /deploy/observability/prometheus.yml to /deploy/observability/prometheus/prometheus.yml keeps the bootstrap compose happy without rationalising the full bootstrap-vs-live compose drift. Someone (ADR-055 Grafana / ADR-056 Caddy parallel work, or a follow-up) will need to reconcile the container names, image pins, and network_mode. This ADR does not fix that; it just doesn't make it worse.
socialproof.gyrum.io serves /metrics (responds 200 to a HEAD as of this ADR date) but is not in the scrape list yet, matching live. Adding it is a separate PR that follows the ADR-050 deploy sequence (drop token first, then add scrape job). Flagged in the PR body so it is not forgotten.

What we have signed up to operate:

scripts/reconcile-prometheus-live.sh in the operator runbook: run it after any hand-edit or as a periodic drift check. Add to the observability-runbook "after a scrape-config change" checklist.
The control-plane-bearer.token file on dark-factory-ops (the migration step that closes the last inline-creds drift).
The ADR-050 gotcha — chown 65534:65534 on all /deploy/observability/prometheus/*-bearer.token files — remains the operator's responsibility. Bootstrap stage 08 creates the prometheus subdir but does not populate tokens (they are secrets).

Alternatives considered

Leave live as source of truth. The status quo. Rejected because it is not recoverable — a dark-factory-ops rebuild silently drops every scrape target added since the git template was last touched. The whole reason dark-factory has a bootstrap pipeline is that we can rebuild any VPS from scratch; the scrape config is the single piece of observability state that falls out of that guarantee if we don't codify it.
Dynamic service discovery via the Hetzner API. Prometheus has first-class HTTP SD and Hetzner SD support; a small discovery service could query the Hetzner project and emit scrape targets for every running VPS. Rejected for scope: we have two VPSs and no plan for more in the next two quarters, per docs/platform-gap-audit.md. The per-product dimension is the interesting one, and products are services running inside a VPS, not VPSs themselves — Hetzner SD would not help with the distill / buzzy / guardian / etc. layer, which is exactly the 17 jobs that prompted this ADR.
Consul / etcd / Nomad service registry. A registry-backed approach with Prometheus's Consul SD. Rejected as overkill for a <30-service fleet on two VPSs. Consul would bring a clustering problem to solve a yaml-editing problem. Revisit if fleet size reaches ~50 services or we go multi-region.
A gyrum-observ-integrate subcommand that writes a git commit. Tempting because it puts the add-scrape-target step inside the same tool that already scaffolds the product side. Rejected for now: ADR-045's CLI runs in the product's repo and would need cross-repo PR machinery to mutate dark-factory. The current PR-in-dark-factory flow is slow but legible. Revisit if the friction becomes a real cost — e.g. >1 scrape addition per week for a month.
Template the per-product jobs via a small generator (jsonnet / cue / envsubst). Rejected because the jobs are nearly identical and YAML repetition is not the pain point. The pain point is "are these jobs checked in at all"; once they are, repetition in the file is ~10 lines per product and legible. A generator would add a build step without closing the recoverability gap any more than a flat file does.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

ADR-045 — the CLI that creates a product's observ wiring; this ADR codifies the scrape-target side of its rollout.
ADR-050 — the credentials_file pattern this ADR standardises in the template and enforces via the reconcile script's auth-shape check.
ADR-052 — operate-mode proxy wiring that produced several of the services now in the scrape config.
ADR-053 — frontend-observability rollout; its backend-side services are among the 17 codified here.
docs/observability-runbook.md — update pending to reference the reconcile script in the post-edit checklist.
scripts/reconcile-prometheus-live.sh — the drift reporter.