ADR-055: Pin every observability-stack container to an exact major.minor.patch

Status: Accepted Date: 2026-04-22

Context

On 2026-04-22 a security-follow-up probe of the observability stack on dark-factory-ops (178.104.63.31) found the live and the bootstrap compose files in three different states of drift. The stack is the one declared in ADR-031 (self-hosted observability on the app VPS) and operationalised by stages/08-observability-stack.sh during a fresh bootstrap. Four of the containers that sit behind grafana.gyrum.ai, alertmanager.gyrum.ai, and the internal scrape mesh (ADR-049, ADR-050) were running on the :latest tag:

In parallel, the git-tracked bootstrap template at infrastructure/server-bootstrap/compose/observability-stack.yml declared an older, pinned world: grafana/grafana:11.1.0, prom/prometheus:v2.53.0, grafana/loki:2.9.6, and no alertmanager service at all — even though Alertmanager had been live since ADR-049 in March. A fresh ops bootstrap run would therefore have come up with a Grafana major-version behind the running host, a Prometheus one major behind, a Loki one major behind, and no Alertmanager. That is the class of drift where "what the repo says it deploys" and "what actually runs" have walked 11 months apart while nobody was watching.

:latest is the root cause. An unscheduled docker pull && docker compose up (or a host reboot after a registry-side tag move) will silently replace any of these four with a new major version, with no change tracked in git, no ADR, and no operator signal other than "Grafana looks different". Observability is the system we use to see other systems break; it is the one we can least afford to have break invisibly.

Two open PRs already addressed the Grafana slice only — #286 (pin Grafana bootstrap to 12.4.3 + an earlier ADR-054 slot that collided with the ai-frontend three-mode-routes decision) and #287 (pre-change live-upgrade ops report). Both are superseded by this ADR, which broadens the pin + live-coordination to all four containers at once so the drift story lives in one place.

Decision

Every container in the observability stack compose template (infrastructure/server-bootstrap/compose/observability-stack.yml) declares an exact major.minor.patch image tag. No :latest, no :X.Y, no registry-managed aliases. A version bump is a git change that:

1. Updates the image tag in the compose template.
2. Amends or adds a short record under docs/operations/ naming the pre/post versions, snapshot paths, and per-container rollback commands.
3. Links to this ADR.

The initial pins applied on 2026-04-22 are:

The alertmanager service is also newly declared in the bootstrap compose so that a fresh ops bootstrap produces the same shape as the live stack.

Container names in the bootstrap compose keep the gyrum-* prefix (gyrum-grafana, gyrum-prometheus, gyrum-loki, gyrum-alertmanager). The live /deploy/observability/docker-compose.yml on dark-factory-ops uses bare names (grafana, prometheus, …) because it predates the prefix convention. The two files are not the same file and do not need to agree; the bootstrap template only matters the next time a fresh ops VPS is stood up, at which point the gyrum-* prefix matches the rest of the stack (ADR-049 already cites gyrum-alertmanager).

Consequences

Easier. A reviewer reading the compose file knows exactly what will be pulled. docker-compose pull is deterministic. A registry-side tag move cannot silently roll a major version onto the ops VPS. The bootstrap-vs-live drift audit becomes a git diff rather than a forensic exercise. Grafana 12.x dashboards keep rendering (no reverse travel to 11.x); the existing panel schemas, folder structure, and pinned datasource UIDs from ADR-047 continue to work.

Harder. A bump now requires a deliberate PR + ops-report pair, instead of a quiet docker pull. Security-patch uptake has an operator-in-the-loop cost. This is the tradeoff we are accepting: we trade speed-of-patch for knowing-what-runs. The observability stack is low-throughput on CVE bumps (roughly one per quarter per component in the last year), so the ceremony cost is bounded.

Signed up to. Rollback discipline: every live bump writes a snapshot of docker inspect <name> --format '{{json .Config}}' and a volume tar.gz into /deploy/backups/ before the pull, and records the rollback one-liner in the ops doc. Quarterly review of pins against upstream release notes — we will notice them because the bump is an explicit task, not an implicit background event.

Per-container rollback plan. Recorded under docs/operations/obs-stack-pin-2026-04-22.md. Each container's rollback is the same shape: docker compose pull <svc> && docker compose up -d <svc> with the tag reverted to the pre-pin column of the table above, backed by the named-volume snapshot if data-layer corruption is suspected.

Alternatives considered

• Pin only Grafana (as in PR #286 / #287). Fixes the most-visible drift but leaves three other :latest tags dragging along. Future docker pull on the ops host can still surprise-promote Prometheus or Loki to a new major; the policy is weaker than the operator-visibility goal.
• Pin to the minor (grafana/grafana:12.4). Slightly cheaper to maintain (patch bumps happen without a PR) but the thing we are defending against — a registry-side tag move rolling a new binary onto the host without a git change — still happens, just scoped to patch releases. Loses the "one compose diff per upgrade" property.
• Pin by digest (@sha256:…). Strongest form. Rejected here because the ergonomics are poor for a human-managed compose file (tags aren't legible), the bootstrap template is read by operators more often than by machines, and we don't yet have a Renovate/Dependabot flow for digest bumps. Revisit when that flow exists.
• Roll Grafana back to 11.6.x. What the original security-follow-up brief asked for, assuming the host was still on 11.1.0. The host drifted to 12.4.x months ago under :latest; a rollback to 11.x would mean 11 months of reverse travel, panel-schema rollback, and re-testing every dashboard. No security benefit we can't get by pinning forward to the latest 12.4.x.

Supersedes: the short-lived ADR-054 draft in PR #286 (Grafana-only pin — the 054 slot was reassigned on main to the ai-frontend three-mode-routes decision on 2026-04-22, so the Grafana pin needed a new number and this ADR broadens the scope at the same time). Superseded by: