ADR-088: Observability lifecycle — register-on-provision, silence-before-destroy

Status: Accepted Date: 2026-04-25 Related: ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-084 (outbound-call chokepoint), ADR-087 (cloud-init bootstrap)

Decision (one paragraph)

Every gyrum-managed host registers with the observability stack as a final step of its provision pipeline and de-registers as the first step of its destroy pipeline. Registration writes a Prometheus file-SD target, configures Promtail with host-specific labels, and adds an Alertmanager route. De-registration silences alerts first, then drops the scrape target, then waits for Prometheus to stop polling before the host itself is destroyed — otherwise the 30-60s gap between server-delete and scrape-config reload fires false "target down" alerts that operators learn to ignore, which is worse than no alerts at all.

Context

Today's gap: provision-and-deploy-warp.yaml produces a running host but Grafana can't see it without operator hand-configuration — somebody has to SSH to the ops host and drop a JSON file into /etc/prometheus/targets/ by hand. Destroy doesn't exist as a playbook at all — the only way to remove a host is hcloud server delete, which fires "target down" alerts for the lag window between VM-vanishes and Prometheus-notices. Operators suppress those alerts by developing a "probably a destroy" heuristic; that heuristic silently kills real alerts on real failures because the category "target down, maybe intentional" becomes ambient noise.

The structural fix is observability lifecycle ownership moving into the pipeline. Provisioning that leaves a host invisible to Grafana is not provisioning — it's half-provisioning. Destroy that fires phantom alerts is not destroy — it's destroy-plus-pager-noise. Both are solvable with the same mechanism: the playbook owns the observability handshake, register on provision and de-register on destroy, with the alert-silence ordering strict enough that the "target down" window never becomes a user-facing event.

The incident-shape we're avoiding is not hypothetical. Every infrastructure team that scales past a handful of hosts has a "somebody manually removed a VM and paged everyone at 3am" story. The fix that teams arrive at independently is always the same: the tool that removes the VM must also remove the observability record, in that order, before the VM goes away. This ADR codifies that order as a playbook invariant so it can't be skipped by a hurried operator reaching for hcloud server delete directly.

The three signal types

Metrics via Prometheus file-SD. Prometheus at dark-factory-ops reads /etc/prometheus/targets/*.json via file_sd_configs. Adding a file means a new scrape target with no Prometheus reload needed. Removing a file means the target goes away. The mechanism is idempotent (write-or-update, delete-if-exists), grep-able (one host per filename), and survives Prometheus restart because the files are on disk. Each target file is a single JSON object with targets: [<host>:9100, <host>:443] and labels: {host, service, env} — the port list covers node-exporter plus the service's own /metrics endpoint, the label set is what Grafana and Alertmanager route on.
Logs via Promtail → Loki. Each host runs promtail as a systemd service, shipping journald and /var/log/ tails to loki.gyrum.ai. The config is templated per-host with labels host=<name>, service=<role>, env=<prod|dev>. Binary install happens in cloud-init (see ADR-087); config rendering is a pipeline concern because the labels are host-specific. Promtail's positions-file (/var/lib/promtail/positions.yaml) survives restarts so log shipping is checkpointed; on a clean re-provision the positions file is gone and promtail starts from the journal head, which is the right default — we don't want to replay a week of logs from a reused hostname.
Alerts via Alertmanager routes. Per-service receivers (ops-warp, ops-distill, etc.) are pre-configured in Grafana at cluster setup time; what changes per-host is the set of hosts each route covers. File-SD labels solve this: the route matches on service=warp and picks up every host whose scrape target carries that label, so no route config edit is needed when a host joins or leaves. The per-host silences that destroy files are orthogonal to route config — a silence targets host=<name> regardless of which route would have matched, so destroy doesn't need to know which receiver covers the host.

Why file-SD over static config or Consul

File-SD is zero-infra — Prometheus already supports it, no extra service needed. It is zero-restart — the file_sd module polls the directory and auto-reloads on change. It is idempotent — writing the same target file twice is a no-op. It is reviewable — a diff of /etc/prometheus/targets/*.json shows exactly what targets the system believes exist. Consul is infrastructure we don't need for five hosts; the operational cost of running it outweighs the expressiveness gain. Static config in prometheus.yml requires operator edits to that file, which don't scale past a handful of hosts and are drift-prone because two operators editing the same file on different days produce merge conflicts rather than idempotent updates.

The file-SD choice also plays cleanly with the pipeline's dry-run mode. A dry-run of destroy prints the path it would delete; an operator can confirm the file exists with ls before authorising the real run. A dry-run against Consul or a Prometheus HTTP API would require mocking out the provider or running against a staging observability stack — structurally more ceremony for the same signal. Choose the boring mechanism when it fits; the interesting mechanisms are for the problems that demand them.

Destroy-order invariant

Silence-before-delete is not an optimization — it is a correctness property. The order is:

Alertmanager silence for host=<name> with a 60-second Time To Live (TTL), filed as "destroy pipeline".
Delete scrape target file — rm /etc/prometheus/targets/<name>.json on the ops host.
Poll up{host=<name>} until Prometheus no longer sees the target, capped at 30s.
Delete the Hetzner server — the destructive action. Everything above is recoverable; this is not.
Delete DNS record in Cloudflare.
Tidy inventory + vault — strip the inventory entry, leave the vault secret (rotation is a separate concern).

Steps 1-3 reverse-order-protect alerting: silence covers the window where Prometheus still thinks the target exists, the file delete closes the window, the poll confirms the window closed. Step 4 is the destructive action, gated behind a dry-run flag. Steps 5-6 are cleanup; failures there are loggable annoyances, not alert-pager incidents.

Register-vs-destroy symmetry

The two pipelines are mirror images:

provision:                          destroy:
  ... (existing steps) ...            silence alerts (60s TTL)
  register scrape target              delete scrape target
  configure promtail                  wait for prometheus to stop polling
  add alertmanager receiver           delete alertmanager receiver
  deploy                              delete DNS record
  verify                              delete Hetzner server
                                      clean inventory + vault
                                      verify Grafana shows empty

Mirror-image is not an accident. The invariant is "at every moment, the set of observability records matches the set of live hosts". Provision appends a record before the host goes live to Grafana; destroy removes the record before the host disappears from Hetzner. The playbook is the enforcement mechanism; the invariant is the correctness property.

What cloud-init does vs what the pipeline does

Cloud-init installs binaries (node-exporter, promtail) at first boot — stateless work that belongs in the baked image. The pipeline writes configs with host-specific labels — stateful work that depends on the host name and role, which cloud-init doesn't know at template-bake time. Keeping the split sharp means the cloud-init template stays reusable across service roles (warp, buzzy, postgres all share the same baseline binaries), and label drift stays out of baked images (changing from env=prod to env=staging is a pipeline-input change, not a template rebuild).

The handover point mirrors ADR-087's pattern: cloud-init writes /etc/gyrum/onboarded as a readiness marker; the pipeline's configure_promtail step checks for it before dropping the config. If the marker is missing, the host failed to bootstrap cleanly and observability config would be wasted effort. The pipeline surfaces the missing-marker error directly, so the failure mode is "cloud-init broke, investigate there" rather than "observability silently missing, discover later".

Consequences

Positive: Grafana auto-picks-up new hosts with no operator-in-the-middle; no alert-noise on destroy because silence-before-delete is structurally enforced; operators never edit prometheus.yml; destroy-without-silence becomes impossible because the playbook is the only supported destroy path. The pipeline is the observability contract; an operator who bypasses the playbook (direct hcloud server delete) gets the noisy-alert behaviour, which is a loud feedback signal that the workflow contract was skipped.

Negative: destroy takes ~30s longer than hcloud server delete would take in isolation (silence filing + the scrape-drop poll); per-service promtail config introduces template sprawl if services multiply without a shared base. Phase 2 generalises the per-service templates. The dependency on ops-host SSH reachability is a new coupling — today's provisioning was ops-host-independent, tomorrow's is not.

Risks: if the ops host itself is unavailable, provisioning blocks on the register step — a fallback skips register on ops-unreachable with a loud warning and marks the host as observability-pending in the inventory, reconciled later. Without the fallback, a dead ops host cascades into a fleet-wide freeze on new provisions, which is worse than a single host running unmonitored for an hour. A second risk is ops-host-compromise blast radius: an attacker with ops-host SSH can write fake scrape targets or delete real ones, which would corrupt the observability view. Mitigation is ansible-vault-protected SSH key rotation on a quarterly cadence and audit-log on every target-file write (Phase 3 provider handles the audit-log; today's shell-SSH flow is opportunistically logged via auth.log).

Phases

Phase 1 — warp + destroy-warp, with ops-host SSH writes. This ADR's scope. destroy-warp.yaml is net-new; provision-and-deploy-warp.yaml gains two steps (register_scrape, configure_promtail) between deploy and verify. The ops host writes are over SSH using the existing ansible SSH key; no API-in-the-middle. Ships alongside the ADR.

Phase 2 — generalise to destroy-service.yaml + register-observability.yaml sub-playbooks. When the second service (buzzy, postgres, or distill) needs the same handshake, extract the observability plumbing into a register-observability subplaybook that takes service, host, env, labels as inputs. destroy-warp.yaml collapses into a thin wrapper that calls destroy-service with warp-specific values. Prerequisite: subplaybook executor (ADR-068 Phase 4).

Phase 3 — outbound-toolkit integration. A pkg/outbound/prometheus/ provider in the gyrum-go monorepo handles target-file writes via typed block instead of shell-SSH. Requires either a Prometheus HTTP API for target management (does not exist today) or a git-committed scrape config model (requires a pipeline → git → Prometheus loop). Scheduled after Phase 2 settles the abstraction and ADR-084's chokepoint lands its second provider.

Phase 4 — cloud-init for promtail config too. The promtail config is mostly static across hosts; the only truly per-host piece is the label set. When ADR-087's init-secret pattern lands (Phase 4 of that ADR), the promtail Loki auth token is delivered via user-data and the labels are rendered into the template at cloud-init time. The pipeline's configure_promtail step then collapses to "cloud-init already did it", same way harden collapsed in ADR-087.

ADR-072 — infrastructure as playbooks. This ADR is a refinement of what a provision pipeline's final step looks like and what a destroy pipeline's first step looks like, not a departure from playbook-as-primitive.
ADR-076 — project-to-host binding. The observability labels written by register_scrape are derived from the same service + host identity this ADR binds at provision time.
ADR-084 — outbound chokepoint. Phase 3's typed Prometheus provider lives alongside the hcloud, cloudflare, and ansible providers there.
ADR-087 — cloud-init bootstrap. Binary installs for node-exporter and promtail live in the cloud-init template; configs live in the pipeline. Phase 4 of this ADR merges the two once init-secret delivery lands.