ADR-045: gyrum-observ-integrate CLI supersedes the Phase-1 manual checklist

Status: Accepted Date: 2026-04-21

Context

The original observability playbook was a Phase-1-shaped checklist: nine numbered steps an operator copy-pasted into a service to wire observ. Each step produced the same edits in every service — the same imports, the same SetBuildInfo call, the same Dockerfile ldflags (ADR-032).

Two things have changed since that playbook shipped:

1. Scope grew. Phase 2a (HTTP middleware), Phase 2b (metrics.Instrument
   • RegisterAppInfo), and Phase 2c (WrapDSN + Probe) each added new wiring steps. The checklist is now 15+ items, including ordering constraints (observ.Access BEFORE metrics.Instrument; metrics.RegisterAppInfo AFTER SetBuildInfo; see distill-gyrum-ai/internal/middleware/stack.go). Every new service gets a fresh chance to mis-order them.
2. The fleet grew. There are 30+ generated product repos plus the named services (distill, buzzy, guardian, control-plane, hiphip, etc.). A manual rollout across them would take hours and drift — the playbook would have been wrong within a week of the first Phase-2 ship, and silently.

The existing warn-level linters in devtools (lib/observ-linters.sh, see ADR-003) already detect drift — they flag log.Print* in observ-importing services, Dockerfiles without observ ldflags, and main.go files missing SetBuildInfo. But they don't fix it.

Decision

Ship a devtools CLI — gyrum-observ-integrate — that owns the wiring contract end to end. The CLI:

• Is the one-command scaffolder for new services.
• Is the one-command upgrader for existing services.
• Has a --check mode that reports integration status without making any change, and exits non-zero when a service is not fully integrated.
• Is idempotent: every stage checks the "already done" state and skips.
• Is conservative: when a change cannot be made safely (ambiguous router, non-reference main.go shape, hand-composed middleware chain), it leaves a TODO observ-integrate: comment and flags the stage rather than guess.

The manual checklist survives as Appendix A of the rewritten playbook — it is the fallback when the CLI hits an ambiguous case and the exact specification that the CLI's apply stages follow. The appendix and the CLI are kept in sync by hand; drift between them is a review-blocker.

A companion dark-factory/scripts/observ-integration-audit.sh walks every gyrum-labs repo and runs gyrum-observ-integrate --check across it, collating the matrix into docs/observability-fleet-status.md. That matrix is the coverage dashboard the platform team reads.

Consequences

• One source of truth for the integration contract. The CLI's apply stages are the machine-readable spec; the playbook's appendix is the human-readable spec; the linters remain the PR-time smoke test. All three trace back to the reference integration in distill.
• Fleet rollout becomes a script, not a project. For each service: gyrum-observ-integrate <slug> --drill-key <key> → review diff → PR. 30+ repos complete in a day, not a quarter.
• --check is the audit loop. The audit script wraps it, the platform team gets a weekly matrix, and the same --check runs inside any CI workflow that wants to gate on full integration.
• TODO comments become the manual-intervention queue. The ambiguous cases the CLI refuses to automate (custom routers, gqlgen wiring, non-cmd/*/main.go layouts) surface as structured TODOs in the output, not silent gaps. They are the known long tail of the contract.
• The playbook gets shorter. Step 1 is the CLI. Steps 2–5 are verification + PR + deploy + post-deploy. The 15-step manual checklist collapses into an appendix operators consult only when the CLI tells them to.
• The CLI owns a growing contract. Every future observ addition (tracing, SLO annotations, profile export) adds a stage. The cost of that growth is carried once, in the CLI, instead of once per service.

Alternatives considered

• Keep the manual checklist; rely on linters to catch drift. What we had. The linters catch the symptoms but require an operator to type the fix into every service. At fleet scale, fixes lag new services being generated, so coverage asymptotes at "most but not all."
• A gyrum-setup flag (--with-observ). Bolts the wiring onto the bootstrap path. Works for new services but offers nothing to existing ones, which is where the coverage gap lives.
• A Go code-mod tool (gofmt-style AST rewrite). Higher fidelity for complex main.go shapes but a much larger maintenance surface, and the conservative-TODO path we want already keeps the CLI safe. Reserved for Phase 3 if the TODO tail is too long to live with.
• CI-only enforcement — force every PR to fully integrate. Brutal and wrong. A service that never logs to stderr or never serves HTTP does not need the full middleware chain. The CLI stays opt-in; the audit surfaces who has opted in and who hasn't.
• A generator template under templates/ instead of an integrator. Templates are excellent for new services but unusable for existing ones (the cost to retrofit is the whole point of the problem). We keep the stack templates in ~/.gyrum/devtools/stacks/ for the new-service path and the CLI for the upgrader path; they share the reference integration.

Supersedes: the Phase-1 version of docs/observability-playbook.md (superseded in the same PR). Superseded by: