ADR-051: Observability-by-default in the `build-full` pipeline

Status: Accepted Date: 2026-04-22

Context

build-full (the 73-step pipeline at ai-research-dev/cmd/server/pipeline/) generates brand-new SaaS products from briefs. Every product shipped through this pipeline today comes out of the factory obs-UNWIRED: no pkg/observ import, no SetBuildInfo, no /metrics or /probe routes, no version/commit ldflags in the Dockerfile, no Grafana folder or dashboards. Someone — usually an operator — then runs gyrum-observ-integrate (ADR-045) to retrofit the 6–8 stage wiring by hand before the product is useful in production.

The retrofit works but it is the wrong default. The factory's whole point is to ship things ready to run; "we generate everything except the one thing that tells you whether it's running" is an obvious hole, and it compounds: the longer a generated product lives without observ, the more its main.go drifts away from the reference shape, and the more surgical the retrofit becomes.

The hard part is the splice. Backend code inside build-full is authored by an LLM (see buildBackend in ai-research-dev/cmd/server/pipeline/build_phase.go, step 15 of BuildPhaseSteps()). The shape of the generated cmd/<slug>/main.go varies run-to-run — the router may be stdlib net/http, chi, or a hand-rolled ServeMux; the middleware chain may be composed inline or hidden behind a helper; imports may be grouped by tool or by hand. Any regex- or AST-based edit that tries to reach into that output and splice in observ.Access() + metrics.Instrument is exactly as fragile as the variance in the generator. An earlier attempt to implement "observ-by-default" this way was correctly halted for that reason.

Live state this ADR must preserve as-is:

ADR-050 — every product's /metrics route is wrapped in observ.MetricsAuth(token), with the token sourced from OBSERV_METRICS_TOKEN_FILE. The docker-volumes.txt mount for the token file and the env-var reference in .env.example are part of the wiring contract.
ADR-048 — Caddy sites on ops VPS are codified, not hand-edited.
ADR-047 — each product gets its own Grafana folder (uid = slug) and pins the prometheus / loki datasource uids.
ADR-046 — the Tier-1 dashboard pack is the canonical per-product set.
gyrum-dashboard-scaffold shells out to templates at dark-factory/infrastructure/grafana-templates/; its contract is the stable seam, not the templates themselves.

Decision

Make the build-full pipeline produce fully obs-wired products on first emit, via a two-part sequenced change: the generator writes most of the wiring itself, and a new verifier/gap-filler pipeline step confirms it and plugs deterministic holes. Arbitrary regex or AST editing of LLM-generated main.go is explicitly rejected as a mechanism — we do not parse the output shape, we steer the generator toward a shape we can splice into at a known marker, and we WARN (never fail) when the marker is missing.

Part 1 — prerequisite: modify the `buildBackend` prompt

The prompt in build_phase.go's buildBackend CustomFn is updated so that the generated backend, from run 1, already includes:

import "github.com/gyrum-labs/gyrum-go/pkg/observ" and the metrics sub-package (ADR-030).
observ.SetBuildInfo(version, commit) at startup (ADR-032), sourced from -ldflags vars.
/metrics route wrapped in observ.MetricsAuth(token) per ADR-050.
/probe route per ADR-009.

A stable marker comment immediately before the final middleware.Chain(...) call in the server's middleware setup:

// observ:wiring-point (do not remove — see ADR-051)
handler := middleware.Chain(
    observ.Access(),
    metrics.Instrument(),
    // ...product middleware
)

The generator becomes the primary author of observ-wired code. The pipeline step that follows is a verifier and gap-filler, not a surgical editor.

Part 2 — the step: "Wire observability"

A new step is added to BuildPhaseSteps() in build_phase.go, immediately after "Build backend from blueprint" (step 15) and before "Build frontend (TDD)" (step 16). The step is a CustomFn, not an AI-prompt step — its logic is deterministic.

For each product directory, the step:

Validates the generated cmd/<slug>/main.go:
- imports pkg/observ and pkg/observ/metrics?
- calls observ.SetBuildInfo(...) at startup?
- registers /metrics (wrapped via MetricsAuth) and /probe routes?
- the // observ:wiring-point marker is present with observ.Access() and metrics.Instrument() inside the chain?
Gap-fills deterministically where the answer is cheap and unambiguous:
- If the marker is present but observ.Access() or metrics.Instrument() is missing from the chain, splice them in at the marker. The splice is text-level — but anchored to the marker, not to parsed AST — so it is local and idempotent.
- If the marker is absent, do not try to guess where to splice. WARN into OBSERV-WIRING.md and move on.
Patches the Dockerfile deterministically: Dockerfile generation already follows a fixed template, so merging ARG VERSION / ARG COMMIT + -ldflags into pkg/observ.buildVersion / buildCommit is safe to do directly (see the reference Dockerfile at distill-gyrum-ai/Dockerfile). If an existing -ldflags clause is present, merge into it — Go honours only the final -ldflags, so doubling the flag silently drops earlier values.
Scaffolds Tier-1 dashboards by shelling out to gyrum-dashboard-scaffold with the product slug (ADR-046, ADR-047). Reimplementing template substitution inline would drift the two paths.
Writes docker-volumes.txt with the MetricsAuth bearer-token mount line per ADR-050, and adds OBSERV_METRICS_TOKEN_FILE to the generated .env.example. The pipeline does not generate the token value — that is an operator secret, provisioned during deploy.sh (see "Open question: token provisioning" below).
Writes OBSERV-WIRING.md at the product root: a short, machine-readable status summary listing which checks passed, which gaps were filled automatically, and which gaps require gyrum-observ-integrate to complete. This file is the operator's landing page for "what did and did not happen."

Explicitly out of scope

Frontend observability is deferred to a future ADR (tracked as ADR-05X). The SvelteKit side needs a different story: structured logging from the server hooks, client-side error reporting, no /metrics endpoint, and potentially a different scrape topology. The gap is known, acknowledged, and deliberately left for that ADR.

Consequences

Good:

Observ-by-default on new products. Every product emitted by build-full after this change ships with pkg/observ wired, /metrics bearer-auth'd (ADR-050), a provisioned Grafana folder (ADR-047), and a Tier-1 dashboard pack (ADR-046).
Bearer-auth ships on day one. There is no "we'll add auth later" debt window. Distill's issue #269 does not repeat for new products.
The retrofit CLI keeps its role. gyrum-observ-integrate does not compete with this change — it becomes the repair tool for pre-existing products and for drifted integrations. New products generated by build-full do not need to run it; operators invoke it only when OBSERV-WIRING.md says a gap was not filled, or when an older product is being brought up to the current contract.

Trade-offs:

The buildBackend prompt becomes load-bearing on the marker. Test coverage must exercise that the marker is emitted across model versions. This is real maintenance surface — when Claude's model version is bumped, the prompt must be re-validated.
WARN-over-fail. If an AI run forgets the marker, the step records it in OBSERV-WIRING.md and the pipeline proceeds. The product deploys but is only partially wired until someone runs gyrum-observ-integrate. This is deliberate — a full pipeline fail on a prompt-drift issue would be worse than a partial wiring with a loud, structured note.
Implementation size. The step is ~200–400 lines (validator + deterministic splicers + Dockerfile patcher + scaffold shell-out + volume/env writer + summary writer). Not tiny, but each sub-unit is testable in isolation against a fixture main.go.
Wall-clock cost. +5–15s on a ~73-step run. Negligible.

Risks:

Prompt drift as model versions evolve. Mitigation: golden-fixture tests that pin the expected marker shape and post-generation validator pass.
Marker silently removed by a future prompt edit. Mitigation: a codified contract comment on the prompt block pointing at this ADR, plus a validator test that fails if the prompt string no longer contains the marker instruction.

Alternatives considered

(a) Pure regex/AST editing of generated main.go. What was initially attempted. Rejected: regex brittleness on LLM output is the entire problem; AST rewriting moves the fragility from "which characters match" to "which syntactic shape matches," and the shape varies just as much. Net win is small; surface is large.
(b) Post-pipeline retrofit step at first deploy. Run gyrum-observ-integrate as the last action of deploy.sh. Rejected: violates "observ by default"; shifts the burden from the build-time context (code, prompts, fixtures all together) to deploy-time (where context is gone). Also brittle — a product deployed by a third party may never reach this step.
(c) Port gyrum-observ-integrate into Go as a pipeline-runnable library. Tempting — the CLI already does all of this. Rejected for this ADR: the CLI's shape is <1 month old; extracting a library now locks in premature abstractions. Revisit once the CLI has stabilised across 2–3 more products.
(d) Marker-splice + generator-prompt change — chosen. See above.
(e) Extend the existing "Monitoring setup" step (step 63 in InfraPhaseSteps()) from AI-generated markdown to real wiring. Rejected: that step writes infrastructure/monitoring.md (docs), not code. The wiring targets (cmd/<slug>/main.go, Dockerfile) are authored in the Build phase — by step 63 they are frozen. Also: "Monitoring setup" and "observ wiring" have different contracts.
(f) Emit observ via a template engine, replacing the LLM for this surface. Rejected: the pipeline's contract is "AI writes production code." Carving out a template island makes that surface non-idiomatic; the marker approach delivers the same outcome while keeping the AI as author.

Explicit decisions the reader should be able to re-examine later

Where the marker comment goes. In the middleware setup, immediately before the final middleware.Chain(...) call in server-setup (commonly in main.go or a registerRoutes-style helper). The concrete snippet is under "Part 1" above. If a future generator structure moves chain composition elsewhere, this is the first thing to revisit.
Fallback when the marker is missing. WARN into OBSERV-WIRING.md, log a structured INFO from the pipeline step (msg="partial wiring — run gyrum-observ-integrate to finish"), pipeline proceeds. The product deploys with /metrics + /probe routes (if the generator got those right) but possibly without observ.Access() + metrics.Instrument() in the chain.
The MetricsAuth token is not generated by the pipeline. The token is an operator secret per ADR-050. The pipeline generates the docker-volumes.txt mount line and the .env.example env-var reference only. First deploy provisions the token from the deploy.sh flow; the observability runbook is the canonical procedure.
Interaction with gyrum-observ-integrate. The CLI becomes the repair tool: run it on products that predate this ADR, and on any new product whose OBSERV-WIRING.md reports a gap. There is no collision — both paths converge on the same contract (ADR-045's integration spec), and the CLI's --check mode is the audit loop for either origin.

Decisions (formerly open questions — reviewed 2026-04-22)

Marker format: human-readable // observ:wiring-point (do not remove — see ADR-051). Directive form was considered and rejected: the marker is load-bearing documentation as much as machine signal, and future human readers of LLM-generated code need the hint to not silently delete it during cleanup.
Where the new step lives: last step of BuildPhaseSteps(), after "Page content verification". A wiring-induced build break is then caught against a smoke-tested baseline, not before the baseline exists. Also localises "observ wiring failure" away from the backend-build step's own failure mode, which keeps root-causing simpler when something goes wrong.
Scope of OBSERV-WIRING.md: per-product status file at the generated product's root. Does NOT write into dark-factory/docs/observability-fleet-status.md — that matrix is regenerated by the ADR-045 audit script and any hand-written content is blown away on the next run. Per-product is where developers of that product first look; cross-product audit is already covered by scripts/observ-integration-audit.sh.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

ADR-008 — single library for observ primitives.
ADR-030 — metrics sub-package.
ADR-032 — build-info identity gauge.
ADR-045 — gyrum-observ-integrate as the retrofit CLI; becomes the repair tool under this ADR.
ADR-046 — Tier-1 pack.
ADR-047 — folder-per-product Grafana convention.
ADR-050 — bearer-auth on /metrics; hard dependency of this design.
ai-research-dev/cmd/server/pipeline/build_phase.go — BuildPhaseSteps() and buildBackend.
docs/observability-runbook.md — operator procedure for token provisioning on first deploy.

ADR-051: Observability-by-default in the build-full pipeline