ADR-009: Probe(ctx, label) — a two-line primitive for end-to-end log-pipeline self-test

Status: Accepted Date: 2026-04-21

Context

The log pipeline has six serial stages (see observability-architecture.md#data-flow--logs): app → stderr → docker json-file → Alloy → Loki → Grafana. Any one of them can drop out silently:

• Alloy dies, docker keeps writing to disk, nobody notices until a later query returns no rows.
• The redactor regresses and starts letting the probe's dummy password through — we wouldn't know until someone checks.
• Loki starts dropping by a label selector after a config change; some services are visible, some aren't.

Without an explicit end-to-end test, the first indication that the pipeline has broken is usually "I went to debug an incident and the logs weren't there" — at exactly the moment logs are most needed.

We need a primitive that:

• Exercises every stage of the log pipeline from inside a running service.
• Exercises the redactor deliberately (proves it fires on a known sensitive key).
• Produces a signal a dashboard can plot and an alert can fire on.
• Requires nothing beyond the library itself — no HTTP middleware, no metrics endpoint, nothing from Phase 2+.

The library ships observ.Probe(ctx, label). Described in gyrum-go pkg/observ/README.md and called from the /healthz handler in the playbook.

Decision

observ.Probe(ctx, label) emits two log lines with a shared probe_id:

1. An INFO line with the label and the probe_id.
2. A WARN line with the same probe_id and a dummy sensitive attribute ("password": "swordfish").

Both lines carry the standard identity attributes (service, version, commit, release_track) and any context-logger decoration. The probe_id is a random 16-hex value; the INFO/WARN pair is grouped in Grafana panels by joining on it.

Conventions for the label argument are in observability-standards.md#probe-naming-conventions: boot, readiness, canary-tick, <stage>-check.

The Phase 5 "Pipeline Health" dashboard panel filters on probe_id != "". A missing probe pair or a WARN line containing the literal string swordfish instead of [REDACTED] are both alert conditions.

Consequences

• One primitive, whole pipeline covered. The pair of lines with matching probe_id exercises all six pipeline stages. If both show up in Grafana, the pipeline works. If only the INFO shows up, something after the INFO's level-gate is broken (unlikely but detectable). If neither shows, shipping or ingest is broken.
• Level coverage is real. INFO + WARN exercises two level thresholds, guarding against a misconfigured OBSERV_LEVEL that would otherwise let INFO through and drop WARN (or vice versa).
• Redaction regression is a P1 signal. The WARN line carries a known-sensitive attribute (password: swordfish). If swordfish ever appears in Grafana, the redactor has regressed — this is stated explicitly in the library README's "Gotchas" section. A regex monitor on {service=~".*"} |= "swordfish" fires immediately.
• Callers choose cadence. Probe(ctx, "boot") in main() fires once per process start. Probe(ctx, "readiness") in a /healthz handler fires every orchestrator poll. Probe(ctx, "canary-tick") on a timer during canary rollout. The library has no opinion on frequency; the label indicates intent and the dashboard groups by it.
• Cost is two log lines per call. Every call produces two Loki lines and paid-for chunks. High-frequency probes (sub-second readiness hits) are the caller's cost to own. The standards recommend matching the orchestrator's poll rate, not exceeding it.
• Probe label cardinality is governed like any other field. probe_id is unbounded (per-call UUID) and stays a field; the label is bounded-ish by naming convention and also stays a field — neither becomes a Loki label. Consistent with ADR-005.

Alternatives considered

• Single log line. Emit one INFO, call it done. Rejected: doesn't exercise the WARN-level gate, doesn't create a pair for the redactor test, and gives the dashboard less to grip — a single line can be dropped silently; a missing half of a known pair cannot.
• Ping a /health HTTP endpoint (external prober). Grafana's "synthetic monitoring" pattern. Rejected: requires HTTP middleware (not shipped in Phase 1), cross-cuts into the networking stack, and only covers the part of the pipeline the external prober can see. The library-internal probe catches failures the external prober would miss (e.g. Alloy tail broken on this host but not others).
• Panic and recover. Emit an ERROR via recover() to exercise the error path. Rejected: panics in a supposedly healthy code path are noise, and the ERROR-level path is exercised on every real failure anyway. A panic-based probe would create false triage load.
• Metrics-based heartbeat (observ_probe_total counter). Would be ideal long-term. Rejected for Phase 1: metrics ship in Phase 2. The log-based probe is the bootstrap; a metrics-based heartbeat is a future addition, not a replacement.

Supersedes: none Superseded by: