ADR-041: The "Verify observability" deploy stage is non-blocking

Status: Accepted Date: 2026-04-21

Context

deploy.sh gained a Verify observability stage (ADR-040) that probes Loki for recent log lines and Prometheus for the app_info gauge (see ADR-032) after the service is live. The point is to catch the "deployed but the dashboards are empty" silent failure class — a container running but not shipping signals.

The natural follow-up question is: if verify fails, should the deploy fail?

The tempting answer is "yes — a service without observability isn't really deployed." But the stages this one probes are an entire second stack (Alloy, Loki, Prometheus) with its own failure modes:

Loki ingest is backed up → recent lines not queryable yet.
Alloy's WAL is replaying from an earlier outage → all services look silent for a few minutes.
Prometheus scrape interval (15 s) hasn't caught the new container yet.
The Grafana/Loki/Prom VPS is itself rebooting for an unrelated reason.

Failing the deploy in any of those cases would roll back a working service because its monitoring is slow. That is worse than the problem we are trying to detect.

Decision

The Verify observability stage never fails the deploy. Each check either logs ✓ on success or ⚠ on failure, and the stage exits 0 unconditionally. The same non-blocking contract applies to the Grafana dashboards stage that precedes it.

The warning line is the signal. Alerts on the observability pipeline itself are a separate on-call concern handled by the platform dashboards (ADR-015), specifically platform-log-pipeline-health and platform-probe-freshness.

Consequences

Deploys stay decoupled from the monitoring stack. A Loki outage cannot stop a service from shipping. That is the correct decoupling — Grafana is a diagnostic tool, not a runtime dependency of the services we deploy.
The warning is visible but not alarming. The Platform UI's stepper will show the stage completed; the operator sees the ⚠ line in the log. Grep-friendly, easy to tail for during an investigation.
Separate alerting catches persistent failures. If observability is actually broken (not just "slow right now"), the platform-level alert on Alloy WAL depth and Prometheus up=0 will fire well before the operator has to notice the deploy-log warnings. The verify stage is a spot check, not a replacement for those alerts.
Requires operator discipline. A ⚠ line can be ignored. Two or three in a row should not be. We lean on the platform dashboards and on on-call training, not on failing deploys, to keep this from silently rotting.
Dry-run shows the same markers. Because dry-run is about the deploy contract not the verification, the verify stage is skipped entirely in dry-run mode; the markers only appear on real deploys.

Alternatives considered

Fail the deploy on verify failure. Safer-feeling, actually worse: ties the service's shippability to an unrelated stack's health. Rejected as a false-alarm generator that would train operators to routinely bypass it.
Retry the check for 5 minutes before failing. Handles the Alloy-replay case but not the "Loki is down" case, and adds 5 minutes to every deploy. Rejected: the verify is a spot check, not a synchronous health gate; if we need a gate, it belongs in the runtime alerting path, not the deploy script.
Emit a machine-readable event (verify.ok=false) in place of the human-readable line. Cleaner for downstream tooling, but we have no consumer today and the stage-marker contract (ADR-141) already carries the grep-friendly stdout we want.
Skip the verify stage entirely — rely on platform dashboards. Simpler, but the "I deployed but my dashboards are empty" class is real and platform dashboards don't look at it on a per-deploy cadence. A tight feedback loop at the deploy boundary catches 90% of the silent failures before the operator has navigated away.

Supersedes: none Superseded by: