ADR-104: Dry-run-on-history adaptive gate

Status: Proposed Date: 2026-04-26 Related: ADR-067 (playbooks-unified-primitive), ADR-068 (playbook-runtime), ADR-083 (rule-promotion-engine), ADR-090 (ai-as-scaffold-pipelines-as-runtime)

Numbering note. The originating brief and warp epic (warp #313) named this "ADR-100". Slot 100 was already filled by 100-factory-growth-boundaries.md (landed earlier). Per the README numbering rule ("Never reuse a number, even for a superseded record"), this decision takes the next free slot — 104 (slot 103 was also taken by 103-pm-stays-in-warp.md). Inbound links from warp #313 / dark-factory CLAUDE.md / cascade tickets that cite "ADR-100 dry-run gate" should resolve here.

Read-first for agents. If you landed here because a --check dry-run gate fired and stopped your fire, the section you want is Decision §4 — Verdict shape and recursive failure below. Read the verdict, do not retry blindly, and do not reach for --skip-dry-run until you have read §5 — Override path.

Context

Today's motivating case

On 2026-04-26 the deploy-warp.yaml playbook silently shipped a no-op fire (warp #312). The Ansible playbook ran, exited 0, and verify_version checked the wrong artifact — so the dashboard reported green while no host had actually been touched. The fault was structural: nothing dry-ran the playbook before the real fire, so the "no inventory parsed → no hosts matched" failure went invisible. A pre-fire ansible-playbook --check would have surfaced "no hosts matched any of the patterns" in the first second of the run, before any state-mutating action could be attempted.

This is not a one-off. It is an instance of a class — playbooks whose failure mode is silent success. Other instances live in the fleet today:

• A Terraform plan that no-ops because all resources match cached state, even when the operator intended a state change.
• A kubectl apply against a stale context that succeeds against the wrong cluster.
• A shell playbook that early-returns from a set -e chain because a guard at the top of the script swallowed the real condition.

Each one ships green. Each one is invisible to the operator, to the dashboard, and to whatever downstream system is supposed to consume the work. The longer the silent-success persists, the more the fleet trusts a lie.

Why an adaptive gate, not an always-on one

Always-on dry-run is the structurally-cleanest answer and the operationally-worst. It doubles the wall-clock cost of every fire, even for playbooks that have run cleanly for months. It pays the dry-run tax against playbooks where the failure mode does not exist (idempotent infrastructure that has been green for 100 fires running). It also trains operators to ignore the dry-run output, because most of the time the dry-run is a noise gate.

The asymmetry is the load-bearing observation: healthy playbooks should pay nothing; failure-prone playbooks should pay the full dry-run tax until they prove healthy again. That is the shape of an adaptive gate.

Why a Claude verdict, not a simple pass/fail

A dry-run output is not a binary. ansible-playbook --check may produce:

• Clean diff with zero changes — playbook is a no-op against the current target. Could be correct (idempotent re-run) or could be the silent-success failure mode. The operator's intent matters.
• Diff with N expected changes — playbook is going to do work. The shape of the work matters: are they the resources the playbook was named after, or something incidental?
• Plugin / connection errors before any task ran — playbook is broken at the harness layer (no inventory, missing credentials, unreachable target). This is the warp #312 class.
• Partial diff with explicit failure — playbook would have half-applied. The most dangerous shape; the operator must decide whether to proceed knowing the rollback story.

Mapping these onto a pass/fail boolean throws away the only signal that distinguishes them. Claude reading the dry-run output and returning a structured verdict (with reason + proposed fix + evidence URL) preserves the signal and lets the wrapper do the right thing per case. This is the same shape as ADR-083 (rule promotion engine): mechanical surfacing, judgment-layer verdict, mechanical execution of the verdict.

Decision

A pipeline-item playbook with any red fire in its last 5 runs routes its next fire through a --check/plan/--dry-run=server pre-flight; the dry-run output is POSTed to a Claude verdict hook, and the wrapper acts on the structured verdict (go / stop / investigate) before — or instead of — firing for real. Healthy playbooks (last 5 runs all green) bypass the gate entirely. The implementation lives across seven phases tracked in warp #313 (r1 this ADR, r2 the dry-run-clean playbook audit ADR — next free slot when it lands, r3 runtime, r4 verdict hook, r5 wrapper, r6 UI, r7 CLAUDE.md cascade); only r1 (this ADR) ships in this PR.

1. Schema — supports_dry_run per-playbook

The playbook front-matter (ADR-067) grows one optional field:

---
kind: pipeline                        # or service_runbook, release_flow, etc.
id: deploy-warp
supports_dry_run: true                # default false; true requires audit (r2 follow-up ADR)
dry_run_driver: ansible               # ansible | terraform | kubectl | shell
---

Default is false. A playbook author claiming supports_dry_run: true is making a structural promise: --check (or the per-driver equivalent) does not mutate state. The r2 follow-up ADR (deferred, slot assigned when it lands) defines the audit gate that verifies this promise; this ADR establishes the field as the contract surface.

dry_run_driver selects which driver-specific dry-run command the runtime invokes. Unknown drivers fall back to "no dry-run available" and the gate cannot trigger (the playbook is never gated; failures surface only at the real fire, as today).

2. Threshold — any-red-in-last-5

For each pipeline-item, the gate computes a 5-fire window over the playbook's run history and triggers if any of the last 5 fires is red. The window is exact (not "rolling 24h" or "since last deploy") because the relevant signal is "this playbook is currently unstable", not "this playbook had problems sometime recently".

• 0 of 5 red — gate skipped, playbook fires directly.
• 1+ of 5 red — gate fires, dry-run pre-flight required.
• 5 of 5 red — gate fires AND a --skip-dry-run override is refused without operator escalation (the playbook is in a bad enough state that bypassing the gate is not a unilateral decision).

The threshold is deliberately conservative on the trigger side (any-red, not majority-red) because the cost of an over-gate is one extra dry-run; the cost of an under-gate is another silent success.

3. Driver matrix

Adding a driver is an ADR amendment — the driver has to declare its dry-run command and the failure-mode class it catches. We expect one or two amendments in the next year (probably helm template and pulumi preview); more than four suggests the driver matrix wants its own ADR.

4. Verdict shape and recursive failure

The runtime POSTs the dry-run output (stdout + stderr + exit code + captured artifacts URL) to the Claude verdict hook. Claude returns a JSON document:

{
  "verdict": "go" | "stop" | "investigate",
  "reason": "no hosts matched 'warp_canary' pattern in inventory",
  "proposed_fix": "update inventory.yaml: add warp_canary group with current host IPs",
  "evidence_url": "https://factory.gyrum.ai/dry-runs/<fire-id>"
}

Wrapper behaviour per verdict:

• go — fire the real playbook. The dry-run was clean enough for Claude to certify it. Logged to ~/.gyrum/dry-run-verdicts.jsonl with the verdict + evidence URL.
• stop — fire is held. An INCIDENT warp item is auto-filed with discovered_during=<fire-id> and the verdict body inlined as the Why section. Claude works the incident (filed against the playbook's repo, claimed by the agent label that fired the playbook), proposes a fix PR, and on merge the original fire is re-queued through the gate. Cap 2 auto-filed incidents per fire-id; the third stop-verdict pages the operator (warp item flagged priority: high, tags: [needs-operator]).
• investigate — Claude is uncertain. Fire is held; verdict is surfaced on the factory /operate page; operator decides go / retry / abandon. No auto-incident, no auto-fix; this verdict is reserved for cases where the dry-run output is structurally ambiguous (partial diff, mixed exit codes, novel failure shape).

The recursive cap (2) is the load-bearing safety. Without it a playbook with a fundamentally broken target could flap between "stop → file incident → fix attempt → stop again" indefinitely, burning operator quota while making no progress.

5. Override path and audit log

The gyrum-fire-playbook wrapper accepts --skip-dry-run "<reason>" matching the --skip-smoke convention (per the existing enforcement gates section in CLAUDE.md). The reason is required (the wrapper refuses an empty string), is appended to ~/.gyrum/admin-overrides.log with {ts, fire_id, playbook_id, reason, agent_label}, and is also surfaced on the factory /operate page so the operator can see who bypassed the gate and why.

The override is refused outright when the playbook has 5 of 5 red fires — at that point the bypass requires explicit operator action (the wrapper exits with a message pointing to the operator-escalation playbook).

6. Audit log paths

No new storage system is introduced. The threshold reader walks the existing pipeline-item health endpoint; the verdict log is a JSONL file matching findings.jsonl shape; the artifact root is whatever the runtime already uses for fire output.

Consequences

What becomes easier.

• The warp #312 regression class is structurally caught. A playbook whose failure mode is "silent success" cannot ship green more than once before the gate engages — the second silent-success fire goes on the run history, and any subsequent fire is gated.
• Playbook authors get a structural place to put "this playbook supports dry-run" — declared in front-matter, not tribal. The r2 follow-up ADR builds on this: the field is the audit input.
• Operators get a single audit log of every gate decision and every override. The Friday-afternoon question "why did the deploy go through despite the red history?" has a one-grep answer.
• Claude verdicts are appended to ~/.gyrum/dry-run-verdicts.jsonl, so the rule-promotion engine (ADR-083) can over time hoist verdict-shape rules into devtools — "if dry-run output contains 'no hosts matched' AND playbook is kind: pipeline, always stop".

What becomes harder.

• Latency. Failure-prone playbooks pay an extra dry-run round trip + Claude verdict (typically 5–30 seconds for the dry-run, + 2–10 seconds for the verdict). This is on top of the real fire. Healthy playbooks pay nothing. We accept this tradeoff because the alternative is silent failures.
• Audit overhead for the dry-run-clean claim. The r2 follow-up ADR is required — without it supports_dry_run: true is a prose claim, not a structural one, and a playbook that mutates state on --check defeats the gate. The two ADRs ship as a pair; delaying r2 leaves the gate dependent on author discipline.
• The verdict hook is a new dependency. A playbook gated on a verdict that cannot be obtained (Claude unreachable, hook timeout) must fail closed (refuse to fire) — failing open would let the silent-success class through. The wrapper degrades to "cannot obtain verdict; treating as investigate" and surfaces to the operator.
• Recursive failure has a sharp edge. Cap-2 means a genuinely broken playbook will eventually page the operator, which is correct, but a flaky-but-recoverable playbook may also page the operator if the second incident-fix attempt happens to also fail. The cap can be raised in a future amendment if the data shows this happens often.
• Necessary, not sufficient. A green dry-run is not a guarantee of a green real fire. State changes between dry-run and fire, external systems failing during fire, race conditions on shared resources — all of these slip through any gate that reasons from static dry-run output. The gate catches the silent-success class and the structural-broken class; it does not catch the live-state class. This rubric is explicit because the temptation, once the gate is in place, will be to treat a passed dry-run as "the fire is safe". It is not.

What we sign up to maintain.

• The driver matrix. Every new playbook driver requires an ADR amendment naming its dry-run command and failure-mode class.
• The Claude verdict hook contract. The verdict shape is part of the ADR; changes are versioned (schema_version joins the verdict body when needed).
• The audit log retention policy. ~/.gyrum/dry-run-verdicts.jsonl grows unbounded today; r4 owns deciding the rotation cadence.
• The cap-2 threshold. If operators report they're being paged on recoverable flapping playbooks, the cap goes up. If they're being paged too rarely on genuinely broken playbooks, the cap goes down (or new triggers fire — e.g. cap-1 for kind: pipeline items tagged tier-1).
• The override audit log. ~/.gyrum/admin-overrides.log is read by Friday-afternoon review tooling; breaking its shape breaks that review. Same shape as --skip-smoke's log entries (per the existing enforcement gates section).

Alternatives considered

• Always-on dry-run. Rejected — see Context. Doubles cost on every fire, trains operators to ignore output, pays the tax against healthy playbooks. The asymmetry between healthy and failure-prone is the load-bearing observation.
• Threshold by majority-red (3 of 5) instead of any-red. Rejected — under-gates the silent-success class. The cost of one extra dry-run on a recovering playbook is small; the cost of one more silent failure on a recovering playbook is the entire reason for the gate.
• Pass/fail boolean from Claude instead of structured verdict. Rejected — see Context. Throws away the only signal that distinguishes "broken at the harness" from "broken at the diff" from "uncertain". The wrapper needs the distinction to choose between auto-incident, hold-for-operator, and proceed.
• Block all fires on red history (no dry-run, no verdict, no override). Rejected — converts a soft signal into a hard stop that operators will route around (manual ansible-playbook calls outside the wrapper). The gate must remain bypassable with audit; bypassable-without-audit is what we have today.
• Fire-then-verify instead of dry-run-then-fire. Rejected — the silent-success class is precisely the case where post-fire verification reports green. Dry-run catches the failure before any state mutation; post-fire verification catches it after, which is the same outcome as today.
• Build the audit gate (r2 follow-up ADR) as part of this ADR. Considered; rejected on shipping cadence. r1 (this ADR) and r2 (the dry-run-clean playbook audit ADR) are related but separable; r1 establishes the schema field and the gate semantics, r2 establishes the proof that the field's claim is true. Splitting keeps this ADR within target length and lets r2 absorb the docker-mock test infrastructure independently.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

References

• ADR-067 — playbooks-unified-primitive (the front-matter that supports_dry_run joins)
• ADR-068 — playbook-runtime (the runtime that gains the dry-run executor in r3)
• ADR-083 — rule-promotion-engine (same surface → judgment → execution shape)
• ADR-090 — ai-as-scaffold-pipelines-as-runtime (the gate is the pipeline; the verdict hook is the AI scaffold around it)
• warp #281 — pipeline-items + health_check_url (the run history the threshold reads)
• warp #312 — deploy-warp.yaml silent-failure (motivating case)
• warp #313 — implementation epic (r1-r7 phases; this ADR is r1)
• ~/.gyrum/dry-run-verdicts.jsonl — verdict audit log (written by r5 wrapper; read by r6 UI)
• ~/.gyrum/admin-overrides.log — override audit log (shared with --skip-smoke per existing enforcement gates)