ADR-118 — Per-block validation: every action proves itself

Status: Accepted Date: 2026-04-27 Tracker: warp #460 Cross-refs: ADR-083 (defense-in-depth gate stack), ADR-110 (observe-and-file as complement, not replacement)

Context

Today the gate stack catches code-level defects (build, test, coverage, complexity, gosec, staticcheck) but does NOT catch the "module ran clean, reality is broken" class of bug. Every state-changing action is currently trusted on the basis of the executor's exit code: if ansible.builtin.docker_container reports ok, we assume the container is doing what we wanted; if a shell step exits 0, we assume the side effect happened. Both assumptions failed today.

The 2026-04-27 incident. During the warp-02 → warp-03 cutover:

The host ansible role's caddy.yml was re-run on warp-03 and reported ok=10 changed=0.
Reality: the Caddy container was running, but cert acquisition for warp.gyrum.ai was failing silently in the background (LE validates by hitting whatever DNS resolves the hostname to — DNS still pointed at warp-02). The verify-via-hostname gate added in migrate-service-to-existing-host.yaml (the playbook-level analogue of this ADR) caught it pre-cutover, but only because we had retrofitted that single gate after an earlier outage.
Earlier in the same migration, docker compose stop warp failed with a depends_on validation error. The wrapping || true swallowed the non-zero exit; the playbook reported step success while source containers stayed up; row-counts drifted during dump.

Both are the same shape: the executor's success signal does not assert the desired effect. A linter doesn't catch it because the syntax is fine; a test doesn't catch it because there's no test of "did the actual outcome occur on the actual host"; the gate stack doesn't catch it because the gate stack runs at PR time, not at action time.

Same shape across past incidents. Looking at the finding journal, this pattern recurs:

warp #356 (warp-02 cutover, 2026-04-26) — role's caddy-vhost task conditionally skipped, no verify caught it
distill #34 (cert-bearer) — metric endpoint reported 200 but bearer auth misconfigured
ai-research #103 (gosec G703 sweep) — five locations with module-ran-clean but actual taint
This session — postgres-migrate-database sub-playbook reported state: failed but the outer migration step ignored it (CLI exit-code propagation gap)

Each had a "module ran clean, downstream effect missing" gap. Each was discoverable only when something downstream broke. None were caught by the gate stack.

Decision

Every state-changing block (ansible task, playbook step, shell script) MUST be paired with a verification that asserts the desired EFFECT.

The gate is fail-closed at authoring time, not at runtime. You cannot ship a state-changing block without proving its effect.

We pursue the structural fix as a three-layer stack, mirroring the defense-in-depth pattern of ADR-083:

Layer 1 — Schema requirement (primary gate, runtime + validator)

The playbook step schema gains an optional assert: block alongside command::

- id: install_caddy
  command: docker compose up -d caddy
  assert:
    max_seconds: 30
    retries: 6
    delay_seconds: 5
    command: |
      curl -sf -o /dev/null http://127.0.0.1/ -H "Host: warp.gyrum.ai"

The runtime executes command, then runs assert.command with retry-and-timeout; assert non-zero (after retries exhausted) is treated as step failure. gyrum-validate-playbook rejects steps that mutate state without an assert block. State-mutating is determined by command-content match against a maintained pattern list (docker stop|rm|run|compose|exec, hcloud, ansible-playbook, ssh.*sudo, cf-cli, curl.*api.cloudflare, gyrum-fire-playbook). Read-only commands (docker ps, hcloud server describe, curl -sf .../healthz) are allowlisted.

Layer 2 — Ansible-side linter (gate-stack check)

gyrum-pipelines adds a check that scans dark-factory/ansible/roles/*/tasks/*.yml. For each task file, the check requires either:

An adjacent <name>-verify.yml file, OR
An inline # === BEGIN: per-block validation block (the convention used in this ADR's reference retrofits).

The check runs in the pre-push hook + CI. Same shape as the existing structural checks; same fail-closed semantics.

Layer 3 — Shell convention (legacy fallback)

For shell steps still in shell shape inside playbooks (the dominant form today, until phase-4 typed-step executors land), the convention is:

set -e
# 1. ACT
<do the thing>
# 2. ASSERT (mandatory)
<probe the thing>

A linter scans for the # 2. ASSERT marker; missing → fail. This is the bridge until Layer 1's schema-assert can fully replace it.

Why three layers

A single layer leaks. Layer 1 gates schema-shaped steps but doesn't cover ansible roles. Layer 2 gates ansible task files but doesn't cover playbook steps. Layer 3 catches the legacy shell shape that the other two don't see. Together, every state-changing surface in the fleet has at least one gate proving its effect.

Consequences

Positive

The "module ran clean, reality is broken" class of bug becomes impossible to ship. Authoring-time enforcement, not runtime discovery.
Failure surface moves left. Today's incident would have aborted at install time on warp-03 ("caddy verify — answers HTTP on localhost:80" assertion would have failed); we'd never have reached the cutover with broken state.
The runbook becomes shorter. Operators don't need a checklist to verify a deploy worked; the deploy itself certifies its outcome.
Reviewer cognitive load drops. Reviewers stop re-deriving "did this actually do what it claimed"; the assert block is the proof, in the diff.
Composes with ADR-110. Per-block asserts catch deterministic effects; observers catch the long-tail anomalies. Stack, not duplicate.

Negative

Authoring overhead. Every new state-changing block costs ~5-15 lines of assert code. Mitigated by the # === BEGIN: per-block validation snippet pattern + the verify-pattern doc.
Retrofit cost. ~30 ansible task files + ~50 playbook steps fleet-wide. Sequenced via warp #460's "PR 4: fleet retrofit sweep" sub-ticket, prioritised by usage frequency.
Schema migration risk. Existing playbooks lack assert: blocks; the validator must support a grace mode for status: deferred playbooks until each is retrofit. Same shape as the dry-run-on-history grace pattern from ADR-104.
Slower step wallclock. Asserts add 1-30s per step depending on retry config. Mitigated by tight max_seconds defaults and parallel-where-possible asserts inside one step.

Neutral

Existing observers (per ADR-110) are unaffected. They remain the long-tail safety net; per-block asserts are the deterministic primary gate. They are complementary, not redundant.
The gate stack (ADR-083) gains a new check, doesn't restructure. Layer 2 of this ADR is one new pre-push gate added to the existing four-layer model.

Rollout

Tracked in warp #460. Four PRs across three repos:

dark-factory (this PR) — ADR-118 + reference retrofits in host/tasks/caddy.yml and warp/tasks/caddy-vhost.yml. Establishes the convention.
ai-research — runtime + validator schema extension for assert: block. Layer 1 primary gate.
gyrum-pipelines — ansible-side lint check. Layer 2 author-time gate.
dark-factory + ai-research sweep — retrofit the remaining ~30 ansible tasks + ~50 playbook steps. Each retrofit is its own small PR; warp #460 tracks meta-progress.

PR 1 unblocks the others by providing the canonical example. PR 2 + PR 3 land in parallel. PR 4 is a multi-week sweep.

Cross-references

ADR-083 — the defense-in-depth gate stack this extends
ADR-110 — observe-and-file as complementary safety net
warp #460 — implementation tracker
warp #719 (ai-research, MERGED v0.33.44) — cutover redesign, contains the playbook-level analog (verify_dest_serves_via_hostname) of this ADR
warp #498 (dark-factory, MERGED v0.2.40) — host-lifecycle doc rewrite documenting the cutover redesign + lessons
2026-04-27 incident: warp-02 → warp-03 cutover took warp.gyrum.ai down ~5 min; root cause was missing per-block validation in the host caddy install + cert acquisition path