ADR-167: Structured failure event schema — the wire format the self-healing fleet observes

Status: Proposed Date: 2026-05-07

Related: warp#1782 (parent EPIC: self-healing fleet), warp#1783 (amendment: self-healing rides the existing agent-executor queue), warp#1735 / ADR-164 (ticket category axis — the categories this schema's auto-filer consumes), warp#1290 (warp audit log — events land here), warp#1779 (today's atomic-deploy regex bug — the canonical motivator), warp#1788 (today's stale-clone bug — would emit event.tool=playbook-executor), warp#1789 (today's GH_ADMIN_TOKEN-stale bug — would emit event.tool=deploy-project), warp#1790 (today's postgres-password-sync gate didn't fire — would emit event.tool=postgres-password-sync), ADR-107 (precedent: gates as playbook primitives — same intuition: lift implicit observation to first-class object), ADR-110 (precedent: observe-and-file dominant pattern — the consumer side of this schema), ADR-126 (precedent: contract ADRs land before consumer code)

TL;DR

Every gyrum-* tool, ansible wrapper, and CI hook that can fail emits a structured JSON event when it fails. The event has six required fields (tool, context, step, error_signature, input_hash, ts) plus four optional ones (severity, host, run_id, repo, diagnosis_hint, operator_run), lands on warp's event log via a single POST endpoint, and is consumed by a single pattern detector that decides whether to file a follow-up warp ticket per ADR-164. This ADR locks the schema so emitters and consumers can ship in parallel without fighting over the wire format.

Context

Today's session (2026-05-07) logged ten distinct break-and-fail events. Every one followed the same five-step shape: symptom existed → detector COULD have seen it → diagnosis was derivable from the failure signature → fix was identifiable → action required operator-time to file a ticket and dispatch the swarm. The atomic-deploy regex bug (warp#1779) is the canonical example: operator fired deploy-project ten times in thirty minutes, every fire failing at the same step with the same error string, before noticing the streak. A pattern detector observing the event stream would have caught it on fire #2.

The detector cannot exist without something to detect. Today's tools do not emit structured events — they print human-readable error text to stderr and exit non-zero. That output is correlated by operator-eyeball, not by signature. Three problems follow:

No correlation primitive. "Same step failed ten times in thirty minutes" requires a stable identifier per step + a timestamp + a same-input marker. Stderr text correlation is fragile; locale changes, tool-version bumps, and partial-line buffering all break it. The fleet has hit at least three "we thought we were detecting X but actually our regex matched a different log line" outages this quarter.
No emitter contract. Every tool today logs whatever it wants. gyrum-review-pr writes to gyrum-review-findings.json; gyrum-complete-pr writes to ~/.gyrum/admin-overrides.log; deploy-project ansible task lines vary per role; release.yml writes to GHA's annotation surface; warp's executor writes nothing. The detector would need eleven different parsers, each fragile to the tool's next output change.
No durable substrate. Stderr is not durable. Today's mitigation is "scroll back through terminal history, hope the relevant lines weren't clipped, hope the tmux pane wasn't recycled." A detector that runs on a different host than the failing tool sees nothing.

warp#1782's premise — "the system fixes its own bugs" — depends on solving these three problems. The premise has no foothold without a single wire format every tool emits and a single substrate the detector reads.

warp#1783 (the amendment) clarified that the substrate already exists in shape: warp's existing audit log per warp#1290 is durable, queryable, and already integrated with the agent-executor queue. The events land on warp; the detector reads warp; the auto-filed tickets ride the existing queue. The contract that has to land first is the wire format — what every emitter writes, what the detector reads.

This ADR locks that wire format.

Decision

Every gyrum- tool, ansible wrapper, and CI hook that can fail emits a failure_event to warp's event log on failure. The event is JSON, has the six required fields named below, posts to a single endpoint (POST /api/v1/events), and is consumed by exactly one detector (built in Phase 2 per warp#1782) that correlates events into ticket-shaped findings per ADR-164.*

The shape has six parts: the wire schema, the emission contract, the substrate (where events land), the consumer contract, the dedupe + correlation primitives, and the migration path from today's stderr-printing tools.

1. Wire schema

A failure event is a single JSON object. Required fields are emitted on every event; optional fields are emitted when the emitter has the data.

{
  "tool":            "deploy-project",
  "context":         "warp",
  "step":            "fetch-ghcr-versions",
  "error_signature": "regex-no-match:ghcr-tag-list",
  "input_hash":      "sha256:7d8c…",
  "ts":              "2026-05-07T12:46:29.728749Z",

  "severity":        "error",
  "host":            "jon-laptop",
  "run_id":          "01HXYZW3-…",
  "repo":            "gyrum-labs/warp",
  "diagnosis_hint":  "atomic-deploy regex changed in PR#NNN",
  "operator_run":    true
}

Required fields (six):

tool — the emitting tool's stable identifier. Closed enum at the wire edge: gyrum-start-work | gyrum-review-pr | gyrum-complete-pr | gyrum-recover | deploy-project | playbook-executor | release-workflow | warp-add | warp-claim | warp-complete | postgres-password-sync | provision-host | factory-cli | <future>. New emitters require a one-line PR adding the value to the validator's allowlist; the closed enum is what makes "list emitters that have not reported in 24h" a tractable health check.
context — what the tool was operating on. Free-form short string; conventionally a project slug, repo name, or host name. Examples: warp, ai-frontend, gyrum-ci-runner-1, factory-cli. Pairs with tool to form the correlation key for "same-step-failing-on-same-target" patterns.
step — the granular step that failed. Free-form short string; conventionally a hyphenated slug. Examples: fetch-ghcr-versions, apply-migration, chown-checkout, vault-decrypt. The detector groups events by (tool, step) first; this is the highest-cardinality correlation primitive.
error_signature — a stable, non-PII slug derived from the underlying error class. Examples: regex-no-match:ghcr-tag-list, http-403:insufficient-scope, command-not-found:falco, vault-undefined-var:ghcr_token, connection-refused:postgres. The signature is what the detector keys on for repeat detection; it MUST be stable across runs of the same root cause and MUST NOT include user-specific paths, secrets, or timestamps. Convention: <class>:<resource> where class is one of regex-no-match | http-<code> | command-not-found | undefined-var | connection-<state> | timeout | exit-<code> | parse-error | assertion-failed | network-<state>.
input_hash — sha256:<hex> of the canonical input the tool was given. Used to recognise "same step failing on same input" — the strongest correlation signal for "a real bug, not just a flake". For deploy-project: hash of the project manifest + target version. For gyrum-review-pr: hash of the PR diff + base SHA. For ansible: hash of the playbook + inventory + extra-vars. The hash is computed AFTER redacting secrets per the existing redaction list (*token*, *password*, *secret*).
ts — ISO 8601 UTC timestamp with microsecond precision (2026-05-07T12:46:29.728749Z). The detector's time-window primitives (N consecutive failures in M minutes) key on this.

Optional fields (five):

severity — info | warn | error | critical. Defaults to error when absent. critical is reserved for "service is currently unhealthy" (per ADR-164's incident category) and triggers a different debounce in the detector. warn is reserved for "soft fail, retry succeeded but worth observing" — emits but doesn't fire pattern detection by default.
host — the hostname the emitter ran on. Used to distinguish "this host is broken" from "this step is broken everywhere". Defaults to os.Hostname() when absent at emit time.
run_id — the parent run identifier when the failure happened inside a multi-step run (a playbook, a CI workflow, a swarm session). Lets the detector correlate "this entire run failed" vs "one step inside an otherwise-healthy run failed".
repo — the repo the failure pertains to. When tool=gyrum-review-pr, this is the PR's repo; when tool=deploy-project, this is the project's source repo. Used by the detector to scope auto-filed tickets to the right repo per ADR-164's repo field.
diagnosis_hint — short free-form string the emitter writes when it can name the likely cause but doesn't want to claim certainty. Consumed by the auto-filer as the seed of the ticket's diagnosis section. Empty when absent.
operator_run — boolean. true when the failing run was triggered by an operator command (interactive shell, manual workflow_dispatch); false when it was triggered by a scheduled job, a webhook, or a recursive agent fire. Lets the detector treat "operator-eyeballed runs that still failed" with higher signal weight than "background flakes".

Forbidden fields: secrets, tokens, raw stderr (use error_signature instead), user PII, full file contents (use input_hash instead). Validator at the wire edge rejects events containing common secret-pattern matches (-----BEGIN, ghp_*, xoxb-*, etc.) — the same redaction list used by the existing audit log per warp#1290.

2. Emission contract

Tools emit at the moment of failure, before propagating the non-zero exit. The emit is best-effort and non-blocking from the failing tool's perspective: a failed POST to the events endpoint is logged but does not change the tool's exit code. Three reasons:

The tool already failed. Wrapping the failure in another failure (because the events endpoint is down) does not improve operator experience.
The detector is the loop-closer, not the operator. A missed event means the detector misses one signal; the operator still sees the original failure on stderr.
The emit MUST NOT race the exit. The tool exits as soon as the POST is dispatched (with a 2-second hard timeout); a slow events endpoint cannot block operator workflow.

Each emitter follows a thin shared library (gyrum-go/failure-event) so the wire format is enforced in one place. The library exposes one function:

// Emit posts a failure_event to warp's event log. Best-effort, non-blocking.
// Returns nil on dispatch (whether or not the POST succeeded); errors only
// for invalid event shape (caller bug, not infra failure).
func Emit(ctx context.Context, ev FailureEvent) error

For non-Go emitters (ansible, bash, GHA workflow steps), the same contract is exposed via a single CLI shim: gyrum-emit-failure --tool deploy-project --context warp --step fetch-ghcr-versions --error-signature regex-no-match:ghcr-tag-list --input-hash sha256:7d8c…. The shim builds the event JSON and POSTs identically to the Go library.

The validator at the wire edge (warp's POST /api/v1/events handler) enforces the closed tool enum, the error_signature format, the timestamp format, and the secret-pattern redaction. Invalid events return 400 with a self-describing error; the emitter logs the rejection to its own stderr but does not retry — a malformed event is a caller bug, not an infrastructure outage.

3. Substrate

Events land on warp's existing event log per warp#1290. The events table is a separate table from warp_items (additive migration), keyed on (tool, step, ts) for the detector's primary access pattern. Retention is 90 days by default (configurable per the existing audit log retention contract); after 90 days, events are aggregated into a daily summary row and the raw rows are pruned.

The events endpoint is POST /api/v1/events; the read surface is GET /api/v1/events?tool=…&step=…&since=…&until=…. Authentication: every emitter holds an events:write scoped token (the existing devtools provisioning code adds one to ~/.config/gyrum/control-plane.env alongside WARP_TOKEN). The token is wire-edge-validated; an unauthenticated POST returns 401, which the emitter logs but does not block on.

The events table schema (additive migration in warp):

CREATE TABLE warp_events (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tool            TEXT NOT NULL,
  context         TEXT NOT NULL,
  step            TEXT NOT NULL,
  error_signature TEXT NOT NULL,
  input_hash      TEXT NOT NULL,
  ts              TIMESTAMPTZ NOT NULL,

  severity        TEXT NOT NULL DEFAULT 'error',
  host            TEXT,
  run_id          TEXT,
  repo            TEXT,
  diagnosis_hint  TEXT,
  operator_run    BOOLEAN,

  raw             JSONB NOT NULL,
  inserted_at     TIMESTAMPTZ NOT NULL DEFAULT now(),

  CONSTRAINT warp_events_severity_check
    CHECK (severity IN ('info','warn','error','critical'))
);

CREATE INDEX warp_events_tool_step_ts_idx ON warp_events (tool, step, ts DESC);
CREATE INDEX warp_events_signature_idx    ON warp_events (error_signature, ts DESC);
CREATE INDEX warp_events_repo_ts_idx      ON warp_events (repo, ts DESC) WHERE repo IS NOT NULL;

The raw JSONB column preserves the entire emitted event; the typed columns are denormalised for the detector's access patterns. The detector queries the typed columns; the audit reader queries raw.

4. Consumer contract

Exactly one consumer reads the events stream: the failure pattern detector built in warp#1782 Phase 2. The detector polls GET /api/v1/events?since=<last-seen-cursor> on a fixed cadence (default 30s), advances its cursor on each batch, and runs each event through its pattern primitives:

N-consecutive-same-signature — count(events where (tool, step, error_signature, input_hash) = X in last M minutes) >= N. Default: N=3, M=10. The atomic-deploy-regex shape (warp#1779) trips this on fire #3.
Sudden-new-signature — error_signature was not observed in the previous 7 days but has now been observed twice in the last 5 minutes. Catches "a fresh-deployed bug just started shipping."
Systemic-across-context — same (tool, step, error_signature) observed across ≥3 distinct context values in the last hour. Catches "this step broke for everyone, not just one project."
Stale-success — tool=release-workflow severity=info saw a success event for (repo, ref) more than M minutes ago, but the live SHA still doesn't match. Composes with warp#1781. Default M=60.
Drift — emitter-specific (e.g. git-ownership-drift from warp#1715, vault-undefined-var from warp#1774).

On match, the detector files a warp ticket per ADR-164's category axis (the auto-diagnosis library entry per ADR-168 chooses the category and the proposed fix). The detector does not dispatch agents; per warp#1783 the existing agent-executor queue handles dispatch via the normal warp-ready poll loop.

There is exactly one detector. Multiple detectors reading the same events would race on cursor advancement and produce duplicate tickets; the single-consumer contract is the structural guard. If a future workload genuinely needs to consume events too (e.g. a metrics dashboard), it reads via GET /api/v1/events directly with its own cursor — but does not file tickets. The "files tickets" surface is monopolised by the detector.

5. Dedupe + correlation primitives

The schema provides three correlation keys, in order of strength:

(tool, step, error_signature, input_hash) — strongest. "Same tool, same step, same error, same input." If this group fires N≥3 in M minutes, the detector files a ticket with high confidence the diagnosis is single-rooted.
(tool, step, error_signature) — medium. "Same tool, same step, same error class — but maybe across different inputs." Used for the sudden-new-signature and systemic-across-context primitives.
error_signature — weakest. "Same error class, but tool/step might vary." Used for cross-emitter correlations (e.g. http-403:insufficient-scope happening across deploy-project AND gyrum-review-pr AND release-workflow simultaneously suggests a global PAT problem, not a per-tool bug).

Dedupe is the detector's responsibility, not the emitter's. Emitters fire on every failure; the detector's dedupe-window primitive (1h sliding window per (tool, step, error_signature) per ADR-168 default) is what prevents auto-filed-ticket storms.

6. Migration path

Today's tools print to stderr and exit non-zero. The migration path is additive and per-tool:

Add the emit at the failure boundary. For Go tools, wrap the os.Exit(1) call site with failureevent.Emit(...) then exit. For ansible, add a failed_when handler that calls gyrum-emit-failure before propagating. For bash wrappers, wrap the exit 1 with the shim.
Backfill error_signature per failure mode. The first emit shipping a tool sets the closed list of signatures that tool emits; future PRs adding new failure modes add new signatures. The detector tolerates unknown signatures (it doesn't reject the event, just doesn't have a library entry to match it against).
Land per-tool in priority order. warp#1782 Phase 1 sequences this: (a) gyrum-review-pr, gyrum-complete-pr, gyrum-start-work first; (b) ansible wrapper second; (c) deploy-project third (composes with (b)); (d) release.yml + workflows fourth via webhook to the events endpoint.
The emit is opt-in per tool. A tool that has not yet been instrumented just doesn't emit; the detector sees nothing for that tool's failures, same as today. There is no "all tools must emit before the detector runs" cliff; the detector ships incremental coverage as emitters land.

The emit landing in a tool is also the structural enforcement that the failure mode is named — adding a failureevent.Emit call at a previously-untyped exit boundary forces the engineer to name the error_signature, which forces them to think about what class of failure it is. This is the same shape as ADR-107's "lift implicit policy to first-class object."

Consequences

Wins

One wire format, eleven (today) emitters, one consumer. The fragility of "the detector parses eleven different stderr formats" disappears. New emitters add to a closed enum; new consumers (none planned) would query a typed API.
Durable substrate. Events live in warp's existing audit log infrastructure; a detector running on a different host sees the same stream. Today's "scroll back terminal history" mitigation ends.
Stable correlation primitives. (tool, step, error_signature, input_hash) is a stable join key across runs, hosts, and tool versions. The atomic-deploy-regex shape (warp#1779) becomes a 5-line library entry per ADR-168, not a fragile log-grep.
Closed-enum tool list. "Which emitters have not reported in 24h" becomes a queryable health check; the fleet finds dead emitters before the operator does.
Operator gate preserved. Per warp#1783, events file tickets via the existing queue. The operator approves / denies in the existing surface; this ADR adds no new approval path.
Per-tool incremental rollout. No big-bang migration; each tool gets instrumented in its own PR, the detector picks up coverage as PRs land.

Costs

One additional warp endpoint + one table. The migration is additive and reuses the existing audit-log retention and indexing patterns; the marginal operational cost is small but non-zero (events will be the highest-cardinality table on warp's Postgres within 30 days at projected emit volumes).
Every emitter holds an events:write token. That's eleven additional credentials to provision, rotate, and audit. Mitigated by reusing the existing token-provisioning shape (gyrum-setup adds the events token alongside WARP_TOKEN); rotation rides the existing rotation schedule.
Closed tool enum is opinionated. A new emitter requires a one-line PR. This is the cost; the benefit is the queryable "emitters not reporting" health check. Without the closed list, that check would degrade into "free-form string matching" with the same parser-fragility this ADR exists to retire.
Best-effort emit means events can be lost. A network blip between the failing tool and the events endpoint loses that event. Mitigated by: (a) the operator still sees the original failure on stderr, so nothing is silently dropped from the operator's point of view; (b) the detector's pattern primitives are time-windowed, so a single lost event doesn't break detection — N=3 in M=10 still fires when fires #4 / #5 / #6 emit successfully; (c) the inserted_at column lets ops queries find "events that arrived but with skewed clocks", which is the audit-trail half of "events that didn't arrive at all".

Risks

Emit-storm on a flaky upstream. A tool that fires every second on a flapping network would emit several thousand events per hour. Mitigated by the detector's debounce (per ADR-168), not by emitter-side rate limiting — emitting is cheap, parsing is the expensive layer; throttling at emit time would break correlation primitives.
Schema drift. A future field added to the wire format must be additive (existing consumers ignore unknown fields, existing emitters omit unknown fields). The shared library is the structural enforcement: schema changes flow through one PR that bumps the library version; emitters using older versions of the library still produce valid events, just without the new field.
Detector single-point-of-failure. Per the contract, exactly one detector reads the stream. If the detector is down, no patterns fire. Mitigated by: (a) the detector is a small stateless poll-and-match loop that ships in warp#1782 Phase 2 with the same heartbeat / restart shape as warp's own service; (b) when the detector restarts, it advances from its persisted cursor, so it catches up on backlogged events; (c) the detector's own failures emit failure_events (recursive dogfood), which would surface a dead detector as a tool=detector shape in the next observer's view.
Forgotten error signatures. A tool that emits with a fresh-novel error_signature not in any library entry produces a no-op (the detector logs but doesn't file a ticket). This is the correct behaviour — first-occurrence-of-a-new-shape MUST NOT auto-fire per warp#1782 — but the operator should be able to see "we observed this but had no library entry" so curation can catch up. The detector exposes a GET /api/v1/events/uncovered endpoint listing signatures with no library match in the last 7 days; library curation per ADR-168 reads this surface.

Mitigations

Wire-edge validator rejects malformed events with 400 + a self-describing error. The closed enum, the timestamp format, the secret-pattern redaction, and the error_signature format are all enforced server-side.
Best-effort + non-blocking emit means the events path can never make a failing tool's failure mode worse than it already is. Even a totally-down events endpoint produces "missed signal" rather than "operator workflow blocked."
Per-tool token scoping (events:write only) means a leaked emitter token cannot read other tickets, claim work, or PATCH item bodies. Same shape as the existing per-scope token model on warp.
Recursive dogfood — the detector's own failures emit events. The system observes itself observing.

Alternatives considered

Free-form structured logging (e.g. emit JSON to stderr, scrape via fluentd / vector)

Have every tool emit a JSON line to stderr; collect via a sidecar log shipper; index in Elasticsearch / Loki. Rejected on three grounds: (a) the fleet does not run a log aggregator and is not committed to standing one up — the operational cost would dwarf the events table by an order of magnitude; (b) the correlation key would be "lines of JSON" with no enforced schema, reproducing the parser-fragility this ADR exists to retire; (c) sidecar log shippers have their own failure modes that are out-of-band of the failing tool's own observability — the detector's "events I haven't seen for X tools in 24h" health check would not be tractable.

OpenTelemetry traces / metrics

Emit per-failure as an OTel span with structured attributes; consume via an OTel collector. Rejected: OTel spans are designed for distributed tracing of a SINGLE multi-step request, not for cross-fleet correlation of independent failures. The data model (parent span → child spans → context propagation) imposes structure the detector does not need and lacks the closed-enum + dedupe primitives the detector does need. The fleet could adopt OTel later for an unrelated reason and still keep this schema; the two are not in conflict.

Direct ticket filing from each tool (skip the events layer)

Have every tool that detects N consecutive failures of itself just file a warp ticket directly. Rejected: this collapses the detector into N independent per-tool implementations, each reinventing the dedupe / debounce / correlation primitives. Worse, cross-tool correlations (http-403 happening across deploy-project AND gyrum-review-pr AND release-workflow simultaneously) become impossible because no single tool sees the cross-tool stream. The events layer is the seam that makes one detector possible.

Reuse warp_items as the event log (one table for everything)

Every failure becomes a low-priority warp_item with a kind: event tag; the detector queries warp_items directly. Rejected: warp_items has lifecycle semantics (ready / in_progress / done / blocked) that don't apply to events; events would need their own status sub-state to avoid being mistakenly claimed by agents. The retention shapes also differ (items are durable; events age out at 90d). Forcing the event stream through the items table would require special-casing both the kanban and the agent-pickup loop. A separate table is the cleaner seam.

Emit synchronously and block on success

Have every tool wait for the events endpoint to confirm receipt before exiting. Rejected: a slow events endpoint would block operator workflow on the failure path — the worst possible time to introduce additional latency. The best-effort + 2s-hard-timeout contract is the structural protection for "the emit must never make the original failure worse."

Cross-references

Operator framing 2026-05-07: "Things break and fail, how can we catch the not deploying or working issue structurally so you can raise a fix automatically, dogfooding ultrathink."
ADR-107 — gates as playbook primitives. Same structural intuition: lift "implicit observation" to "first-class object" so the detector can quote it.
ADR-110 — observe-and-file dominant pattern. The consumer side of this schema runs in the same shape: read events, decide on anomaly, file a warp ticket.
ADR-126 — durable engineering content in git. Contract ADRs land before consumer code; this ADR is the contract that ADR-168 (auto-diagnosis library shape) and ADR-169 (auto-swarm dispatch + auto-merge gating) compose against.
ADR-164 — ticket category axis. The auto-filer uses category: warning | approval-needed | incident per the library entry's approval_mode per ADR-168.
warp#1782 — parent EPIC; Phase 0 sub-ticket S0.1 is this ADR.
warp#1783 — amendment: self-healing rides the existing agent-executor queue; this ADR's events feed the queue, not a new dispatcher.
warp#1779 — today's regex bug, the canonical motivator. The first library entry per ADR-168 (atomic-deploy-regex-404) keys on this schema's (tool=deploy-project, step=fetch-ghcr-versions, error_signature=regex-no-match:ghcr-tag-list).
warp#1290 — warp audit log; provides the substrate this schema's events land on.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one