ADR-024: Runbook required per alert — no runbook, no ship

Status: Accepted Date: 2026-04-21

Context

An alert without a runbook is a notification that wakes someone up with a problem they don't know how to solve. Real incidents — and years of other teams' postmortems in industry — consistently show that the minutes spent fumbling for "what does this alert mean, where is the dashboard, who owns this service?" are the difference between a 5-minute incident and a 45-minute one. By 45 minutes, an outage is user-visible; by 45 minutes, people are tweeting.

The team is small and the runbook discipline is fragile under time pressure. On-call rotations with 10 people can afford a runbook-absent minority because the next on-call fills in; on-call rotations with 1-2 people cannot. The discipline has to be enforced, not encouraged.

This ADR establishes the enforcement mechanism alongside the philosophical commitment. Related: ADR-023 (alerts-as-code gives us the review gate); alerting-playbook §step-2.

Decision

Every alert rule must carry an annotations.runbook_url pointing at a file at docs/runbooks/<name>.md that exists, is readable, and follows the runbooks/_template.md structure. Rules without this annotation fail the validator; rules with a broken link fail the validator; the YAML does not provision.

Shape:

• annotations.runbook_url is a required field, not an optional one. The validator (gyrum-alert-validate, Phase 2 devtool) enforces presence.
• The URL must resolve to a file checked into the repo at deploy time. The validator opens the file path and verifies it exists.
• The referenced runbook must have every template section filled in (no {{placeholder}} strings). The validator greps for {{.
• In the PR that adds the rule, a reviewer asks: "would I know what to do at 3am from this?" — a pass is a subjective but concrete check.
• A Phase 1 runbook may be a stub — clearly marked ⚠ DRAFT — populate after first real incident. The stub exists so the link isn't 404; operators flesh out with experience. Stubs still follow the template structure.

Consequences

• The runbook is part of "done". A PR that adds an alert without a runbook is not mergeable. The social cost of merging is high enough that the author writes the runbook first, not last.
• The runbook is the first surface an on-call touches. The path from "Slack notification" to "here's what to check" is one click. The notification body includes the link via the runbook_url annotation.
• Stubs are permitted, placeholders are not. A stub says "I don't know yet — here is the shape and the author's current best guess". A placeholder ({{triage command}}) is the template leaking through. The validator distinguishes: ⚠ DRAFT is OK; {{ in the body is not. This matches how we already handle template markers in dashboards (ADR-013).
• Low-value alerts self-select out. If you cannot write a useful runbook, you probably cannot write a useful alert. The friction is pedagogical — the act of writing the runbook forces clarity about why the alert exists.
• Broken runbook links are a CI failure, not a 3am surprise. Moving a runbook file breaks the alert's annotation at PR time, not at the first fire. The mechanical check (does the file exist at this path?) catches renames before deploy.
• We accept a productivity cost. Writing a runbook takes 15-30 minutes per alert. The 15-alert tranche in this PR is ~5 hours of runbook writing. This is not free. The argument for paying it: we pay it once; we pay the alert's value back every time it fires. An alert without a runbook is a liability; with one, it is an asset.
• Stub runbooks decay without maintenance. A stub that sits as a stub for 6 months is a runbook that wasn't needed — either the alert never fired, or fires were dismissed without updating the stub. The monthly alert review catches this: stubs at review time must either be populated (fires happened, document them) or justified (alert is dormant; here's why we keep it).

Alternatives considered

• Runbook link optional, reviewer discretion. Ship an alert with a link to a generic incident doc if no specific runbook exists. Rejected: "reviewer discretion" is indistinguishable from "reviewer overwhelmed". Post-hoc audits in industry consistently find that discretion is the first thing to slip under pressure. The mechanical enforcement is cheaper than retraining.
• Runbook in the PR description, not in the repo. Include "triage steps" in the PR body. Rejected: PR bodies are not discoverable from the fire. An on-call person in Slack at 2am cannot reasonably click through to GitHub, find the PR that introduced the alert, read the body. The runbook has to live where the fire points it.
• Runbook auto-generated from the rule expression. A templated page with "This fires when X > Y. Run docker logs service to debug." Rejected: generic runbooks are almost worthless. The 50% of the runbook that matters is the "what does this mean in our system?" specifics; generators cannot produce that.
• No runbook for info / warning severity. Runbooks only for page and critical. Rejected: info/warning alerts that someone does look at deserve the same treatment. And warnings often upgrade to pages on subsequent events — the runbook is pre-positioning.
• Runbook hosted in an external knowledge base (Notion, Confluence). Rejected: auth expiration at 2am is a known failure mode; the runbook location must be something the on-call person can reach without re-authenticating. Checked-in to the code repo is readable with the same access they already have.

Supersedes: none Superseded by: