ADR-025: SLO-based burn-rate alerts — multi-window multi-burn, not threshold

Status: Accepted Date: 2026-04-21

Context

User-visible contracts (latency, availability, success rate) are the concerns we most want to alert on correctly. The two failure modes of alerting on them:

• Threshold too tight — fires on transient blips (a single slow request in a 5-minute window of otherwise-healthy traffic). On-call fatigue.
• Threshold too loose — misses the slow burn (a steady 1%-too-high error rate that consumes the month's error budget by week 3). No alert until users have been unhappy for days.

The SRE literature (Google's SRE Workbook chapter 5, then followed by almost every SRE team that has written publicly) has converged on multi-window multi-burn-rate alerting:

• Define an SLO as a target over a window (e.g. "99.9% success over 30 days").
• Compute the error budget (1 - target).
• Alert at different burn-rate thresholds over different time windows. The combination catches both fast and slow burns.

The canonical pair:

• Fast alert: 1h window, burn rate × 14.4 → fires when the 1h error rate is consuming the budget at 14.4× the sustainable rate. At that rate, the budget is gone in 30d / 14.4 ≈ 2d. Alert fires quickly (within minutes) on active outages.
• Slow alert: 6h window, burn rate × 6 → fires when the 6h error rate is consuming at 6× the sustainable rate. Catches drift that fast-burn misses. Alert fires within an hour on slow degradation.

The constants (14.4, 6) are from the SRE Workbook's "Alerting on SLOs" chapter and are calibrated to a 99.9% target over 30 days. They change with target and window; the shape is the same.

This ADR captures the shape of how we alert on SLOs, not the specific SLOs themselves. Specific SLOs belong in infrastructure/slos/*.yml (the SLO catalogue), and the burn-rate rules are generated from those.

Decision

All user-visible SLOs are alerted with a multi-window multi-burn-rate pair: a fast-burn rule and a slow-burn rule, pointing at the same runbook with the same severity.

Shape:

• SLO definitions live under infrastructure/slos/<service>-<sli>.yml. Each file declares: target, window, error budget, and the query that produces the current error rate.
• A generator (scripts/generate-slo-alerts.sh, Phase 2+ devtool) reads the SLOs and emits the alert rule pair into infrastructure/alert-rules/slo-generated.yml. The generated file is committed — the SLO files are the source of truth, and the generator output is the compilation target, like go generate output.
• Day-one: two SLOs hand-written (no generator yet). Both land in this tranche as a worked example:
  • distill-report-generation-success — 99.5% success over 7 days. Fast window 30m, slow window 6h.
  • guardian-probe-freshness — 99.9% probes within 10m of each other over 24 hours. Fast window 15m, slow window 2h.
• Severity: both fast and slow map to page. The reason a slow-burn alert fires is also budget-consuming; if the operator ignores it, the problem becomes a fast-burn. Treating slow-burn as lower severity trains the operator to ignore it.
• Runbook: one runbook per SLO, not per alert rule. The fast and slow alerts point at the same file. The runbook's triage section handles "am I looking at a fast or slow burn?" as a branch.

Consequences

• Users see incidents sooner. Fast-burn fires before manual complaints typically arrive. Slow-burn fires before the month's budget is blown. The combination catches what threshold-based alerts miss.
• Fewer false positives. The multi-window pattern requires consistent badness across both windows to maintain the fire. A single slow request doesn't move the 6h window; a 10-second connection blip doesn't maintain the 1h window. False-positive rate is measurably lower than threshold alerts in published SRE data.
• The SLO file is the documentation. A reader looking at infrastructure/slos/distill-report-generation-success.yml sees the target, the window, and the measurement. They don't need to read the alert rule to know what the system is promising. This separation — SLO = intent, alert rule = mechanism — is the single biggest readability win.
• More rules per SLO. Two rules per SLO, not one, means the monthly review (alerting-architecture §alert-review) has twice the volume. Mitigation: group the fast/slow pair in the review UI. Grafana's alert list supports filter-by-label, and SLO-generated rules carry slo=<name> for grouping.
• Tight coupling between SLO and constants. Change the SLO target from 99.9% to 99%, and the burn-rate constants change (6× becomes 0.6× — the alert fires only on 10× the new budget). The generator handles this; hand-written SLOs (day-one) re-derive by hand. Document the math in each SLO file's comments.
• SLOs are new content discipline. We didn't have SLOs before. Adding them means committing to "this is what we promise" — a declarative statement that a reviewer can hold the author to. Some services won't have meaningful SLOs yet; those services use threshold alerts until the SLO is defined. Do not invent SLOs to avoid threshold alerts; define them when they reflect a real user contract.
• Not everything wants burn-rate. Security tripwires (security-tripwires.md) are event-driven: a honeytoken hit is binary, a credential storm is burst-y. Burn-rate math is wrong for these. Threshold alerts remain the mechanism for event-driven cases; SLO-burn is for the contract-tracking cases.

Alternatives considered

• Single-window threshold alerts (the status quo before this ADR). One rule per SLO, error_rate > 0.1% for 15m. Rejected: either fires on blips or misses slow drift. No single threshold catches both ends of the spectrum well. Industry has moved past this for a reason.
• Three-window variant (5m + 1h + 6h). Some teams run three burn-rate windows for fine-grained signal. Rejected at our scale: three rules per SLO is three × the monthly-review load, and the 5m window adds more fires than it adds signal at our traffic volumes. Start with two; add a third per-SLO if a specific SLO needs it.
• Burn-rate alerts only; drop threshold alerts entirely. Rejected: security tripwires and availability binaries (up == 0) are not SLOs — they're events. Trying to express "service is alive" as a burn-rate is contortion.
• Hand-calculated constants per SLO. Never generate; always write by hand. Rejected: the math is mechanical (burn_rate = budget_duration / alert_window), the generator saves errors at scale. For day-one with two SLOs and no generator, hand-writing is fine; Phase 2+ gets the generator.
• Use a dedicated SLO tool (Sloth, OpenSLO). Tools exist that turn an SLO file into a ruleset. Rejected for now: a 50-line bash script (Phase 2+ target) is simpler than adopting a tool, and the SLO file's shape should be something we control so we can evolve it cheaply. If we outgrow the script, Sloth's input format is a superset of what we'd need.

Supersedes: none Superseded by: