ADR-023: Alerts as code, not UI — provisioning API only, UI edits overwritten

Status: Accepted Date: 2026-04-21

Context

Grafana lets alert rules be created and edited from the UI. That path is attractive: click-to-save, immediate effect, no YAML syntax. But a rule edited in the UI:

Leaves no review trail. The change goes live with zero second-opinion.
Diverges silently from the checked-in source. Two sources of truth is one source of confusion.
Disappears at the next deploy.sh run if provisioning is set to overwrite — sometimes days later, when the operator has forgotten the edit ever happened.

ADR-010 established the same principle for dashboards: JSON in the repo is canonical; UI edits are either round-tripped back to YAML or overwritten. The dashboard provisioning cycle has proved that discipline over the last quarter. Alert rules get the same treatment, with an extra-strong justification — alerts are the things that wake people up, and the shape of an alert directly affects how on-call interprets a fire. The review history of "why this rule exists and why its threshold is what it is" is load-bearing at 3am.

The Grafana provisioning API was not available in the first Grafana-alerting release; in current versions (11+) it is stable and supports the same atomic-replace semantics as dashboard provisioning.

Decision

Every alert rule is authored in YAML under infrastructure/alert-rules/*.yml, reviewed in PR, provisioned via the Grafana Alerting provisioning API at deploy time. UI edits are allowed for experimentation but do not persist — every deploy reconciles the UI-state to the repo-state.

Shape:

Source of truth: infrastructure/alert-rules/*.yml (one file per category).
Provisioning: deploy.sh (or a dedicated scripts/provision-alerts.sh invoked from it) POSTs each rule file to Grafana's /api/ruler/grafana/api/v1/rules/<namespace>/<group> endpoint. Atomic per-group replace.
Deletion: deploy.sh --prune-alerts removes any rule in Grafana that is not in the YAML set. This is guarded behind a flag to avoid accidental deletion during partial deploys.
UI edits: permitted, but survive only until the next full provisioning pass (typically < 1 day). Operators needing a temporary change use Grafana's silence feature, not the edit feature. Silences are the UI-level escape valve.
Idempotency via uid (ADR-014): a rule's identity is its UID, not its title. A title rename updates the existing rule; deletion is explicit.

Consequences

Rule changes go through code review. A reviewer sees the threshold change, the label change, the runbook link; they can push back, ask for justification, run the test in Grafana Explore. None of that is possible on UI edits.
Git log is the rule-change history. "Why did this fire at 9 AM today?" is answered by git log -p infrastructure/alert-rules/<file>.yml — not by clicking through Grafana audit logs (which have a 90-day retention).
UI experimentation has a safety net. An operator can tune a threshold in the UI to see how it would fire, then either write the change as a PR or let the next deploy revert. This preserves the ergonomics of the UI without giving it persistence.
Silences are the only permissible UI change. This is intentional: silences are inherently temporary (they have an expires_at), and their YAML provisioning for permanent entries goes through review. The UI-silence-path covers the "during incident response, mute this for an hour" case without compromising the source-of-truth invariant.
Drift detection is the prune flag. deploy.sh --prune-alerts is not run on every deploy (default off); it's run explicitly when the operator wants to sync. The convention is monthly during alert review (see alerting-architecture §alert-review-cadence). Any deleted rule still in Grafana fires a pre-deploy diff and the operator confirms — the diff is human-readable.
Validation runs on the YAML, not on the provisioned state. gyrum-alert-validate (Phase 2 devtool) parses the YAML, checks runbook_url presence, checks UID uniqueness, exercises the expression against a local Grafana. This gatekeeps the PR, so the thing that gets deployed has been checked. The provisioned-state check is the monthly review.
Rollback is a git revert. No additional procedure. The next deploy re-provisions from the reverted state.

Alternatives considered

UI as primary, code as backup. Rules written in Grafana, periodically dumped to YAML via API. Rejected: the backup lags the source, review happens post-hoc ("we already have the rule; please bless it"), and the mental default becomes "edit in UI". The direction of truth matters; making code primary is the only way to make it the default.
Provisioning at every deploy without prune. Keep the source-of-truth invariant but never clean up stale UI-created rules. Rejected: stale rules rot — their runbook links break, their expressions reference columns no longer populated, they fire on dead conditions and train operators to dismiss. Periodic prune is necessary.
Round-trip the UI edits back to YAML. A webhook-driven pipeline that commits UI edits to the repo as a PR. Rejected as premature: the tooling is not free, and the UI-edit pattern it enables ("I edited in UI, a PR appeared") still bypasses the pre-edit review step that is the whole point. Round-tripping makes sense at the dashboard layer (experimentation is the norm); at the alert layer, experimentation is dangerous.
Bulk-disable UI editing. Grafana 11+ supports locking alert rules to provisioning-only. Rejected for now as too strict — operators lose the experimentation affordance entirely. The soft-lock (UI edits don't persist) preserves exploration while keeping the invariant. Revisit if UI-experimentation turns out to be a source of outages rather than learning.
Both: code-primary AND UI-locked. The combination with the alert-lock would prevent confused UI edits altogether. Reserved for when the team grows past one operator and UI-vs-code confusion becomes a real problem.

Supersedes: none Superseded by: