← Decisions operate mode →

Decisions

ADR-022: Grafana Alerting over Prometheus Alertmanager

[ADR-001](001-self-hosted-observability-stack.md) put Grafana + Loki + Prometheus on a self-hosted VPS. Phase 5 needs alerting: a rule evaluator, a router, and contact points. The choice is between Grafana's built-in…

#022

ADR-022: Grafana Alerting over Prometheus Alertmanager

Status: Accepted Date: 2026-04-21

Context

ADR-001 put Grafana + Loki + Prometheus on a self-hosted VPS. Phase 5 needs alerting: a rule evaluator, a router, and contact points. The choice is between Grafana's built-in alerting subsystem (formerly "unified alerting", landed in Grafana 9) and Prometheus Alertmanager, the industry default partner for Prometheus metrics.

Team size today is one human operator plus an agent. The rule volume is projected under 50 day-one (security-tripwires gives 15; availability, infra, SLO give roughly 25 more; 50 is the generous bound). The LogQL alert use-case (tripwires on log patterns) is central to this tranche — security tripwires 1-13 are Loki-based, not Prometheus-based.

Alertmanager natively routes Prometheus alerts but does not evaluate LogQL. To use Alertmanager for LogQL-based alerts we would need Loki's Ruler running alongside, writing to Alertmanager over HTTP. Grafana's alerting handles both in one process.

The CLAUDE.md project principle "simplicity first — the best code is code you didn't write" applies directly: two services (Loki Ruler + Alertmanager) doing what one already does (Grafana's alerting) is 2× the config, 2× the failure surface, 2× the upgrade cadence.

Decision

Use Grafana's built-in alerting as the sole alert evaluator and router. No Prometheus Alertmanager and no Loki Ruler in this stack.

Shape:

  • Rules authored in YAML under infrastructure/alert-rules/*.yml and provisioned via Grafana's provisioning API at deploy time (ADR-023).
  • Contact points and notification policies in YAML under infrastructure/alerting/ (alerting-architecture §contact-points).
  • Both Prometheus and Loki datasources are queryable from the same rule evaluator; rules span metrics and logs without inter-service routing.
  • Silences live in Grafana (UI-managed for dynamic, YAML-provisioned for permanent).

Consequences

  • One alerting surface to learn, run, and debug. The operator understands one UI, one YAML dialect, one failure mode. The mental overhead matters when the rule count is small — a monolithic Grafana is the right complexity level.
  • No multi-tenant isolation for alerts. Grafana Alerting is not per-team. Once we add a second operator or a second org, we migrate to per-org rule namespaces (already supported) or move to Alertmanager for the routing layer. Not a blocker at team-of-one.
  • LogQL and PromQL alerts are siblings. A rule reads both without any inter-service hop. This is the largest concrete win over Alertmanager — security tripwires, which are the majority of this tranche, are LogQL, and Alertmanager cannot evaluate them without Loki Ruler.
  • Upgrades are coupled to Grafana upgrades. A Grafana upgrade that regresses alerting blocks observability dashboards in the same release. This is a real risk; mitigation is the monthly alert-review cadence catching regressions inside 30 days, and pinning Grafana version in the deploy config (ADR-010 pattern).
  • Exit cost is bounded. Grafana alerting YAML is a superset of Prometheus alerting YAML; a yq pass extracts the Prom-native subset. The LogQL rules would need Loki Ruler at migration time, but they're the minority if we later add Alertmanager. The migration path is named in alerting-architecture §migration so we know it's bounded and not a lock-in.
  • No first-class deduplication across instances. Grafana Alerting runs per-instance; a future HA Grafana pair would double-evaluate alerts without Alertmanager-style grouping. We don't run HA today. When we do, Alertmanager becomes the natural target.
  • Paging integration is Slack + email today. Grafana supports PagerDuty contact points; we just don't use one. The path to PagerDuty is a contact-point YAML addition, not a re-architecture.

Alternatives considered

  • Prometheus Alertmanager + Loki Ruler. The canonical pair. Rejected: two services, two configs, two upgrade schedules, two failure modes. Buys us multi-tenancy and HA that we don't need today, in exchange for complexity we'd pay for daily. The migration path is reserved for when the rule volume or team size makes Alertmanager's isolation valuable.
  • Grafana Alerting + external Alertmanager (webhook). Use Grafana to evaluate, Alertmanager to route. Rejected: two routing layers is worse than one. The double-route is tempting when you want Alertmanager's silence semantics on top of Grafana's evaluation, but Grafana's silence layer has the features we need at our scale.
  • Prometheus Alertmanager only (no Loki alerts). Drop the LogQL alert use-case entirely; do everything by counting metrics instead of patterns in logs. Rejected: the security tripwire story requires pattern matching on log lines (e.g. scanner user-agent, [REDACTED:*] spike). A log_lines_matching_pattern_total metric exists but re-implementing every tripwire's LogQL as a counter is duplicated surface. The tripwires read the logs once, directly.
  • Third-party hosted service (Datadog / New Relic / Grafana Cloud Alerting). Rejected by the same rationale as ADR-001: Cloud-cost scales with log volume, which scales with deploy cadence, which is our product. Self-hosted on the observability VPS is a flat monthly cost.

Supersedes: none Superseded by: