← Decisions operate mode →

Decisions

ADR-015: Separate per-service (Tier-0) and fleet-wide (Tier-1) dashboards

On-call triage and platform health are different questions asked by different people at different times.

#015

ADR-015: Separate per-service (Tier-0) and fleet-wide (Tier-1) dashboards

Status: Accepted Date: 2026-04-21

Context

On-call triage and platform health are different questions asked by different people at different times.

  • The service owner asks per-service questions: "is distill healthy?", "what error is my user hitting?", "did my deploy break anything?".
  • The platform operator / on-call engineer asks fleet-wide questions: "what is on fire right now?", "is the log pipeline alive?", "is the redactor still catching secrets?".

A service-owner dashboard cluttered with platform signals slows service triage. A platform dashboard cluttered with per-service detail hides the "which service" question the platform operator is asking.

The cardinality also differs. Per-service dashboards scale with service count (one set of Tier-0 dashboards per service, instantiated by the scaffold). Platform dashboards are a fixed set, hand-maintained, one copy fleet-wide. Treating them as the same artefact would require the scaffold to know about "platform mode" as a special case, which is the same kind of coupling we avoided in ADR-013.

Decision

Grafana dashboards are partitioned into two tiers:

  • Tier-0 (service): Per-service, automatically generated. One set of templates in infrastructure/grafana-templates/service-*.json.tmpl; the scaffold CLI materialises them into observability/grafana/<service>/ per service. Tagged ["gyrum", "tier-0"]. UID suffix is the service slug.
  • Tier-1 (platform): Fleet-wide, hand-maintained. Single copies in infrastructure/grafana-templates/platform-*.json. Not templated. Tagged ["gyrum", "tier-1"]. UID is the platform-type slug, no service suffix.

A dashboard belongs to Tier-0 if its question names a service (or is answerable only about one service at a time). A dashboard belongs to Tier-1 if its question is about the fleet, or about the observability substrate itself (log pipeline health, redaction events, probe freshness).

The scaffold CLI only touches Tier-0 by default. gyrum-dashboard-scaffold --platform is the admin path for instantiating Tier-1; it copies without substitution.

Consequences

  • Service-owner triage is uncluttered. The Tier-0 set (service-overview, service-errors, service-resources, service-canary-compare, service-deploy-health) covers every per-service question. The service owner never needs to filter out fleet noise.
  • Platform operator has a single coherent surface. platform-overview lists every service; platform-incident is the 3am view; platform-redaction-events and platform-probe-freshness cover the substrate. An on-call engineer can bookmark six dashboards and answer every fleet-level question from them.
  • Duplication is allowed by design. service-overview for distill shows distill's error rate; platform-overview shows every service's error rate. Same query, different scope. The duplication is not waste — the two dashboards answer different questions (service vs fleet) and both are valuable.
  • The scaffold stays simple. Scaffold logic reads Tier-0 templates from service-*.json.tmpl, substitutes tokens, writes to the service repo. It does not need a "platform mode" — the --platform flag is a separate path that just copies files. The common case (a service developer running the scaffold) never touches Tier-1.
  • Per-tier authoring rules. Tier-0 templates ship in the Tier-0 set and change on a different cadence than Tier-1 platform dashboards. A new Tier-0 template requires editing the scaffold's default list; a new Tier-1 dashboard is a file add in the templates directory. Different cadences, different reviews.
  • Tag-based Grafana folders. Grafana's folder organisation leans on the tags field; tier-0 and tier-1 drive folder placement consistently across the fleet. See the scaffold output behaviour.
  • Tier-0 templates and Tier-1 platform dashboards don't cross-reference each other's UIDs. A platform dashboard that deep-linked into service-overview-distill would break for any service not yet scaffolded. Instead, Tier-1 dashboards show fleet rollups; the engineer clicks through to the specific service dashboard via a Grafana folder search.

Alternatives considered

  • Single tier: everything is per-service, platform view is a Grafana variable. Rejected: "fleet-wide" isn't a service-filter value, and force-fitting it into a template variable hides the different author/cadence story behind a dropdown. Also, platform-incident's four-quadrant layout is materially different from a per-service overview; trying to make one template answer both is a worse version of both.
  • Single tier: only platform dashboards, filter by service in every query. Rejected: per-service dashboards are how service owners reason about their service, and every filter-by-service query is a worse version of a dedicated dashboard for that service. Also, the per-service drill-down story (pasting a user_id or video_id to reconstruct an incident) needs per-service templating for the drill-key variable — not something a fleet dashboard does well.
  • Three tiers: service, platform, infra. Separate infra (VPS-level) from platform (observability). Rejected as premature: we have one platform-infra dashboard; splitting a tier for one dashboard is structure-for-structure's-sake. Revisit if the platform-infra bucket grows past four or five dashboards.
  • Per-org tier. Useful if we multiplexed many customers' services on shared Grafana. Rejected: we don't, and won't for the foreseeable future. Single-org keeps the model simple.

Supersedes: none Superseded by: