ADR-046: Tier-1 discipline dashboard pack (db / auth / billing / jobs / email)

Status: Accepted Date: 2026-04-21

Context

The Tier-0 pack (ADR-013, ADR-014) answers five versions of one question: is this service up? — overview, errors, resources, canary-compare, deploy-health. Every service gets the full set the first time gyrum-dashboard-scaffold runs against it.

That leaves a second layer of questions unanswered at the template level: how is the business doing? Specifically, once a service grows a database, authentication, billing, background workers, or transactional email — the "Tier-1 disciplines" every real SaaS ends up with — the service-overview dashboard no longer tells the operator whether the subsystem is healthy. On-call rebuilds the same "logins failed / payments succeeded / jobs backing up / db slow" panels in each service's directory, and they drift.

Three forces make this decidable now:

pkg/email (ADR-042..044) has shipped. Products are starting to send transactional mail; the operator question "how did today's password resets actually deliver?" is live today, not theoretical.
Issue dark-factory#261 called out a concrete Tier-0 gap: service-db.json.tmpl had been referenced in ADRs and tutorials but never created, even though observ.WrapDSN is wired in distill.
The scaffold CLI's discovery is already glob-based (service-*.json.tmpl) — new templates drop in without a code change. There is no ongoing cost to adding templates ahead of integrations.

Decision

Ship a five-template Tier-1 discipline pack in infrastructure/grafana-templates/: service-db, service-auth, service-billing, service-jobs, service-email. service-db closes issue #261 as a Tier-0 addition; the remaining four are Tier-1, tagged accordingly.

Each template:

Follows the existing substitution contract — ${SERVICE} only (no new tokens), uid ends in -TEMPLATE, ≤ 12 non-row panels (ADR-012), one-sentence description answering one question (ADR-011).
Carries a header comment naming the metrics it queries and which integration has to land before each panel populates.
Includes at least one Loki-backed panel that is useful today, so a freshly scaffolded dashboard is never entirely empty.

The scaffold CLI picks up the new templates automatically via its existing glob discovery. gyrum-dashboard-scaffold <service> now writes ten dashboards per service instead of five; templates whose backing library is not yet integrated show no-data panels and a header comment explaining what is expected next.

Consequences

Dashboards precede their metrics. Four of the five templates query Prometheus series that do not yet exist. This is deliberate — the dashboard documents the shape the metrics must emit, and reviews for the integration PRs (e.g. pkg/auth, pkg/billing, pkg/jobs) have a concrete target to match. When the integration lands, panels populate without a dashboard edit.
Every service gets every discipline dashboard. A product that does not send email still gets service-email. That is acceptable because an empty dashboard is cheap and the on-call failure mode we want to avoid is "the dashboard didn't exist when I needed it".
Bodies are never queried for email. service-email queries bytes_text, bytes_html, subject, recipient count, and the msg=email.sent / email.send.failed events only. This matches ADR-044 and is enforced at the template level: the dashboard cannot surface body content because it never queries it.
Panel budgets consumed. service-billing and service-jobs sit at the 12-panel ceiling. Future additions require removing an existing panel, not relaxing the ceiling. Candidates to drop first if pressed: billing's Refunds issued (also visible in the audit feed), jobs' Retry count distribution (summary signal, DLQ is the actionable one).
Tier labels are per-template, not per-file-pattern. service-* is no longer synonymous with Tier-0 — two of the new templates (db is Tier-0 because DB health is a table-stakes service question; auth/billing/jobs/email are Tier-1) live in the same file-pattern. The CLI's --list-templates output still groups by file-pattern today; a follow-up can add per-template tier annotation.
Issue #261 closes. The DB dashboard gap is a specific commitment we had not delivered; shipping it here resolves it.

Alternatives considered

Per-service bespoke dashboards. Each product builds its own service-billing dashboard when it gets to billing. Rejected: we already lived this with the pre-Tier-0 world and watched four services drift on what "healthy" meant for the same shared concept — the ADR-008 argument applies here equally.
Skip dashboards until the backing metrics are proven. Wait until pkg/billing lands, then write service-billing.json.tmpl shaped around the metrics it happens to emit. Rejected: the order leads to dashboards that mirror whatever the first caller conveniently registered rather than what the operator needs to see. Template-first lets the dashboard pin the integration contract before it calcifies in code.
Ship only service-db (the requested gap) and defer the four Tier-1 templates. Rejected: the marginal cost of the four extras is small (the substitution contract and CLI pick them up for free), while the benefit is "the template is already there when the product needs it". The non-shipped version of this ADR spends more words justifying the deferral than the templates themselves spend describing the panels.
Extend existing templates instead of adding new ones. Cram the DB panels into service-overview etc. Rejected: the 12-panel ceiling and the one-question-per-dashboard rule (ADR-011/012) cannot absorb 40+ additional panels without either dashboard growing beyond its cognitive budget. The one-file-per-question shape is load-bearing.

Supersedes: none Superseded by: