← Decisions operate mode →

Decisions

ADR-010: Dashboards as raw JSON, not jsonnet / grafonnet / Terraform

Phase 3 of the observability rollout ([`observability-architecture.md`](../observability-architecture.md#whats-not-here-yet)) wires deploy-stage dashboard provisioning — every service gets a working Grafana dashboard…

#010

ADR-010: Dashboards as raw JSON, not jsonnet / grafonnet / Terraform

Status: Accepted Date: 2026-04-21

Context

Phase 3 of the observability rollout (observability-architecture.md) wires deploy-stage dashboard provisioning — every service gets a working Grafana dashboard the first time it deploys. Before that wiring lands, we have to choose how dashboards are authored: the file a PR reviews, the file source control stores, the file the scaffold CLI consumes.

The candidates the community actually uses:

  • Raw Grafana JSON (.json dashboard model exported from the UI).
  • Jsonnet + the grafonnet library — a typed, composable DSL.
  • The Grafana Terraform provider — declare dashboards as grafana_dashboard resources.
  • Grizzly / k8s-style YAML wrapping a JSON dashboard body.

Every option eventually produces the same Grafana dashboard JSON. The question is what a service author writes and what a PR reviewer reads.

Constraints that are non-negotiable:

  • Dashboards must be PR-reviewable by someone who doesn't use the authoring tool daily.
  • The substitution story for per-service Tier-0 dashboards (${SERVICE}, ${DRILL_KEY}, see ADR-014) must be trivially scriptable from devtools — the scaffold CLI is a bash script by design, not a Go program.
  • Grafana's own export format is JSON. Round-tripping from UI → file must be lossless.

Decision

Dashboards are committed as raw Grafana dashboard JSON, with service templates using a .json.tmpl extension and simple ${TOKEN} string substitution. No jsonnet, no Terraform provider, no Grizzly layer.

The authoring loop is: build the dashboard in the Grafana UI, export the JSON model, commit it, strip environment-specific bits, and — for service templates — replace service-specific strings with ${SERVICE} / ${DRILL_KEY} tokens. A reviewer reads the same JSON Grafana renders.

Consequences

  • Zero build step between file and dashboard. A jq . or cat is enough to read it. A reviewer looking at a PR diff sees the final shape, not a DSL that renders to the final shape.
  • Substitution is a 20-line bash loop. gyrum-dashboard-scaffold does sed 's/${SERVICE}/<name>/g' (conceptually). A jsonnet pipeline would require a Go or jsonnet runtime at scaffold time, which pulls in a toolchain services do not otherwise need.
  • Composition is manual. We do not have a commonPanels() function we can reuse across templates. If two templates share a panel, we copy-paste. Acceptable trade for a fleet of ~11 dashboards; would reconsider at 50+.
  • UI-authored dashboards come home cleanly. The round-trip (edit in UI → export → commit) is what Grafana supports natively. A jsonnet pipeline adds a lossy translation step at the UI→code boundary.
  • PR diffs are large for layout changes. Moving a panel two grid cells is a handful of line changes in the JSON gridPos block. Acceptable: layout changes are uncommon and the diff still fits in a PR body.
  • We give up type-checking. A typo in a PromQL metric name inside a grafonnet build would (arguably) be caught at build time; in raw JSON it is caught at render time when Grafana shows "no data". We accept this: the scaffold CLI's --validate mode catches malformed JSON and missing UIDs, and Phase 3's deploy-stage provisioning will fail loudly if the POST to Grafana 400s.
  • Upgrading Grafana is safer. Grafana ships compatibility for older dashboard schemaVersions; jsonnet/grafonnet libraries lag the upstream schema by weeks. A raw-JSON dashboard keeps working across Grafana upgrades without library PRs.

Alternatives considered

  • Jsonnet + grafonnet. The canonical "code your dashboards" approach. Rejected: the DSL is one more thing every service author learns, the build tool is one more thing the deploy pipeline runs, and the round-trip from the UI is lossy. The composition benefit is real but small at our dashboard count, and we can revisit if the fleet grows.
  • Terraform provider (grafana_dashboard). Fits a broader IaC story. Rejected: dashboards are operational objects that change on a different cadence than infrastructure; a terraform apply to tweak a panel title is friction the authoring flow cannot afford. Also couples the dashboard lifecycle to the TF state file, which the on-call engineer does not own.
  • Grizzly / grr. YAML wrapper over JSON with k8s-style metadata. Rejected: it adds a wrapper format whose only job is to reintroduce a concept (UID as metadata) that raw Grafana JSON already has. The extra indirection buys nothing here.
  • UI-only, no source control. Author in Grafana, accept that it's the source of truth. Rejected: dashboards silently diverge across environments, nobody reviews changes, and a Grafana outage or human deletion loses the work. The factory-model thesis (shared substrate) requires dashboards in source control.

Supersedes: none Superseded by: