ADR-040: Grafana dashboard provisioning runs at deploy time, not CI time

Status: Accepted Date: 2026-04-21

Context

Phase 3 of the observability rollout (ADR-013, ADR-014) produces per-service dashboard JSON files in observability/grafana/<service>/. Something has to push those files into the running Grafana instance so the on-call view at grafana.gyrum.ai matches what source control says.

Three places could own the push:

CI (GitHub Actions). Runs on every PR merge.
A GitOps operator in-cluster, reconciling a CR to Grafana.
infrastructure/deploy.sh on the engineer's workstation, right after the service ships.

The Grafana instance lives on the observability VPS behind Caddy. The grafana.gyrum.ai hostname is public (Cloudflare DNS, TLS cert via Caddy's ACME), but the service-account token that can write dashboards is not a GitHub secret — there is no workflow on the fleet that owns managing Grafana credentials in a third-party CI vault. Adding one is possible but is a whole new risk surface: another place the token can leak, another rotation cadence, another on-call boundary.

CI also has no context for what was actually deployed. A PR merge does not imply a deploy; a deploy can happen from a tag, from a manual override, or from a rollback SHA. Tying dashboard sync to PR merge would sync dashboards for code that is not running.

Decision

infrastructure/deploy.sh runs a Grafana dashboards stage after the screenshot stage on every deploy. It reads credentials from /deploy/.secrets/grafana.env on the VPS, iterates observability/grafana/<service>/*.json under the synced project dir, and POSTs each file to $GRAFANA_URL/api/dashboards/db with overwrite: true. A second stage, Verify observability, probes Loki and Prometheus with the same service slug to confirm signals are arriving (see ADR-041).

Both stages are non-blocking: a failure logs at WARN level and the stage exits 0. A Grafana outage must not roll back an otherwise-healthy deploy.

Consequences

Operator sees provisioning in the deploy console. The same stdout stream that drives the Platform UI's stepper carries the provisioning outcome — a ✓ provisioned: service-overview.json line per dashboard. No separate tab to check, no CI retry loop.
Dashboards track what is actually running. A rollback deploy re-posts the dashboards of the code that is running, not the code that got merged last. If source control and production are briefly out of sync (cherry-pick hotfix, deferred merge), the dashboards follow production.
Credentials live on the VPS, not in CI secrets. The Grafana token is written to /deploy/.secrets/grafana.env with chmod 600. Only users with SSH to the deploy host can read it; rotation is ssh + edit file. Same pattern as the existing cloudflare.env.
A broken Grafana does not break deploys. The non-blocking design is load-bearing: Grafana is a diagnostic tool, not a production dependency of the services. Coupling service deploys to Grafana availability would be an own-goal.
Missing grafana.env skips silently. A new VPS without the file gets a WARNING: No Grafana credentials... line and no dashboards. This is the correct UX for a bootstrap state — operators add the file when they're ready.
Dashboards drift if you don't deploy. A service that hasn't shipped in six months has stale dashboards. The mitigation is the scaffold CLI's --sync action plus a routine redeploy; we do not run a cron to re-provision idle services.

Alternatives considered

CI-based provisioning. GitHub Actions runs the POST on every merge to main. Rejected: needs a GitHub secret for the Grafana token (new credential surface), and syncs on PR merge rather than deploy (lies when rollbacks or cherry-picks diverge from main). CI also has no "which service was just deployed to which environment" signal — we'd have to invent one.
GitOps operator in-cluster. A controller reconciles a Dashboard CR against Grafana. Rejected: overkill for our scale (one Grafana, low double-digits of services), adds a Kubernetes-ish layer to a Docker-on-VPS fleet, and the failure modes (controller silently stops, webhook not firing) are harder to debug than a stage marker in the deploy log.
Manual operator POST. Run the curl by hand after each deploy. Rejected: drift is guaranteed, every engineer reimplements the call slightly differently, and the "did someone remember to do it?" failure mode is exactly what automation exists to remove.
Dashboards owned by the Grafana server's provisioning directory. Grafana can load dashboards from /etc/grafana/provisioning/. Drops the API entirely. Rejected: file sync to the Grafana host is itself a deploy problem; dashboard-as-files-on-disk reopens the "who owns the truth?" question ADR-014 resolved via stable UIDs.

Supersedes: none Superseded by: