ADR-047: Folder-per-product Grafana convention + UID-pinned datasources

Status: Accepted Date: 2026-04-22

Context

The 2026-04-22 manual end-to-end wire-up of distill into the observability stack (dark-factory issues #266–#273) surfaced three related failure modes in the deploy-time provisioning we already had in ADR-040:

1. Dashboards land in the default folder. Every service shipping a Tier-0 + Tier-1 pack today (services/templates inventory: infrastructure/grafana-templates/) drops 5–10 dashboards into the single shared "General" folder. Operators cannot tell at a glance whether service-overview belongs to distill, buzzy, or the control plane. The fleet matrix (docs/observability-fleet-status.md) projects 24 services; the default folder collapses under that load.
2. The deploy-host vs VPS-host split is wrong. The current Grafana stage runs bash <<HEREDOC over SSH and reads $REMOTE_DIR/observability/grafana/<service>/*.json on the VPS — but only the build artefact rsyncs to the VPS, not the source tree. The dashboards exist locally on the deploy host (the engineer's workstation, or the platform UI worker), not on the VPS. Today this works for distill only because distill happens to be deployed from a layout that includes those files in the rsync set; the registry-deploy and most other services would not.
3. Grafana datasource UIDs are random. grafana-datasources.yml creates the Loki and Prometheus datasources by name but leaves uid unset, so Grafana mints a fresh random uid on every fresh bootstrap. Every dashboard in infrastructure/grafana-templates/ references its datasource by uid (Grafana's own runtime addressing). On a fresh bootstrap, the dashboards point at uids that don't exist — every panel reads "Datasource not found".

These are three different failures of the same shape: things that should be set structurally (where dashboards land, which host the POST runs from, what a datasource's identity is) were left implicit, producing fleet-wide drift the moment a second product joined.

Decision

We adopt three structural conventions, all owned by infrastructure/server-bootstrap/ + infrastructure/deploy.sh:

1. One Grafana folder per service. The folder uid equals the service slug (the leading dot label of DEPLOY_DOMAIN: distill.gyrum.ai → distill). The deploy.sh Grafana stage POSTs /api/folders once with {uid: $service, title: $service} (HTTP 412 / 409 swallowed as "already exists"), then includes folderUid: $service in every dashboard upsert. Folder management is owned by deploy.sh; the scaffold CLI does not need to know.
2. Dashboards push from the deploy host. The Grafana stage in deploy.sh no longer SSHes into the VPS to read dashboards. It pulls the Grafana credentials off the VPS into a 0600 tmpfile, then iterates $PROJECT_DIR/observability/grafana/<service>/*.json on the deploy host and POSTs from there. This decouples provisioning from rsync content; it works identically for every project-shape (registry-deploy, source-on-VPS, build-artefact-only).
3. Pin Grafana datasource UIDs. infrastructure/server-bootstrap/config/grafana-datasources.yml sets uid: prometheus and uid: loki literally. Dashboards under infrastructure/grafana-templates/ may reference these uids directly. The uids are now part of the bootstrap contract — they do not change, and the YAML carries an inline comment noting that.

The provisioning stage stays non-blocking (ADR-041) and stays on the deploy-time path (ADR-040). Stable dashboard UIDs (ADR-014) keep their meaning — folder placement and dashboard upsert are orthogonal idempotency layers.

Consequences

• Operators see one folder per product in Grafana. "Show me distill's dashboards" is a single click. The folder hierarchy in Grafana now mirrors the fleet.
• A fresh bootstrap produces a working dashboard pack on first boot. No "open Grafana → edit datasource → fix uid → refresh" ritual. The pinned uids match what the templates expect.
• The deploy script no longer assumes the source tree was rsynced. Adding a new project shape (Go binary built locally, static dist rsync, etc.) does not require thinking about the Grafana stage — it reads from $PROJECT_DIR regardless.
• Grafana credentials briefly leave the VPS. The deploy stage now fetches /deploy/.secrets/grafana.env over SSH into a 0600 mktemp file on the deploy host. The file is removed by an EXIT trap; the token never lands on disk persistently. Operators with deploy access have always had this token (ssh + cat would have read it); the new code path materialises it for ~1s on the deploy host. We accept this in exchange for the simpler local-execution model.
• Folder uid is now load-bearing. Renaming a service (changing the leading label of DEPLOY_DOMAIN) leaves the old folder behind in Grafana — the new deploy creates a fresh folder. Operators must manually delete or rename the orphan in the Grafana UI; we do not garbage-collect folders on rename.
• Dashboard JSONs that hardcode the legacy random datasource uid break. Every dashboard under infrastructure/grafana-templates/ is being rebuilt against prometheus / loki as the canonical uids; existing in-Grafana dashboards that an operator built by hand against the old random uid silently fail until the operator edits them. We accept this — the Tier-0/Tier-1 pack is the only source of truth for dashboards in the fleet (ADR-013), and no service shipped in production yet has a hand-built one.
• The split-VPS topology is now first-class in vars/example.env. SERVER_ROLE (ops|app) and OPS_LOKI_PRIVATE_URL are documented; the ops stage (08) and the new app-VPS Alloy stage (11) self-skip on the wrong role. Single-VPS bootstraps continue to work unchanged — the default SERVER_ROLE=ops reproduces the old behaviour.

Alternatives considered

• Folder-per-team or folder-per-tier. Group dashboards by team (frontend/backend/platform) or by tier (Tier-0/Tier-1/Tier-2). Rejected: team membership is fluid and not in source control; tier is already encoded in the dashboard template name (service-overview vs service-billing). Folder is the right axis for "who owns this signal" — a product owns its signals.
• Folder-per-environment. Production / staging / dev as folders. Rejected: we do not have multiple environments per service yet (gyrum.dev → gyrum.ai is a domain difference, not an environment we observe separately). Re-examine when we do.
• Provision dashboards via Grafana's filesystem provisioning (/etc/grafana/provisioning/dashboards/). Drop the API entirely; drop a JSON file on disk and reload Grafana. Rejected for the same reasons as in ADR-040 — file sync to the Grafana host is itself a deploy problem, and we already solved it via the API.
• Generate uids per-bootstrap and re-write dashboards in place. Keep random uids; the bootstrap script rewrites every dashboard JSON to match. Rejected: the bootstrap would need to know the full set of dashboard files (it does not — services scaffold their own), and it would couple infrastructure/server-bootstrap/ to every service's source tree. Pinning uids is a much smaller commitment.
• Grafana service account per product. A separate grafana.env per product, each with its own folder-scoped token. Rejected for now: we have one Grafana, one operator pool, and the blast radius of a leaked admin token is the same whether scoped or not (still all dashboards). Re-examine when a product needs to self-provision dashboards from its own CI without operator involvement.

Supersedes: none Superseded by: