← Decisions operate mode →

Decisions

052 — Operate-mode data sources proxied server-side, not direct-browser

The Platform UI's Operate mode ([ai-frontend §5](https://github.com/gyrum-labs/ai-frontend/blob/main/docs/platform-ux-redesign-proposal.md#5-operate-mode)) renders fleet health and alerts in two regions that need live…

#052

052 — Operate-mode data sources proxied server-side, not direct-browser

Date: 2026-04-22 Status: Accepted Related: ADR-003 (ai-frontend, three-modes platform UI), ADR-049 (Alertmanager bearer at Caddy), ADR-050 (per-product /metrics bearer), ADR-047 (Grafana folder-per-product)

Context

The Platform UI's Operate mode (ai-frontend §5) renders fleet health and alerts in two regions that need live data:

  1. Service health grid — one row per product, showing up / 5xx-rate / p95-latency over the last five minutes. Source: Prometheus at https://grafana.gyrum.ai/api/datasources/proxy/uid/prometheus (via the Grafana datasource proxy; the Prometheus container on 10.0.0.4:9090 has no public TLS and no public DNS).
  2. Active + recent alerts — firing now and resolved-in-7d. Source: Alertmanager at https://alertmanager.gyrum.ai/api/v2/alerts, which is bearer-gated as of ADR-049.

Both sources require secrets. The Grafana datasource-proxy call needs a Grafana service-account token (same one deploy.sh uses to provision dashboards, lives at /deploy/.secrets/grafana.env on gyrum-prod-1). The Alertmanager call needs the bearer from /deploy/.secrets/alertmanager-bearer.token on dark-factory-ops.

A decision is needed because the ai-frontend is a SvelteKit app with a Node adapter — the UI can run queries from either side of the wire. Three broad architectures are plausible:

  • A. Direct browser → Prometheus/Alertmanager. Ship the tokens to the browser.
  • B. Browser → backend → upstreams. Backend (ai-research cmd/server, the :9400 Go process) holds the tokens and proxies.
  • C. Browser → Grafana plugin embeds. Iframe Grafana panels, let Grafana handle auth.

The ntfy + Alertmanager incident context (ADR-049) sharpens the threat model: a bearer leak to the browser would let any page with that token silence alerts pre-attack — the exact vector ADR-049 closed. This is stronger than a "tokens should live server-side" generic preference.

Decision

The ai-research Go backend (:9400) proxies both Prometheus and Alertmanager for the Operate mode UI. Two new endpoints, server-rendered with tokens held in backend env vars:

  • GET /api/v1/operate/service-health — queries Prometheus (via the Grafana datasource proxy) for up, 5xx-rate, and p95 latency. Merges on the job / service label. Caches the merged payload for 30 seconds.
  • GET /api/v1/operate/alerts — fans out two requests to Alertmanager (active=true&silenced=false + the 7-day resolved window). Strips Alertmanager internals (fingerprint, receivers, generatorURL, inhibitedBy, silencedBy) from the response and projects onto a UI-shaped payload.

The backend reads four env vars — PROMETHEUS_URL, PROMETHEUS_TOKEN, ALERTMANAGER_URL, ALERTMANAGER_BEARER_TOKEN — and never writes either token to a response body or header. Tests assert the bearer never leaks.

Grafana dashboard links are rendered in the UI (grafanaFolderUrl per row) per ADR-047's folder-per-product convention; we link out, we do not iframe.

Consequences

Good:

  • The Alertmanager bearer closed by ADR-049 stays closed. A stolen browser session cannot silence alerts.
  • The Grafana service-account token never touches a browser. Its scope (Admin in Grafana) is wide enough that browser exposure would be an org-level incident.
  • CORS surface does not grow. The browser only talks to :9400, which already has a CORS origin policy (cors.go).
  • Caching happens in one place (backend, 30 s TTL). Hundreds of operators refreshing Operate mode do not fan out to hundreds of Prometheus queries — Prometheus is a single process and was not sized for that.
  • Response shape is UI-oriented, not upstream-oriented. When the UI needs a new field (e.g. "traffic sparkline" in §5 Phase 3), the backend aggregates once instead of the browser stitching three endpoints.

Trade-offs:

  • The ai-research backend grows a dependency on two upstream services. A Prometheus outage breaks the service-health region (degrades to an empty grid with a banner); an Alertmanager outage hides firing alerts (a real incident hazard — the ops team must monitor Alertmanager health separately, e.g. via an ntfy "Alertmanager not reachable" deadman alert).
  • Two more env vars to provision. Documented in .env.example; deployed via the standard /deploy/.secrets/ mechanism.
  • Query logic lives in Go, not in a Grafana dashboard. Evolving the PromQL means a backend release instead of an ops-side JSON edit. Accepted — the queries are simple and change rarely; a backend release is already the path for any UI-shape change.

What this does NOT solve:

  • Per-user auth on :9400. Today the ai-research backend assumes the operator is already authorised (no login wall). When the platform grows multi-user access, Operate mode is one of the surfaces that needs auth — tracked as a future ADR.
  • Grafana dashboard embedding. The UI still link-outs to Grafana for deep drill-down. Embedding Grafana panels would require solving Grafana's own auth-from-iframe story; out of scope.
  • Write operations on Alertmanager. Silence creation / deletion is not exposed through the backend. That is intentional — writing through a server-side proxy does not make the operation less dangerous; it just removes the attribution (Alertmanager's API does not know who the bearer-holder was). Silences remain a CLI / runbook operation for now (see observability-runbook).

Alternatives considered

  1. Browser → Prometheus/Alertmanager directly, tokens shipped via a public env var. Rejected. ADR-049 explicitly closed the Alertmanager bearer to anonymous; handing it to every browser window undoes that. The Grafana service-account token has Admin scope — browser exposure equals full Grafana compromise. This option is not recoverable by rotation alone: a token leaked to a browser is a token in browser history, extension memory, possibly uploaded in crash dumps.

  2. Browser → Grafana with panel embeds (iframes). Considered. Grafana's own login / allow-list takes care of the Prometheus side. Rejected because (a) Grafana does not expose a low-friction embed-with-auth for our IP-allow-list setup, (b) iframe layouts fight with the UI's density requirements — Operate mode wants a compact table, not a stack of panels, (c) we would still need a separate proxy for Alertmanager (no Grafana datasource for raw AM writes).

  3. A thin separate BFF service (not the ai-research :9400 backend). Rejected. We already have a backend the UI talks to. Adding a sibling service doubles the ops surface (separate container, separate port, separate CORS config, separate deploy). The :9400 backend already proxies to other read-only upstreams (/api/v1/infra/* reads hetzner-snapshot JSON); Operate mode follows the same pattern.

  4. Server-side Svelte +page.server.ts handlers instead of Go. Plausible — keeps data-fetching in the same runtime as the UI. Rejected for now because (a) the SSR node runs alongside the UI but is typically sized for render throughput, not cross-service proxying, (b) the Go backend already holds a pattern for secrets-from-env and aggressive caching, (c) splitting Operate-mode data between two runtimes (Prom in Node, something else in Go later) would create a second stitching point. Single language for the proxy tier keeps the blast radius contained.

  5. A static dashboard in Grafana that the UI iframes for Operate mode. Rejected. The operator wants one place for "what is broken", not a link-out to a tool that requires a second login context. Grafana remains the place for detailed investigation — linked from every service-health row — but not the Operate-mode landing itself.

  6. Cache in the browser (ServiceWorker / sessionStorage) instead of server-side. Rejected as the cache layer question. Browser cache does not solve the "30 operators hitting Prometheus concurrently" problem — each operator still does the initial fetch. Server-side cache solves both the browser-refresh storm and the across-operator concurrency, for one TTL.

Verification

# With both endpoints deployed:
curl -s http://localhost:9400/api/v1/operate/service-health | jq '.[] | {slug, status, errorRate5m}'
# expect: rows for every product scraped by Prometheus (3+ as of 2026-04-22:
#         distill, arbibot, buzzy; 14 more merged, awaiting deploy).

curl -s http://localhost:9400/api/v1/operate/alerts | jq '.active | length, .recent7d | length'
# expect: active=0 (no alerts configured yet), recent7d=0 or small.

# Bearer containment — the token must never appear in responses or headers:
curl -sv http://localhost:9400/api/v1/operate/alerts 2>&1 | grep -i 'authorization\|bearer'
# expect: no output.

# Backend refuses /operate/* without upstream tokens:
#   Running the backend with PROMETHEUS_TOKEN unset yields a 502 from
#   /service-health with "prometheus HTTP 401" in the error body — deliberate
#   fail-closed behaviour.

Rollback

Back out both endpoints and the env-var block from .env.example. The Operate mode UI tolerates 502 responses from both paths (degrades to "Prometheus unreachable" / "Alertmanager unreachable" affordances per the UI Promise.allSettled loader), so rollback does not take the /operate route down — it just shows empty regions until the endpoints return. Mode-switcher and sub-routes (/operate/users etc.) are unaffected.