ADR-053: Frontend observability — `/client-log` endpoint + slim browser SDK

Status: Proposed Date: 2026-04-22

Context

ADR-051 wired observability into every backend emitted by the build-full pipeline: pkg/observ import, observ.SetBuildInfo(version, commit) at startup (ADR-032), /metrics bearer-auth'd per ADR-050, slog JSON flowing to Loki via Alloy, Prometheus scraping per ADR-047, Alertmanager alerts per ADR-049. That ADR explicitly deferred the frontend story to "a future ADR-05X"; this ADR is that story.

The frontend gap today, observed across the 14 services integrated on 2026-04-22 (distill, buzzy, guardian, hiphip, and the 10 build-full products already retrofitted via gyrum-observ-integrate, ADR-045):

No build identity. We cannot tell which commit of the frontend bundle is currently running in a user's browser — there is no equivalent of app_info for the browser. Rolled-back backends + stale browser caches are an active class of incident we cannot diagnose.
No error capture. console.error is used ad-hoc; there is no central ingestion of window.onerror, unhandled rejections, or CSP violations. When a SvelteKit component throws in production, the first signal is a user email.
No user-visible performance signal. LCP / INP / CLS are invisible to us. Perf regressions land silently until someone notices.
No correlation. Even when a frontend error is reported, there is no way to stitch it to the backend request_id that slog emits on the server side — so the whole point of having a unified observ stack is half-realised.
No alerts on frontend-side failure modes. 5xx on static asset fetch, TLS expiry mid-session, auth-loop (redirect storms on refresh failure) all fall to the floor.

An older KB note at docs/knowledge-base/frontend-error-logging.md (duplicated under projects/104-dark-factory-control-plane/3-development/docs/knowledge-base/) sketches a similar shape — browser posts to /api/client-error, backend logs, Loki picks up via Promtail. That note pre-dates the current observ stack (log.Printf and Promtail, not observ.Logger() and Alloy) and has no build identity, vitals, correlation, or Prometheus surface. This ADR supersedes it.

What frontend obs must deliver, in decreasing order of urgency:

Build identity — which commit + release-track is the bundle the user is running?
Error capture — JS errors, unhandled rejections, CSP violations end up queryable in the same Loki instance as backend logs.
Web vitals — LCP, INP, CLS per page, so perf regressions have a dashboarded signal.
Correlation — a frontend error carries the X-Request-Id of its most recent server interaction, so a single search stitches the browser trace to the backend log trail.
Alerts — once frontend error and vitals counters are flowing in Prometheus, Alertmanager can page on spikes the same way it does for backends.

Anti-requirements — what this ADR does NOT promise:

Not RUM-as-a-product. We are not building Sentry or Datadog RUM. Coverage is "the same obs story as backends, extended into the browser" — no product-grade features (issue grouping, release dashboards, saved queries UI) beyond what Grafana on Loki already gives us.
Not user behaviour analytics. Journey tracking, funnel analysis, feature-flag exposure metrics — that is a product-experimentation concern and belongs in a separate tool (PostHog, Plausible, etc.), not in the observ stack.
Not session replay. The privacy surface (DOM capture includes form values), the payload size, and the storage cost are all disproportionate to the signal for our current fleet.

Decision

Frontend observability lands via option (A): ship to the backend's existing Loki pipeline via a new /client-log endpoint on each service, plus a slim browser SDK. For the narrow case of pre-auth / pre-load errors where the app cannot reach its own backend, a secondary carve-out is added (see "Pre-auth carve-out" below).

The backend remains the single point that holds secrets, owns rate limits, and writes to Loki. The browser never talks to loki.gyrum.ai directly and never holds a Loki ingest token.

The `/client-log` endpoint (backend side)

A new handler lives in gyrum-go at pkg/observ/clientlog. Mounted by every service at POST /api/v1/client-log, it:

Accepts a bounded JSON body (LimitReader at 16 KiB) with a documented schema: level, msg, fields (flat map), build (service, version, commit, releaseTrack), url, userAgent, requestId (optional, echoed from an earlier X-Request-Id), clientTs.
Validates schema, drops fields with a known-bad shape, applies a redaction pass on the fields map (the same deny-list the backend uses for slog attributes — password, token, authorization, cookie, session, and prefixes of each).
Emits one slog record via observ.Logger() with a source="frontend" label and a service label matching the backend's. Loki picks it up via the Alloy pipeline that is already in production; no scrape config changes.
Increments Prometheus counters on the backend's own /metrics surface:
- frontend_errors_total{service, kind} where kind ∈ {"error", "unhandledrejection", "csp", "log"}
- frontend_vitals{service, metric, p} (histogram) for metric ∈ {"lcp","inp","cls"} This is how frontend signals enter Prometheus — there is no browser scrape surface (see "What we scrape vs what we log" below).
Rate-limits per source-IP and per build.service (bucketed; a small allowance to absorb a page with a genuine burst of errors without falling open to a malicious flood). Limits are configurable via env var, defaulted conservatively.
Is wired from the backend's middleware stack before any auth-required middleware — the endpoint must be reachable when the user's session has expired. CSRF is not required (no state mutation beyond log emission) but CORS is same-origin only; cross-origin submissions are rejected.

The browser SDK

A small TypeScript module (proposed at frontend-observ-gyrum-io, a new repo — see open questions) exports:

setBuildInfo({ service, version, commit, releaseTrack })
logger.info / warn / error (msg, fields?)
reportWebVitals()      // wraps the web-vitals npm package
install()              // auto-registers window handlers

setBuildInfo(...) is called once at app boot; mirrors observ.SetBuildInfo on the backend. Values are injected by the build-full pipeline as import.meta.env.VITE_* from the same ldflags the backend's Dockerfile already owns — one source of truth for build identity across both sides.
install() registers handlers for window.onerror, window.onunhandledrejection, and securitypolicyviolation; each handler converts to the SDK's log shape and enqueues.
reportWebVitals() wraps web-vitals and batches LCP/INP/CLS on visibilitychange=hidden and pagehide — one flush per page lifetime plus a 60s safety flush.
Correlation. The SDK wraps fetch to read X-Request-Id off each response and stash the last value; subsequent log records carry it as requestId. Deliberately shallow — "frontend error → most recent backend call's request_id" without a full W3C trace context (out of scope).
Transport. navigator.sendBeacon() on pagehide, fetch(..., { keepalive: true }) otherwise. Bounded queue (64 entries, oldest-dropped); 5s flush interval during interactive sessions.
Redaction. Same deny-list as the backend handler, applied client-side too (defence in depth, cheaper at source).
Budget. Zero deps beyond web-vitals (~2 KB gzipped); SDK total ~6 KB gzipped — small enough for the critical path.

Pre-auth carve-out

A login-page crash, a CSP violation on the landing route, or a network-layer failure that prevents the service's own /client-log endpoint from resolving will leave the user's browser with no way to report. For these cases:

The backend exposes /api/v1/client-log on the unauth'd route tree (stated above) so session state is not a gate.
If the service's own origin is unreachable, the SDK falls back to not sending (no cross-origin shipment to a central loki.gyrum.ai/loki/api/v1/push, no browser-held tokens — option (B) below was rejected on this axis).
The trade-off is acknowledged: a hard-down frontend origin means a hard-down frontend error path. For the current fleet this is acceptable; a central collector (option (C)) would close this gap and can be revisited as a separate ADR once the fleet crosses ~25 services or we see a recurring unreported-failure pattern.

What we scrape vs what we log

Loki gets the structured log line unchanged, with source="frontend" and the backend service's label set. The existing service-errors dashboard filters by source to split backend vs frontend panes without dashboard duplication.

Prometheus never scrapes a browser. The /client-log handler is the adapter: every received record increments frontend_errors_total (by kind), and every received vitals record adds to frontend_vitals{metric,p}. Those counters live on the backend's own /metrics endpoint, which is already bearer-auth'd per ADR-050 and already scraped per ADR-047. No new scrape target, no new datasource, no new auth surface.

Consequences

Good:

One obs story for backends and frontends — same Loki, Prometheus, Grafana, Alertmanager. Investigators filter by service and toggle source; no second system to learn.
Correlation out of the box. requestId on frontend records stitches to the backend's request_id from observ.Access() — one LogQL query spans both sides.
Build identity on the browser. setBuildInfo closes the "which bundle is running" gap; rollback impact becomes visible via frontend_errors_total{service,version}.
No new secret surface. Browser holds no Loki token; rate-limit enforcement stays server-side.
Retrofit is mechanical. The shape mirrors ADR-045 — one CLI pass wires SDK import + setBuildInfo + install() on the frontend and the handler registration on the backend.

Trade-offs:

Every service grows a new endpoint — another rate-limit target, another route for the Caddy ACL. The code is a single handler registration from gyrum-go, but the surface is real.
The SDK is a new frontend dependency to maintain (model drift, browser-API deprecation, bundler compat), kept small (~6 KB) but not zero-cost.
Pre-auth blind spot per carve-out. Accepted for now.
Cross-origin cases (Platform UI talking to multiple backends) need a policy call on which backend receives the client-log. SDK default is same-origin; Platform UI needs explicit configuration.

What this does NOT solve (candidates for later ADRs):

Session replay.
Real user-funnel / behaviour analytics.
Sentry-style breadcrumbs (a running circular buffer of user actions attached to each error).
Full W3C distributed-tracing traceparent propagation.
A central collector service (option (C)) for pre-auth / hard-down origin coverage.

Alternatives considered

(a) Do nothing, rely on Playwright. E2E catches what E2E catches — a sliver. Real-browser diversity (iOS Safari, corporate proxies, content-blockers) is invisible; post-deploy user-side regressions stay invisible until a user mails in. Rejected: the gap ADR-051 flagged does not close itself.
(b) Sentry or self-hosted Glitchtip. Full-featured (issue grouping, breadcrumbs, source maps). Rejected as primary: another box to run with its own DB, auth, and handbook, when Loki + Prometheus + Grafana is already paid for. May land later as a specialised layer on top of this ADR's Loki pipeline.
(c) This ADR's proposal — chosen. Ship to Loki via the backend's own /client-log endpoint with a slim SDK. Single obs story, minimum new surface.
(d) Ship directly from browser to loki.gyrum.ai/loki/api/v1/push with a browser-safe ingest token. Cuts the backend hop but puts a Loki token in browser JS — any page can exfiltrate and flood Loki, forcing a rotation that cascades across every frontend. Rejected as primary; may revisit as pre-auth carve-out if option (e) does not happen.
(e) A central client-log collector service. New repo, new container on dark-factory-ops; owns tokens and rate-limiting. Clean design for a 50+ service fleet, overkill for 14. Rejected for now; revisit at ~25 services or if pre-auth coverage becomes a real pain point. The SDK's transport is a single URL behind a flag so switching later is cheap.
(f) OpenTelemetry browser SDK to an OTLP collector. Natural road to full distributed tracing, but the JS OTel SDK is tens of KB gzipped — a perf tax for a capability we do not yet consume. Additive later if we want it; no rewrite needed.
(g) Accept the gap until an incident forces the hand. What we have been doing. Rejected: ADR-051 flagged it, the KB note has sat stale for weeks, and the fleet has grown past the point where the cost of waiting beats the cost of shipping.

Rollout plan

This ADR is a design decision; the code follows in separate PRs with explicit scope per phase.

Phase 1 — library. Ship the handler and the SDK as library code only, both tested against fixtures:

gyrum-go/pkg/observ/clientlog — the /client-log HTTP handler, redaction, rate-limit, Prometheus counters, slog emission.
frontend-observ-gyrum-io — the TS module, with unit tests and a Playwright fixture exercising install + flush + correlation.
No service changes. No frontend changes. Merge gates on this ADR being accepted.

Phase 2 — build-full integration. Extend ADR-051's "Wire observability" step so that newly emitted products get the frontend SDK imported, setBuildInfo called with the injected build vars, and install() invoked in the SvelteKit root layout. The backend half gets the /client-log route registered from the same generator edit that registers /metrics and /probe today.

Phase 3 — retrofit. Extend gyrum-observ-integrate (ADR-045) with a --frontend flag (preferred over a separate gyrum-observ-integrate-frontend — one audit matrix, one CLI) to roll the SDK + handler into the 14 existing services' frontends. The audit script at dark-factory/scripts/observ-integration-audit.sh grows a frontend column.

Phase 4 — alerts. Once frontend_errors_total and frontend_vitals have been flowing long enough to baseline (two weeks is the working number, revise after first baseline pass), add Alertmanager rules for error-rate spikes per service, INP regression against the rolling baseline, and TLS-expiry / auth-loop signatures visible from the frontend's error kinds. Lives in the existing alert-rules tree so there is no new Alertmanager surface.

Open questions

Left open — to be resolved during implementation PRs, not in this ADR:

SDK home. New repo frontend-observ-gyrum-io (own release cadence, matches naming) vs. folding into ai-frontend shared (fewer repos, tighter coupling). Default: new repo; decide in Phase 1 kickoff.
Rate-limit defaults. Concrete numbers (req/min/IP, req/min/service) set during Phase 1 against synthetic load.
Vitals cardinality. One histogram × three metrics is cheap; url-label tail may need bounding once real traffic lands. Phase 4 baseline pass decides.
Cross-origin Platform UI. platform.gyrum.ai talks to multiple backends — which one owns its frontend client-logs? Default "ship to platform itself" approximates option (C) for one service. Resolve in Phase 3.
Source map handling. Stack traces will reference minified bundle names. Where source maps live, and whether Grafana can resolve them at query time, are open. Out of scope for Phase 1; decide before Phase 4 alerts turn on.

Supersedes: docs/knowledge-base/frontend-error-logging.md (and its duplicate under projects/104-dark-factory-control-plane/3-development/docs/knowledge-base/) — the KB note pre-dates the current observ stack; this ADR replaces it, and the note will be reduced to a pointer. Superseded by: leave blank until a later ADR reverses this one

ADR-032 — setBuildInfo mirrors this on the browser.
ADR-045 — grows a --frontend flag in Phase 3.
ADR-050 — the new frontend counters live on the same bearer-gated /metrics.
ADR-051 — backend observ-by-default; this is its frontend companion.
ADR-052 — same "backend holds secrets" principle as the /client-log choice here.
docs/knowledge-base/frontend-error-logging.md — superseded.

ADR-053: Frontend observability — /client-log endpoint + slim browser SDK