← Decisions operate mode →

Decisions

ADR-050: Bearer-auth product /metrics endpoints via a gyrum-go helper

**Note on ADR number:** drafted as ADR-049 in parallel with the alertmanager bearer-auth work. A parallel agent merged an unrelated `ADR-049: Alertmanager bearer-auth at Caddy` (dark-factory#278) before this one landed,…

#050

ADR-050: Bearer-auth product /metrics endpoints via a gyrum-go helper

Status: Accepted Date: 2026-04-22

Note on ADR number: drafted as ADR-049 in parallel with the alertmanager bearer-auth work. A parallel agent merged an unrelated ADR-049: Alertmanager bearer-auth at Caddy (dark-factory#278) before this one landed, so this was renumbered to 050. The two ADRs are complementary: ADR-049 puts a bearer on the operator-facing alertmanager.gyrum.ai route at Caddy; ADR-050 puts a bearer on the scraper-facing /metrics route inside each product binary.

Context

The 2026-04-22 end-to-end observability wiring exposed a quiet gap. Every product's /metrics endpoint is reachable without auth over the public internet:

$ curl -o /dev/null -w "%{http_code}\n" https://distill.gyrum.ai/metrics
200

That response leaks:

  • app_info{version,commit,go_version,service,release_track} — version disclosure that shortens the window between a published CVE and an attacker knowing the instance is vulnerable.
  • http_requests_total{route} — a route enumeration of the service (the bounded-cardinality label set is still a map of every endpoint the product serves).
  • Full Go runtime telemetry — memory, GC pause histograms, goroutine counts — which is useful for timing attacks and resource-exhaustion reconnaissance.
  • Every counter the product publishes via gyrum-go pkg/observ/metrics, including per-product business counters like reports_generated_total.

The existing pattern solves the problem for exactly one service — dark-factory-control-plane — by carrying an inline Bearer token in the Prometheus scrape config. That works, but the token lives in /deploy/observability/prometheus/prometheus.yml on dark-factory-ops (which is in git as a template, but the token is live-only), the match is done ad-hoc in the control-plane's own router, and nobody else on the fleet is copying it because there is nothing to import. Distill was deployed unauth'd, documented as "we'll fix it" (issue #269), and the same shape will repeat for every future product.

The fleet is about to grow — buzzy, socialproof, guardian, research, and the other *-gyrum-io products all integrate with the observ stack via gyrum-observ-integrate (see ADR-045). Without a library primitive, each of them will either (a) copy the control-plane code by hand, (b) skip the auth entirely, or (c) roll their own slight variation. That is exactly the drift we set up pkg/observ to prevent (ADR-008).

Related in flight:

  • gyrum-go pkg/observ already owns HTTP middleware, build identity, and the metrics sub-package.
  • dark-factory infrastructure/deploy.sh runs products via docker run with an --env-file pattern; secrets on disk at /deploy/.secrets/.
  • The Prometheus side supports both inline credentials and credentials_file — the file path reloads on every scrape, which means token rotation is a file write and no restart (the container does not have to be recreated).

Decision

Add observ.MetricsAuth(token string) func(http.Handler) http.Handler to gyrum-go pkg/observ, wire it into every product's /metrics route via the gyrum-observ-integrate CLI's routes stage, and standardise on Prometheus authorization.credentials_file (not inline) on the scrape side.

The helper:

  • Rejects requests with a missing or non-Bearer Authorization header as 401.
  • Compares the incoming token against the configured token with crypto/subtle.ConstantTimeCompare to close the early-return timing side channel.
  • When constructed with an empty token, is a pass-through middleware AND logs a single WARN via observ.Logger() on first construction. This is the deliberate escape hatch that (a) lets local dev curl /metrics without ceremony and (b) makes the deploy-sequencing story work (see "deploy sequencing" below).

Products wire it exactly once, in their registerInfraRoutes:

mux.Handle("/metrics", observ.MetricsAuth(token)(metrics.Handler()))

Token arrives via a twin-envvar convention that mirrors Prometheus's own credentials / credentials_file split:

  • OBSERV_METRICS_TOKEN — inline env var; wins when non-empty.
  • OBSERV_METRICS_TOKEN_FILE — path to a file; standard Docker-secret flow; trailing whitespace stripped. Unreadable file is a WARN, not fatal — the service stays up, the operator sees two log lines telling them what went wrong.

Token storage (no value in git):

  • On gyrum-prod-1: /deploy/.secrets/<product>-metrics-bearer.token, mode 0600, owner deploy. Injected into the container via --env-file or explicit -v mount.
  • On dark-factory-ops: /deploy/observability/prometheus/<product>-bearer.token, mode 0400, chowned to 65534:65534 (the nobody uid the official prom/prometheus image runs as). This is the gotcha that cost us a scrape cycle during the rollout — writing the file as deploy makes it unreadable from inside the container. Bootstrap scripts must set the container uid on the file.

The Prometheus scrape config always uses authorization.credentials_file, never inline credentials:

authorization:
  type: Bearer
  credentials_file: /etc/prometheus/<product>-bearer.token

File-based credentials reload on every scrape with no Prometheus restart, so token rotation becomes a write new token → chown nobody → no container action needed flow. That is meaningfully simpler than the inline pattern the control-plane currently uses.

Deploy sequencing

The helper's empty-token pass-through exists specifically to make a safe deploy sequence possible. Any step fails → roll back the previous one; no step breaks the currently-working scrape.

  1. Ship the new binary with MetricsAuth wired BUT OBSERV_METRICS_TOKEN unset. Behaviour is identical to before: /metrics is 200 for all comers; a single WARN is emitted. Scrape keeps working.
  2. Drop the token file on both gyrum-prod-1 and dark-factory-ops. No observable behaviour change on either side yet.
  3. Update prometheus.yml to add authorization.credentials_file:. Restart Prometheus. Scrape still succeeds because distill's binary does not check the header.
  4. Set OBSERV_METRICS_TOKEN_FILE on the distill container and restart. Now /metrics returns 401 for public curls and 200 for the Prometheus scrape.
  5. Verify up{job="distill"} == 1 in Prometheus; verify unauthenticated curl → 401.

Reverse order for rollback: unset the env var first, then revert prometheus.yml, then remove the token files.

Consequences

Good:

  • One code path for every product. gyrum-observ-integrate wires it automatically from now on.
  • No new dependencies on gyrum-go — stdlib crypto/subtle only. The base pkg/observ stays dep-lean (the point of ADR-030).
  • Token rotation is a file write plus a chown; neither container restarts. The old control-plane pattern required editing prometheus.yml and recreating the container — not hard, but not routine either.
  • Version disclosure and route enumeration are closed for the whole fleet once gyrum-observ-integrate re-runs on every product repo.
  • The one-time WARN in empty-token mode means a production deploy that forgot to set the token is loud in Loki — not silently open.

Trade-offs:

  • Every product now needs a unique token provisioned on two VPSs. The runbook gains a "rotate a product's metrics bearer" procedure. Not onerous — two file writes, one chown, no restart — but it is new ops surface.
  • Prometheus's credentials_file loads on every scrape from disk. On a 15s scrape interval across ~10 products the syscall volume is trivial, but it is non-zero. Acceptable.
  • The empty-token escape hatch is deliberately silent-ish (one WARN) rather than loud (refuse to start). We accept this tradeoff because a product that refuses to start without the env var breaks the step-1 ship-binary-first deploy sequencing. The WARN is the compromise.

What we have signed up to operate:

  • Per-product token files in two locations.
  • The chown-to-65534 gotcha must stay documented; a future ops VPS rebuild needs to reapply it.
  • dark-factory-control-plane still uses inline credentials in prometheus.yml — it pre-dates this ADR and we are leaving it alone until a dedicated migration PR lands (tracked against the next quarter). The knowledge-base note in projects/104-dark-factory-control-plane/.../grafana-dashboards.md flags this to avoid confusion.

Alternatives considered

  • Keep /metrics unauthenticated — leaves the disclosure + enumeration path open on every product. Tenable for a single-tenant hobby VPS, not tenable for a commercial platform. Rejected.

  • IP-restrict via Caddy remote_ip on *.gyrum.ai/metrics — Prometheus on dark-factory-ops has a stable public IP (178.104.63.31), so Caddy could allow only that IP to hit the /metrics path. Simpler than bearer auth. Rejected because (a) we already have bearer-auth plumbing in gyrum-go, (b) IP allow-lists break the moment the ops VPS moves, (c) bearer-auth composes with future layers (mTLS, per-scraper tokens for multi-tenancy) without a rewrite, and (d) the product repos can test auth in-process without an HTTP hop to Caddy — testable design wins.

  • mTLS between Prometheus and the products — strongest option. Each scrape presents a client cert; Caddy terminates TLS, verifies the client cert, forwards to the app. Rejected for now: adds a cert management surface (issue, rotate, revoke), the Caddy config becomes significantly more complex per site, and we have no evidence the bearer pattern is insufficient. Revisit if we need per-product scraper identity (we do not).

  • Replace the Prometheus pull model with an OpenTelemetry push — the collector-on-every-host authenticates to a central OTLP endpoint, and /metrics can disappear entirely. Big architectural move. The current Prometheus setup is <2 weeks old and working; trading it for OTLP to solve an auth problem would be hugely disproportionate. Rejected; revisit only if we also need push (e.g. ephemeral jobs, Lambda-like workloads).

  • Have every product roll its own bearer check — what distill would have done without this ADR. Same code in 10 products is a guaranteed drift vector and a doubled surface for a bug in the comparison path (the non-constant-time == case). Rejected.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

Related

  • ADR-008 — why a single library for this kind of primitive.
  • ADR-030 — metrics split out of the base observ package; this ADR adds an observ primitive that the metrics endpoint uses.
  • ADR-045 — the integrate CLI is the tool that wires this everywhere.
  • observability-runbook.md — operator procedure "I want to enable bearer-auth on a new product's /metrics".
  • observability-stack-topology.md — updated "Where the secrets live" table.
  • dark-factory#269 — the issue this ADR closes.