049 — Bearer auth at Caddy for alertmanager.gyrum.ai

Date: 2026-04-22 Status: Accepted Closes: Part of #268 (alertmanager portion) Related: ADR-048 (Caddy site codification)

Context

Issue #268 was filed describing alertmanager.gyrum.ai and ntfy.gyrum.ai as "publicly exposed with no auth", severity HIGH.

On investigation today (2026-04-22) the picture differentiated:

Alertmanager: severity is actually CRITICAL, not HIGH. Anonymous POST /api/v2/silences succeeded — I verified by silencing a fictional alert. That means an attacker could disable monitoring before an attack (silence everything, then breach, then no alerts fire). They could also exfiltrate alert content (which leaks internal hostnames, fault patterns, runbooks).
Ntfy: severity is LOW. Anonymous POST /<random-topic> returns 403 — ntfy's own per-user/per-topic auth (anonymous role: no access to any topics) is enforcing. The "no Caddy auth" is shallow defense-in-depth, not an actual hole.
Internal blast radius of locking alertmanager: zero. Prometheus reaches Alertmanager at localhost:9093, never through Caddy. Alertmanager → ntfy uses ntfy's own access token. Nothing else hits alertmanager.gyrum.ai. (Verified: empty Caddy access logs for the past hour for that host.)

So we can lock alertmanager down hard with no user-visible impact. Ntfy stays as-is for now (its own auth layer is sufficient).

Decision

Add bearer-auth at Caddy for alertmanager.gyrum.ai, mirroring the existing pattern for loki.gyrum.ai. The token lives in /deploy/.secrets/alertmanager-bearer.token on dark-factory-ops and is provisioned by stage 12 from __ALERTMANAGER_BEARER_TOKEN__ in vars/<host>.env.

alertmanager.gyrum.ai {
    @authorized header Authorization "Bearer __ALERTMANAGER_BEARER_TOKEN__"
    handle @authorized {
        reverse_proxy localhost:9093
    }
    respond 401
}

The operator's local ~/.config/gyrum/alertmanager.env carries the token for any laptop tools that need to query the API (silence management, alert inspection).

Consequences

Good:

Anonymous silence creation/deletion is no longer possible. The most severe pre-attack "disable monitoring" vector is closed.
Anonymous read of current alerts/silences also gone — eliminates the info-disclosure side too.
Pattern is identical to loki.gyrum.ai (already shipped in ADR-048), so the operator doesn't have a third bespoke auth model to learn.
Stage 12 enforces the token at bootstrap — fresh ops VPS gets the same lockdown automatically.

Trade-offs:

Operator now needs the bearer in any HTTP tool they use to inspect alerts (curl, browser plugin, Grafana's Alertmanager datasource if ever added). Documented in observability-runbook.md.
Token rotation is a multi-step ceremony (generate, drop on ops VPS, reload Caddy, distribute new token to operator clients). The runbook covers this; not painful for a token that doesn't rotate often.

What this does NOT solve:

Ntfy stays without Caddy bearer auth — its app-level auth is sufficient for the current threat model. We could add Caddy bearer later for defense-in-depth but it's not load-bearing.
Token revocation — if the bearer leaks, manual token rotation + Caddy reload is the only remediation. No JWT, no OIDC, no refresh-token dance. Acceptable because the population of clients is "the operator" today.
Audit log of who silenced what — Alertmanager's API doesn't identify the bearer-holder. Once we have multiple operators, a reverse-proxy that injects a header per-bearer would help; out of scope here.

Alternatives considered

Cloudflare Access in front of alertmanager.gyrum.ai. Rejected for now: same reasoning as ADR-048 — wider Cloudflare Access setup is a parallel project, doing it for one host first creates two auth models on the same VPS. When CF Access lands for grafana.gyrum.ai, alertmanager rides along.
mTLS instead of bearer. Rejected: client cert distribution is harder than "drop a token in an env file" for the single-operator case. Worth revisiting at multi-operator scale.
IP allow-list like grafana.gyrum.ai uses. Rejected: same operator-IP-rotation pain that motivated rotate-grafana-allowed-ip.sh, plus bearer is a stronger guarantee (knowledge factor + can't be spoofed via header injection upstream).
Block alertmanager.gyrum.ai entirely and require SSH-tunnel for any access. Rejected: no convenient flow for operator alert inspection from a phone, browser, or Grafana plugin.
Take the path the security audit had documented in docs/security-audit-2026-04.md — the audit suggested respond 401 with no bypass; equivalent to "block entirely". Rejected for the same reason as #4.

Verification

# Stage 12 still passes its smoke test (16 assertions):
bash infrastructure/test/test-caddy-sites-stage.sh

# Live behaviour after applying:
curl -s -o /dev/null -w "no-auth: %{http_code}\n" \
  https://alertmanager.gyrum.ai/api/v2/silences
# expect: 401

curl -s -o /dev/null -w "with-auth: %{http_code}\n" \
  -H "Authorization: Bearer $(cat ~/.config/gyrum/alertmanager.env | cut -d= -f2)" \
  https://alertmanager.gyrum.ai/api/v2/silences
# expect: 200

# Internal Prometheus → Alertmanager pipeline unchanged:
ssh deploy@178.104.63.31 'curl -s http://localhost:9093/-/healthy'
# expect: OK