ADR-056: Caddy per-product routing — guard against backend-port collision; codify only placeholders

Status: Accepted Date: 2026-04-22

Context

On 2026-04-22 the Platform UI's Operate-mode screen flagged research.gyrum.ai and guardian.gyrum.ai as reporting identical 11.36 % error rate and 21.7 ms p95 — suspiciously identical. The cause: both /deploy/caddy/sites/research-gyrum-ai and /deploy/caddy/sites/guardian-gyrum-ai on gyrum-prod-1 contained reverse_proxy localhost:8082. Research has no container on that host (docker ps | grep research returns nothing). Guardian owns :8082. So every request to research.gyrum.ai was transparently served by guardian's binary — including /metrics, where the service="guardian" label revealed the truth on inspection but did not in any way interrupt the traffic.

The collision was structurally invisible to everything we had:

Caddy validate — passed, because two different hostnames proxying the same backend is valid Caddyfile.
Cloudflare — saw a TLS endpoint responding 200; no reason to alert.
Prometheus — scraped service="guardian" from the guardian container. The research scrape would have scraped the same container, producing duplicate-label scrape errors Prometheus silently drops (and which we do not alert on).
infrastructure/deploy.sh step 4 — cleans up stale hostname claims (two files both declaring research.gyrum.ai {). It does not inspect the reverse_proxy target. So whoever scaffolded the research site by copy-pasting guardian and forgetting to change the port shipped it, and no subsequent deploy caught the mistake.

The surrounding architecture is relevant. App-VPS Caddy sites (/deploy/caddy/sites/ on gyrum-prod-1) are generated dynamically on every deploy — each product's own infrastructure/deploy.sh run overwrites the file with the right port. They are deliberately not centrally codified: doing so would duplicate the deploy contract and create a merge-conflict surface across every product team. ADR-048 centralises the ops-VPS Caddy sites for a different reason (fixed set of four known hosts, no per-product deploy pipeline, role-gated bootstrap stage is the natural home).

So the question this ADR answers is:

Given that the dynamic-per-deploy model is the right default, how do we prevent a port collision from silently routing one product's traffic to another product's container?

And:

Given that research.gyrum.ai has DNS and a public expectation but no container, what does Caddy serve for it?

Decision

Three complementary changes, none of them a full codification of app-VPS sites.

Backend-port collision guard in infrastructure/deploy.sh step 4. Before writing a new site file, grep every other file in /deploy/caddy/sites/ for reverse_proxy localhost:<same-port>. If any match, fail the deploy with a clear message that cites this ADR and the runbook. Escape hatch: CADDY_ALLOW_PORT_SHARE=1 for deliberate aliasing (rare). This catches the exact class of bug we hit: copy-pasted site files.
Placeholder directory at infrastructure/caddy/sites-placeholders/ containing Caddy site files for hostnames that are publicly routable but not yet deployed. These files return HTTP 503 with a short "not deployed" message. Today the directory holds one file (research-gyrum-ai); it will grow by one file whenever we stand up DNS ahead of a deploy. Its README documents the one-off scp + reload install procedure.
Runbook at docs/runbooks/caddy-per-product-routing.md for the collision alert pattern: how to detect it, which three causes to rule in/out, which fix applies in which case.

The live fix (2026-04-22) swapped /deploy/caddy/sites/research-gyrum-ai on gyrum-prod-1 from reverse_proxy localhost:8082 to the 503 placeholder. curl https://research.gyrum.ai/metrics now returns 503, not service="guardian".

Consequences

Good:

The exact failure mode we hit (copy-pasted site with wrong port) is now impossible to deploy. The guard runs before the site file is written, so even a well-meaning manual deploy.sh invocation can't slip it past review.
research.gyrum.ai returns an honest 503 instead of impersonating guardian. Operators and scrapers get a clear signal; no accidental data-surface confusion if research ever handles auth in the future.
The runbook turns "this dashboard looks suspicious" into a five-step triage with named causes and fixes.
Placeholder pattern generalises — any future "DNS before container" situation has a home and a documented install path.

Trade-offs:

Deploy now fails loudly in one new scenario. Anyone genuinely wanting to front one backend with multiple hostnames has to set CADDY_ALLOW_PORT_SHARE=1. We believe this is the right trade (false positives are one env-var away; false negatives are silent data leaks), but it is a behavioural change.
Placeholders are installed out-of-band (ssh + scp), not by any pipeline. That's a deliberate scope limit — see alternative (d) below. The cost is that adding or removing a placeholder is a manual operation; the benefit is no new daemon, cron, or bootstrap stage owning app-VPS Caddy state.
The collision check is a simple grep — not a full Caddyfile parser. A sufficiently exotic site file (multi-line reverse_proxy block, upstream @matcher with embedded port) could fool it. The whole fleet today uses the flat reverse_proxy localhost:<port> shape the deploy script emits, so this is acceptable; we revisit if that assumption breaks.

Alternatives considered

(a) Full codification of app-VPS site files — mirror the ops-VPS pattern from ADR-048 into a new caddy-sites-app/ directory plus a new bootstrap stage. Rejected for now. Every product team would have to PR dark-factory to change its own site config, and that PR would have to land before the deploy that needs it. That re-introduces the coupling the dynamic-per-deploy model was built to avoid, and it duplicates the reverse_proxy localhost:<port> line in two places (template + deploy.sh heredoc) which is its own drift risk. The port-collision guard in (b) gives us 90 % of the safety at <1 % of the cost.
(b) Port-collision guard alone — just add the grep to deploy.sh and call it done; don't codify research's placeholder and don't write a runbook. Rejected. Does not answer the research.gyrum.ai question (it would still have 404'd after cleanup, which is less clear than 503 with a message) and leaves no written triage pattern for the next time this happens with a different hostname.
(c) Runbook + placeholder only, no deploy.sh guard — treat the collision as an operational concern detectable via the Operate UI, not a deploy-time concern. Rejected. This makes the collision a monitoring problem rather than a prevention problem. The whole reason the incident went unnoticed for hours is that our monitoring was the detection surface, and even then it only showed up as "two hostnames with suspiciously identical numbers" — a human had to notice the coincidence. Prevention at deploy time is cheaper and more reliable.
(d) Placeholder-sync daemon / bootstrap stage — add an app-VPS bootstrap stage that reconciles sites-placeholders/ into /deploy/caddy/sites/ the same way stage 12 does for the ops VPS. Rejected. The placeholder set is tiny (one entry today) and turns over slowly. A daemon owning a subset of /deploy/caddy/sites/ creates an unclear division of authority with deploy.sh — if a real deploy races the reconcile, whose write wins? An out-of-band scp install, documented in the placeholders README, is cleaner until we have more than a handful of placeholders.
(e) Default catch-all 503 for any unclaimed hostname — configure the root Caddyfile with a wildcard block that returns 503 for any host not matched by a sites/* import. Rejected. Caddy already returns 404 for unknown hosts; changing that default hides real misconfigurations (a site file silently removed becomes 503 instead of 404, indistinguishable from the "not yet deployed" case). The placeholder pattern keeps "deliberately 503" explicit in git.

Supersedes: none Superseded by: