ADR-037: SafeFetch denies private ranges by default; opt-out per call

Status: Accepted Date: 2026-04-21

Context

Distill's YouTube handler takes a user-supplied URL and dereferences it server-side. So do (or will) summarisers, feed fetchers, OEmbed integrations, and any future "paste-a-link" workflow. Every one of these is an SSRF primitive waiting to be weaponised:

• An attacker who can POST a URL can enumerate internal services through the pod network.
• 169.254.169.254 is the AWS/GCP/Azure metadata endpoint — fetching from it on a container with instance-role credentials leaks them.
• Cloud-hosted databases are usually reachable on 10.0.0.0/8; an attacker who can GET them may be able to dump via error messages.
• IPv6 opens further private ranges (::1, fc00::/7, fe80::/10).

The naive mitigation — "reject URLs on the private-IP list before calling http.Get" — misses DNS rebinding and open redirects on public hosts. The correct mitigation is: parse, resolve, vet every hop, and re-vet on redirects.

Decision

pkg/security.SafeFetch(ctx, rawURL, opts...) is the only approved primitive for dereferencing user-supplied URLs from gyrum services. It applies three layers of defence:

1. Scheme allow-list. Only http and https. file://, ftp://, gopher://, javascript: and friends are rejected with ErrScheme.
2. DNS-then-deny. Host is resolved via net.LookupIP; every A/AAAA answer is checked against a deny list (loopback, RFC1918, link-local, unique-local, multicast, unspecified, and IPv6 equivalents). Any hit returns ErrPrivateIP.
3. Re-check on redirect. http.Client.CheckRedirect re-runs the entire validation on every hop up to five hops (overridable).

The opt-out knob is AllowList(hosts...) — a per-call narrowing, not a per-call relaxation. A caller who pins to api.openai.com is STILL subject to the private-IP deny (a compromised DNS record for api.openai.com pointing to 10.0.0.5 is rejected).

The test-only allowLoopbackForTest() helper is unexported: production code cannot opt out of the loopback deny.

Consequences

Easier:

• Every consumer that would otherwise reach for http.Get on user input has a drop-in safer alternative.
• Canonical errors mean callers can branch on errors.Is(err, security.ErrPrivateIP) to produce useful UI messaging ("URL must be publicly reachable") without parsing strings.
• Redirect checking survives the most common bypass (public host 302s to private IP).

Harder:

• httptest.NewServer binds to 127.0.0.1, so every test using SafeFetch against a test server has to go through an internal test hook. Intentional; avoids a production flag that could be misused.
• A service genuinely needing to fetch from an internal allowlist (e.g. a pod-local sidecar) cannot use SafeFetch. Such services must reach for http.Client directly and document WHY the deny is inappropriate — with an explicit review.

Signed up to operate:

• The deny list is static code. New private ranges (e.g. IPv6 carrier- grade NAT fc00::/7 as unique-local is already covered; future RFC updates may add more) require a release.
• Security standards doc mandates SafeFetch for any handler taking a URL from user input. Code review gates on this.

Alternatives considered

• Deny list as configuration — rejected. Config drift is a classic source of security regressions. Ranges change rarely; a code change with an ADR is an appropriate cadence.
• Explicit allow-list only (no default deny) — rejected. Every new integration would need an allow-list entry; missing entries would be rejected loudly, so "fail safe" in one sense — but the allow-list becomes a mis-specified mess, and common cases (fetching arbitrary public pages for summarisation) don't have a natural allow-list at all.
• Resolve the host and call http.Get with an IPAddr directly — rejected. Many CDN hosts route based on SNI/Host header; substituting an IP breaks TLS certificate validation and HTTP virtual hosting. The CheckRedirect approach preserves host-based routing while still validating each hop.

Supersedes: none Superseded by: —