← Decisions operate mode →

Decisions

ADR-122 — Safe-cutover pattern: cert pre-acquire + read-only window

Today's warp-01 → warp-02 cutover failed twice: cert acquisition raced with DNS propagation (Caddy http-01 challenge couldn't validate because Let's Encrypt's resolver still saw old IP), and any writes during the…

#122

ADR-122 — Safe-cutover pattern: cert pre-acquire + read-only window

Status: Proposed (2026-04-26)

Context

Today's warp-01 → warp-02 cutover failed twice: cert acquisition raced with DNS propagation (Caddy http-01 challenge couldn't validate because Let's Encrypt's resolver still saw old IP), and any writes during the snapshot window risked data loss.

Two failure modes the original cutover playbook didn't address:

  1. ACME race — Caddy on destination needs cert for the hostname. http-01 challenge requires DNS to already point at the destination → chicken-and-egg during DNS flip.
  2. Split-brain writes — clients with stale DNS cache POST to source between snapshot and DNS-cut → those writes never reach destination.

Decision

Two structural primitives, paired:

1. Cert pre-acquire via DNS-01 (Cloudflare)

  • New role task roles/warp/tasks/cert-preacquire.yml uses acme.sh + Cloudflare API token
  • DNS-01 proves ownership via TXT record (no HTTP roundtrip → no DNS-flip dependency)
  • Cert + key land in /etc/caddy/certs/<hostname>.{fullchain.crt,key} (host-mounted into caddy container)
  • Auto-renewal handled by acme.sh's own cron (~60-day cycle)
  • Opt-in via warp_cert_preacquire: true (default off for fresh-host installs that use http-01)

2. Read-only window via Caddy 503-to-writes

  • New role task roles/warp/tasks/readonly-toggle.yml swaps the warp vhost site config
  • Readonly snippet: returns 503 for POST/PATCH/PUT/DELETE; reads pass through normally
  • Toggle modes: warp_readonly_state: enabled | disabled
  • Caddy reload is ~50ms; reversible (.bak file restored on disable)
  • Brief write outage (30s-2min during snapshot+restore) — acceptable for internal tool

Cutover sequence (safe)

1. Pre-acquire cert on destination via DNS-01     (warp #365)
2. Set source readonly: enabled                   (warp #368)
3. Verify writes blocked (assert)
4. pg_dump on source → pg_restore on destination
5. Verify count parity (assert)
6. Cut DNS → destination
7. Verify destination serves writes (smoke POST + DELETE)
8. PAUSE for operator confirm
9. Destroy source (separate auth)
```text

## Consequences

**Positive:**
- Today's two failure modes (cert race, split-brain) become structurally impossible
- Brief 30s-2min write outage is acceptable for internal tools (warp, factory, admin UIs)
- Same pattern reusable for any project's cutover (project-deploy role inherits)
- Rollback path defined per phase (lift readonly, revert DNS)

**Negative:**
- Adds CF_API_TOKEN + CF_ZONE_ID vault deps to host setup
- acme.sh is shell-script (not battle-hardened like certbot) — mitigated by maturity (10y old, used by Cloudflare's own tooling)
- Brief read-only window visible to clients — mitigated by clear 503 message + retry semantics
- Renewals via acme.sh cron are out-of-band from playbook fires — mitigated by /etc/caddy/certs being host-persistent + caddy reloading on cert change

## References
- warp #365 (cert pre-acquire structural fix)
- warp #368 (read-only window structural fix)
- warp #357 (cutover playbook — extends with these)
- warp #321 (deploy postmortem — motivating saga)
- ADR-088/089 (destroy-needs-silence — same "do prereqs before destructive step" shape)
- ADR-092 (host owns edge — extends with cert lifecycle)

## Alternatives considered

**certbot instead of acme.sh** — certbot is the canonical Let's Encrypt client; rejected because (a) it's a system package (apt install adds OS-level deps) where acme.sh is a single shell script with no install footprint; (b) acme.sh's Cloudflare DNS-01 module is well-maintained and matches Caddy's own DNS provider naming. acme.sh is mature (10+ years, used by Cloudflare's own tooling).

**Caddy with custom build (xcaddy + caddy-dns/cloudflare plugin)** — would put DNS-01 inside Caddy's own ACME flow. Rejected because (a) requires custom Caddy image build per fleet release, (b) Caddy's automatic-renewal is in-process (less observable than acme.sh cron), (c) ties cert lifecycle to Caddy upgrades. Pre-acquired static certs decouple the two.

**Cloudflare Origin Certificate (CF-issued, valid for any IP)** — rejected because today's `proxied=False` setup means browsers connect directly (CF Origin Certs aren't browser-trusted without proxy). Switching to `proxied=True` would also work, but adds CF as a critical-path dependency on the request-response loop.

**Application-level read-only mode (WARP_READ_ONLY env in warp Go code)** — rejected because (a) requires warp deployment + rolling restart to toggle, (b) brief downtime when restarting, (c) can't be reused for non-warp services. Caddy 503-to-writes is generic across all gyrum services + zero-downtime toggle.

**Stop warp container entirely during snapshot (Option γ from postmortem)** — rejected because total outage (reads + writes) for the whole window. Caddy 503-to-writes preserves reads, only blocks the operation that could lose data (writes).

**Wait + retry on ACME race (no structural fix)** — what today's failed cutover did. Rejected because timing-dependent (could work or not on any given attempt) and stale-cache writes still get lost during the window.