ADR-073: Preview environments per feature branch

Status: Accepted Date: 2026-04-23 Related: ADR-059 (project-first IA — multi-repo projects), ADR-067 (playbooks unified primitive), ADR-068 (playbook runtime architecture), ADR-069 (playbook security), ADR-070 (playbook UI contract), ADR-071 (product name — Weft), ADR-072 (infrastructure as playbooks), ADR-075 (Fleet-E2E cross-product branch-testing service — Phase D's Playwright source and its CI-plane counterpart)

Decision (one paragraph)

Every feature branch opened as a pull request on any repo inside a project (ADR-059) spins up its own ephemeral, full-stack preview environment at a private, Access-gated URL. The preview is composed from production Dockerfiles, namespaced per-PR, gated by Cloudflare Access behind the PR author and its reviewers, and torn down on PR close, PR merge, or a 72-hour cron-enforced timeout — whichever comes first. It is expressed as two Weft playbooks — release_flow: preview-up and release_flow:preview-down — composed from the ADR-072 infra sub-playbook catalogue (cloudflare-dns-upsert, cloudflare-tunnel-route-upsert, cloudflare-access-app-upsert, the systemd/caddy pair). Cross-repo coordination is name-match by branch: when spinning up a preview for PR feat/dark-mode on buzzy-frontend, every other repo in the same project is checked for a branch of the same name, and if present it is used instead of main. Capacity is one Hetzner box with FIFO queueing and a four- concurrent cap for Phase 1; overflow is old previews auto-torn-down after 24 hours if a newer PR is queued. We ship in four phases (manual invocation → webhook-driven → /ops/previews UI → Playwright-against-preview in the AI review flow); Phases A and B are committed scope, C and D are named horizons.

Context

Testing a feature that spans buzzy-frontend and buzzy-backend today — and increasingly, features that span three or four repos inside one project (ADR-059) — means one of two things, and neither is good:

1. Merge and hope. The frontend PR and the backend PR each pass their own CI and their own reviewers, each merges to main, and the first time the two pieces run against each other in the same address space is on the production host five minutes later. If the API contract the frontend assumed is not quite the contract the backend shipped, the breakage is live and nobody saw it during review. The "is it tested in context" question is answered by the failure, not before it.
2. Run locally and hope it matches prod. The reviewer git checkouts both branches, runs docker compose up, and clicks around. Sometimes the reviewer has a different Node version. The reviewer's local database is not prod-shaped. The reviewer's browser is not sending Access headers. The operator whose review comment is "LGTM, looks fine locally" has answered a question the reviewer was not asked — how does it behave on the fleet — with evidence from a different system.

The common industry answer to this is review apps (or "preview environments", "PR environments", "ephemeral environments"): every open pull request gets a full-stack, publicly (or privately) addressable copy of the system running the PR's code; humans and AI reviewers exercise it end to end; teardown happens automatically on PR close. Heroku invented the pattern, Vercel scaled it to 10M developers, GitLab and Render and Netlify ship it as a checkbox.

Gyrum does not ship it today. The reasons are accidental, not principled: we did not have the primitives (ADR-067/068 landed this cycle), we did not have the infra composition (ADR-072 just merged), we did not have the multi-repo project model (ADR-059 landed a week ago), and we did not have the production Dockerfiles (the Dockerfile-per-repo work is in-flight this week). The accidental absence means every PR review in the last three months has been lower-evidence than it could have been. That is a compounding cost: review confidence is lower, regressions are caught later, and the "merge to main to find out" muscle builds habit that preview environments specifically exist to unlearn.

Three converging pressures force the decision now:

1. The eight-repo project is on the runway. Buzzy (ADR-059 reference project — Product class) already spans buzzy-frontend and buzzy-backend; the mobile repo is planned. Focus spans three. Two features in the current backlog touch three repos each. The mental load of coordinating review across three open PRs without a shared preview is what breaks reviewers.
2. ADR-072 shipped the sub-playbook catalogue. The eight sub-playbooks in ADR-072 §5 are exactly the pieces a preview environment composes: one DNS record, one tunnel route, one Access app, one Caddy site, one systemd-or-docker origin. A preview environment is a deploy-service (ADR-072 §6) with a per-PR namespace, a tighter TTL, and a more restrictive Access policy. We get preview environments nearly for free if we commit to the pattern before other patterns ossify.
3. Production Dockerfiles are landing. The Dockerfile-per-repo PR across buzzy-frontend, buzzy-backend, distill-gyrum-ai, and the others lets us build a single OCI image per commit that the preview stack can run. Without it we would need a per-repo "dev server" mode that is almost-but-not-quite production — exactly the divergence preview environments exist to eliminate. With it, preview runs the same image production will run. The only difference is the URL and the Access policy.

The fleet E2E compose (distill-gyrum-ai's e2e-integration/) is the shape we generalise: a docker-compose stack that boots the production images of every service the test touches, wires them together, and exercises them with Playwright. A preview environment is the same shape with per-PR namespacing on the compose project name, the port block, the DNS record, the tunnel route, and the Access app — and without the Playwright runner (Phase D re-introduces it).

This ADR commits the pattern and specifies the two playbooks that implement it.

Why not Vercel / Render / Heroku?

The obvious question and the right one to answer early. See §11 for the full argument; the short version is that Vercel covers the frontend beautifully and has no answer for the backend, Render and Heroku both require moving services off Hetzner (loss of the ADR-020 "shared infra on our VMs" stance), and all three introduce a third-party billing surface whose cost scales with PR rate. We already run the primitives (Cloudflare Tunnel, Access, Caddy, Weft runtime) that compose into preview environments; adopting a vendor for the preview story means we operate two preview systems (the vendor's for the frontend, our own for the backend) or we offload the whole thing and lose the control surface ADR-072 just committed to. We do neither.

Why now — and not after we hit pain?

Every PR merged without a preview is a data point that makes the "merge to main to find out" habit sturdier. The cost of adopting preview environments grows superlinearly with the rate of multi-repo PRs; the benefit starts the first time a reviewer catches a cross-repo regression on the preview URL instead of on production. Waiting for a concrete production incident to justify the work — the default we apply to most infra investments — is wrong for this class of tool, because the incident is exactly what preview environments prevent. We name this explicitly so the reviewer of this ADR does not ask "what breakage does this fix?" (the answer is: the breakage we are not yet measuring because we don't look for it during review).

The pattern

The pattern has six moving parts. Every one composes on pieces we already own: projects (ADR-059), playbooks (ADR-067/068/069), Cloudflare primitives (ADR-072), production Dockerfiles (the current Dockerfile-per-repo PR), and the fleet E2E compose shape.

1. Branch coordination — name-match across the project's repos

When a preview is requested for a PR feat/dark-mode on repo buzzy-frontend, the preview stack needs to know which version of every other repo in the buzzy project to use. The rule is:

For every repo listed in the project's repos: (ADR-059 §4), look for a branch named feat/dark-mode. If the branch exists on that repo, use its HEAD SHA. Otherwise use main on that repo.

This is the name-match convention and it is load-bearing. We commit to it rather than to an alternative (branch annotations, a .gyrum/preview.yaml declaring the cross-repo branch mapping, manual operator input) for one reason: it matches how operators already work. When an operator writes a feature that touches both the frontend and the backend, they create the same branch name on both repos — that is the existing muscle memory, not a new discipline we need to teach. The name-match convention codifies the muscle memory rather than layering a new annotation on top of it.

The coordinator is the projects-know-repos primitive already satisfied by gyrum-projects (ADR-061). The preview-up playbook takes a project_slug input, reads projects/<slug>.yaml, iterates its repos: list, and for each repo queries the GitHub API for the branch by name. The output of this phase is the repos_with_branches[] input for the rest of the playbook — a list of {repo, sha} tuples that is the exact thing docker-compose needs in order to pull the right images.

Edge cases, committed:

• Branch exists on an unrelated repo. If buzzy-frontend has feat/dark-mode and ai-research-run (a Platform repo not in Buzzy's project but which has a coincidentally-named branch) does too — we only consult repos listed in the triggering project's repos:. Cross-project branch-name collision does not matter.
• Branch exists with the same name but a different author. The author of the triggering PR is taken from the GitHub webhook; we do not cross-check authorship against the name-matched branch on the other repo. If two operators chose the same branch name on two different repos for unrelated work, they collide — but that is the same breakage they would have if they both tried to work on the same branch name on the same repo. Not a new failure mode.
• Branch deleted mid-preview. Rare. If a name-matched branch on a sibling repo is deleted between preview-up and the next webhook-driven redeploy, the next redeploy falls back to main for that repo. The preview URL does not change; the composition does. We log this as a preview_repo_fallback event so the operator can notice.

2. Naming — preview URLs

Every preview gets a stable URL for the lifetime of the PR. The URL format is:

https://<slug>-pr<N>.preview.gyrum.ai

Where <slug> is the triggering project's slug (ADR-059 §4 — e.g. buzzy, focus) and <N> is the PR number on the triggering repo. Example: buzzy-pr142.preview.gyrum.ai.

Why not <shortsha> in the hostname? We considered preview-<pr-number>-<shortsha>.gyrum.ai and rejected it for three reasons. First, the SHA changes on every push, which would mean a new DNS record, a new Tunnel route, and a new Access app for every commit — three extra mutations per push. Second, a stable hostname lets the PR description link the preview URL once and have it keep working for the life of the PR; reviewers who bookmark it do not get stale links. Third, the SHA adds zero isolation — the Access policy is already per-PR-author-plus-reviewers, not per-SHA. The SHA goes in the preview environment's /api/version output (ADR-072 verify_sha step), which is where operators check "am I looking at the right commit?"

Why a dedicated preview.gyrum.ai subdomain (not *.gyrum.ai or pr142.buzzy.gyrum.ai)? The subdomain namespace is the isolation boundary. A lint-bypass bug in a preview playbook that somehow issues a DNS record for *.gyrum.ai is constrained to the preview.gyrum.ai zone by scoped tokens (ADR-072 §2.4). Production hostnames (buzzy.gyrum.ai, weft.gyrum.ai) live on gyrum.ai directly; the two zones are separate Cloudflare zones with separate API tokens. A preview environment cannot, by construction, mint a DNS record that affects production. This pattern mirrors the weft.gyrum.ai pattern ADR-071 §5 introduced (the product's own subdomain is its own namespace).

Apex preview.gyrum.ai itself is a static landing page (Caddy-served) listing "what is this?", "how do I get access?", and a contact link. Not a bare NXDOMAIN — that would confuse search engines and any operator who hits the apex by accident.

3. Isolation — each preview is its own namespace

No two previews can contaminate each other. Contamination includes: shared database state, shared ports, shared tunnel routes, shared Access policies, shared Caddy site blocks, shared filesystem. The isolation contract is:

• Docker-compose namespace. Each preview has a compose project name of preview-<project_slug>-pr<N> (e.g. preview-buzzy-pr142). All containers, networks, and named volumes are prefixed by this. docker compose -p preview-buzzy-pr142 up on the Hetzner box lives in its own network namespace; inter-preview traffic is impossible without cross-namespace routing we never set up.
• Port block. Each preview gets a dedicated 10-port range, assigned at preview-up time from a pool. The range is stored in a preview_port_allocations table in Postgres (a new one-off table owned by the preview-up playbook; lives in dark-factory's application DB alongside playbook_runs). Allocation is first-fit; deallocation on teardown. The pool size is 100 ports in blocks of 10 — enough for 10 concurrent previews even though the capacity cap (§8) is 4 today.
• Ephemeral Postgres. Each preview's docker-compose.yml declares its own Postgres instance, in its own container, with its own named volume. The volume is named preview_buzzy_pr142_pg_data, prefixed as above, and is deleted at teardown. No preview reads from the production database; no preview writes to a shared staging database.
• Tunnel route — one per preview. cloudflare-tunnel-route-upsert (ADR-072 §5.3) adds <slug>-pr<N>.preview.gyrum.ai → http:// localhost:<preview-port-8080> to the ingress list of a dedicated preview tunnel (preview.gyrum.ai tunnel, separate from the production tunnel). One tunnel, many ingress entries, each entry scoped to one preview. Concurrent preview-up runs race on the tunnel config; ADR-072 §5.3's If-Match: ${etag} pattern serialises them, retry on conflict.
• Access app — one per preview. cloudflare-access-app-upsert (ADR-072 §5.4) creates an Access app for the preview's hostname with a policy allowing the PR author's email plus the emails of the PR's reviewers (read from GitHub API at preview-up time).
• Caddy site — one per preview. caddy-site-upsert (ADR-072 §5.6) adds a site block for the preview hostname on the Hetzner box, reverse-proxying to the preview's allocated port. This is how the tunnel termination inside the Hetzner box reaches the right container.
• Filesystem. All preview state — compose files, per-preview .env files with the generated secrets — lives under /var/lib/gyrum/previews/preview-<slug>-pr<N>/. Teardown rm -rfs the directory. A preview cannot read another preview's state.

No cross-preview state. No "staging is shared so everyone sees everyone's data" footgun. The cost is 2GB of RAM per preview (§8) rather than the ~500MB a shared-staging environment would use; we take that trade explicitly.

4. Access — always gated, never public

Preview URLs are not public. Every preview's Access policy (cloudflare-access-app-upsert, ADR-072 §5.4) includes:

• The PR author's email (from GitHub API).
• The email of every explicit reviewer on the PR (from GitHub API at preview-up and on every webhook-driven redeploy, so adding a reviewer grants them access within ~10 seconds of the review- request event).
• Any email matching the @gyrum.ai domain (internal team access for cross-PR review).

Never public. An operator who clicks the preview URL without being signed in to Access gets the standard Cloudflare Access login page, authenticates, and — if their email does not match the policy — sees the "you are not authorised" page. The URL is not leaked to search engines (via X-Robots-Tag: noindex set by Caddy at the preview site block) and the preview origin does not serve any response before Access has satisfied the policy at the edge.

Why not optional? Because the preview environment runs unauthenticated code paths (migration scripts, seed data generation, admin endpoints exposed for preview debugging) that are absolutely not safe to put on the public internet, even behind a long hostname. The policy is "gated by default, no opt-out"; a preview environment that a reviewer wants to share with an external collaborator is a new explicit cloudflare-access-policy- upsert call to add the collaborator's email, not a flag to make the preview public.

5. Lifecycle — the four transitions

Every preview has exactly four lifecycle transitions. Each is a trigger that invokes one of the two playbooks:

The 72-hour timeout is always enforced, no override. A PR that needs its preview alive longer than 72 hours needs a new push (which resets the clock, per preview_up_at = now() on every preview-up run). This is deliberate: a preview that stays alive for a week is a preview the reviewer has lost interest in, and it is consuming capacity (§8) another reviewer needs. The forced refresh-or-die pattern prevents zombie environments. We chose 72h rather than 24h or 7d after looking at what other teams do (Render defaults to 72h inactivity; Vercel has no limit but they have unlimited capacity; Heroku legacy was 24h and operators complained): 72h is long enough for a weekend review cycle, short enough to force operator action rather than indefinite drift.

Idempotency. Both playbooks are idempotent in the ADR-072 §2.1 sense. preview-up on a preview that already exists with the same SHAs and reviewers is a no-op: every sub-playbook reports changed: false. preview-up after a webhook push with a new SHA computes a delta on the deploy_app step and applies only that step — DNS, Tunnel, Access, Caddy all report changed: false. preview-down on a preview that does not exist is also a no-op. This matters because webhooks can be delivered more than once (GitHub retries); the idempotent playbook makes duplicate delivery safe without an idempotency-key layer.

6. Authenticated ingress — webhook secret + replay safety

The GitHub webhook that drives the lifecycle posts to a small endpoint (Phase B — §6 below). That endpoint MUST verify:

• The X-Hub-Signature-256 HMAC over the request body, using a webhook secret stored in the ADR-069 vault as vault.github_ preview_webhook_secret. A request without a valid signature is dropped with 401 and a playbook_run_events entry of preview_webhook_rejected.
• The X-GitHub-Delivery id is recorded in a preview_webhook_deliveries table. A second delivery with the same id is acknowledged with 200 (to stop GitHub's retry loop) but does not re-invoke the playbook — idempotency-by-id belt-and- braces on top of the playbook's own idempotency (§5).
• The repository.full_name and pull_request.base.repo.owner. login match the allow-list of repos we accept webhooks for. Only repos listed in any projects/*.yaml under repos: are accepted. An unknown repo's webhook is rejected with 404 (not 401 — the signature may be valid, the sender is just the wrong sender).

The webhook secret is rotated every 90 days; rotation is a separate playbook (github-webhook-secret-rotate) following the ADR-072 scoped-token pattern. Out of scope for this ADR.

The two new playbooks — specs (not full YAML)

Both playbooks live in dark-factory/playbooks/release_flow/ — they are kind: release_flow (ADR-072's deploy-service is also this kind), distinct from kind: service_runbook (the infra sub- playbooks). Release flows orchestrate; sub-playbooks mutate one surface.

Full YAML bodies are explicitly out of scope for this ADR and land in two follow-up PRs (one per playbook, each ≤ 800 lines per ADR- 067's PR cap), each citing ADR-073.

A. release_flow:preview-up

---
id: preview-up
kind: release_flow
title: "Preview: bring up an ephemeral full-stack preview for a PR"
persona: sre                          # manual invocation today; webhook-driven Phase B
owner: "@jon"
status: active
trigger: "Invoked by the preview-webhook-receiver on pull_request.opened/synchronize/reopened/ready_for_review/review_requested events, or manually by an SRE for bootstrap/debug"
success_metric: "Preview reachable at https://<slug>-pr<N>.preview.gyrum.ai, returns the PR's HEAD SHA on /api/version, Access policy includes PR author + reviewers"
tags: [infra, preview, release_flow, release]

allowed_personas: [sre, owner]
allow_local_execution: false

# Union of sub-playbook required_scopes (ADR-072 §2.4 — computed at parse time)
required_scopes:
  - {vendor: cloudflare, scope: "Zone:DNS:Edit", zone_filter: "${env.CF_ZONE_ID_PREVIEW}"}
  - {vendor: cloudflare, scope: "Account:Cloudflare Tunnel:Edit"}
  - {vendor: cloudflare, scope: "Account:Access: Apps and Policies:Edit"}
  - {vendor: ssh, scope: "host:ops-vps:sudo:caddy"}
  - {vendor: ssh, scope: "host:ops-vps:sudo:docker"}
  - {vendor: github, scope: "repo:read"}

runtime:
  inputs:
    - {name: pr_number, type: int, required: true}
    - {name: project_slug, type: string, required: true}
    - {name: triggering_repo, type: string, required: true}
    - {name: dry_run, type: bool, default: false}

  steps:
    # 1. Resolve cross-repo branch state (the name-match rule)
    - id: resolve_repos_with_branches
      type: http                                  # GitHub API
      # for each repo in projects/<slug>.yaml :repos:, GET /repos/{r}/branches/feat/dark-mode
      # output: repos_with_branches[] = [{repo, sha}, ...]
      save_as: repos

    # 2. Resolve reviewers from the PR
    - id: resolve_reviewers
      type: http
      # GET /repos/{triggering_repo}/pulls/{pr_number} - pluck author + reviewers
      save_as: reviewers

    # 3. Allocate a port block (or reuse an existing allocation for this PR)
    - id: allocate_ports
      type: http                                  # calls the preview-registry service
      # POST /previews/{pr_number}/port-allocation — idempotent
      save_as: ports

    # 4. DNS — upsert <slug>-pr<N>.preview.gyrum.ai → preview tunnel
    - id: ensure_dns
      type: subplaybook
      playbook: service_runbook:cloudflare-dns-upsert
      with:
        zone_id: "${env.CF_ZONE_ID_PREVIEW}"
        name: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        type: CNAME
        content: "${env.CF_PREVIEW_TUNNEL_ID}.cfargotunnel.com"
        proxied: true
        dry_run: "${inputs.dry_run}"
      save_as: dns

    # 5. Tunnel — upsert an ingress entry for this hostname → preview caddy port
    - id: ensure_tunnel_route
      type: subplaybook
      playbook: service_runbook:cloudflare-tunnel-route-upsert
      with:
        tunnel_id: "${env.CF_PREVIEW_TUNNEL_ID}"
        hostname: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        service_url: "http://localhost:${ports.caddy_port}"
        dry_run: "${inputs.dry_run}"
      save_as: tunnel

    # 6. Access — gate the hostname behind author + reviewers + @gyrum.ai
    - id: ensure_access
      type: subplaybook
      playbook: service_runbook:cloudflare-access-app-upsert
      with:
        domain: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        name: "preview-${inputs.project_slug}-pr${inputs.pr_number}"
        allowed_emails: "${reviewers.emails + ['*@gyrum.ai']}"
        policy_name: "preview-author-and-reviewers"
        dry_run: "${inputs.dry_run}"
      save_as: access

    # 7. Render docker-compose.yml for this preview
    - id: render_compose
      type: shell
      target: host:ops-vps
      # generates /var/lib/gyrum/previews/preview-<slug>-pr<N>/docker-compose.yml
      # with per-repo image tags (repos_with_branches[].sha) and the allocated ports
      save_as: compose

    # 8. docker compose up (production images, per-PR namespaced)
    - id: bring_up_stack
      type: ssh
      target: host:ops-vps
      # docker compose -p preview-<slug>-pr<N> -f <compose_path> up -d --pull always
      when: "!inputs.dry_run"
      save_as: stack

    # 9. Caddy — reverse-proxy the preview hostname to the preview's caddy port
    - id: ensure_caddy
      type: subplaybook
      playbook: service_runbook:caddy-site-upsert
      with:
        host: ops-vps
        hostname: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        backend_url: "http://localhost:${ports.caddy_port}"
        dry_run: "${inputs.dry_run}"
      save_as: caddy

    # 10. Wait-ready — poll /api/version until it returns, max 90s
    - id: wait_ready
      type: shell
      target: local
      when: "!inputs.dry_run"
      # for i in 1..45; curl -sf https://<host>/api/version && exit 0; sleep 2
      timeout: 90s

    # 11. Verify — /api/version returns the triggering repo's HEAD SHA
    - id: verify_sha
      type: http
      when: "!inputs.dry_run"
      # GET /api/version — assert body.sha == repos.by[triggering_repo].sha
      save_as: verify

    # 12. Register — record the preview in preview_registry for the UI and reconciler
    - id: register
      type: http
      # POST /previews/{pr_number} {url, up_at, reviewers, repos}
      save_as: registered

  outputs:
    preview_url: "https://${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
    pr_number: "${inputs.pr_number}"
    up_at: "${registered.up_at}"
    repos: "${repos}"
    reviewers: "${reviewers.emails}"
    dns_changed: "${dns.outputs.changed}"
    tunnel_changed: "${tunnel.outputs.changed}"
    access_changed: "${access.outputs.changed}"
    caddy_changed: "${caddy.outputs.changed}"
    stack_changed: "${stack.outputs.changed || false}"
    dry_run: "${inputs.dry_run}"
---

Shape notes:

• Steps 4-6 and 9 are subplaybook: calls into the ADR-072 catalogue. Zero new infra mutators are introduced.
• Steps 1-3, 7-8, 10-12 are the preview-specific logic: cross- repo branch resolution, port allocation, compose rendering, docker up, verify, registry.
• dry_run threads through exactly as ADR-072 §2.2: the operator can run preview-up in dry-run mode to see what would happen without mutating the system. In Phase A this is how an SRE validates the playbook before letting webhooks drive it.

B. release_flow:preview-down

---
id: preview-down
kind: release_flow
title: "Preview: tear down an ephemeral preview and release its resources"
persona: sre
owner: "@jon"
status: active
trigger: "Invoked by the preview-webhook-receiver on pull_request.closed, or by the preview-reaper cron on 72h timeout"
success_metric: "DNS, Tunnel ingress, Access app, Caddy site, and docker-compose stack for the preview are all deleted; port block returned to pool"
tags: [infra, preview, release_flow, teardown]
destructive: true                       # ADR-069 §6 — all teardown playbooks are destructive

allowed_personas: [sre, owner]
allow_local_execution: false

required_scopes:
  - {vendor: cloudflare, scope: "Zone:DNS:Edit", zone_filter: "${env.CF_ZONE_ID_PREVIEW}"}
  - {vendor: cloudflare, scope: "Account:Cloudflare Tunnel:Edit"}
  - {vendor: cloudflare, scope: "Account:Access: Apps and Policies:Edit"}
  - {vendor: ssh, scope: "host:ops-vps:sudo:caddy"}
  - {vendor: ssh, scope: "host:ops-vps:sudo:docker"}

runtime:
  inputs:
    - {name: pr_number, type: int, required: true}
    - {name: project_slug, type: string, required: true}
    - {name: dry_run, type: bool, default: false}

  steps:
    # 1. Read preview registration (is there a preview for this PR?)
    - id: read_registration
      type: http
      # GET /previews/{pr_number}
      # if 404 → preview already down; remaining steps all no-op via when:
      save_as: reg
      on_error: continue

    # implicit approval step — ADR-069 §6 (destructive: true)
    # prompt: "Tear down preview for PR #${pr_number} at ${reg.url}? This is irreversible."
    # skipped when dry_run: true OR when reg.status == 404

    # 2. Docker compose down (stop and remove the stack + its volumes)
    - id: tear_down_stack
      type: ssh
      target: host:ops-vps
      when: "reg.status != 404 && !inputs.dry_run"
      # docker compose -p preview-<slug>-pr<N> -f <compose_path> down -v

    # 3. Caddy — delete the site block
    - id: delete_caddy
      type: subplaybook
      playbook: service_runbook:caddy-site-delete
      with:
        host: ops-vps
        hostname: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        dry_run: "${inputs.dry_run}"
      when: "reg.status != 404"

    # 4. Access — delete the app (which cascades its policies)
    - id: delete_access
      type: subplaybook
      playbook: service_runbook:cloudflare-access-app-delete
      # NOTE: not in the ADR-072 §5 catalogue today; adding it is part of the
      # preview-down follow-up PR — a straightforward delete-by-id sibling of
      # cloudflare-access-app-upsert following the §3 shape.
      with:
        domain: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        dry_run: "${inputs.dry_run}"
      when: "reg.status != 404"

    # 5. Tunnel — remove the ingress entry
    - id: delete_tunnel_route
      type: subplaybook
      playbook: service_runbook:cloudflare-tunnel-route-delete
      # NOTE: also a new sibling playbook — the route-upsert removes the entry
      # by setting service_url: null in the ingress list, symmetric to §5.3.
      with:
        tunnel_id: "${env.CF_PREVIEW_TUNNEL_ID}"
        hostname: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        dry_run: "${inputs.dry_run}"
      when: "reg.status != 404"

    # 6. DNS — delete the CNAME
    - id: delete_dns
      type: subplaybook
      playbook: service_runbook:cloudflare-dns-delete
      with:
        zone_id: "${env.CF_ZONE_ID_PREVIEW}"
        name: "${inputs.project_slug}-pr${inputs.pr_number}.preview.gyrum.ai"
        dry_run: "${inputs.dry_run}"
      when: "reg.status != 404"

    # 7. Filesystem — rm -rf /var/lib/gyrum/previews/preview-<slug>-pr<N>/
    - id: delete_fs
      type: ssh
      target: host:ops-vps
      when: "reg.status != 404 && !inputs.dry_run"

    # 8. Release port block back to pool
    - id: release_ports
      type: http
      # DELETE /previews/{pr_number}/port-allocation
      when: "reg.status != 404"

    # 9. Deregister — remove from preview_registry
    - id: deregister
      type: http
      # DELETE /previews/{pr_number}
      when: "reg.status != 404"

  outputs:
    pr_number: "${inputs.pr_number}"
    torn_down: "${reg.status != 404}"
    dns_changed: "${dns.outputs.changed || false}"
    tunnel_changed: "${tunnel.outputs.changed || false}"
    access_changed: "${access.outputs.changed || false}"
    caddy_changed: "${caddy.outputs.changed || false}"
    dry_run: "${inputs.dry_run}"
---

Shape notes:

• Destructive: true — every preview-down invocation goes through an ADR-069 §6 implicit approval step. When the webhook drives it in Phase B, the approval is auto-approved by the webhook-receiver service (using the ADR-069 approval-token pattern). Manual invocations pause for an operator click. Cron- driven (72h timeout) runs use the same service-token auto- approval as the webhook.
• reg.status != 404 guards every destructive step. If the preview registry says "no such preview" the playbook short- circuits to a no-op. This is the idempotency for teardown: running preview-down on an already-torn-down preview is safe.
• Two new sub-playbooks are named — cloudflare-access-app- delete and cloudflare-tunnel-route-delete. They are straight destructive siblings of the §5.4 and §5.3 upserts in ADR-072; adding them to the catalogue is part of the preview-down follow-up PR and does not require a new ADR — ADR-072 §11 and §12 explicitly contemplate the catalogue growing with additions.

Both playbooks share the _index.yaml + related: convention from ADR-067 §6 — they cross-reference each other, ADR-072's sub- playbooks, and this ADR.

Implementation plan — phased

Each phase is independently useful. No phase blocks the next; each is a separately-shippable PR or cohort of PRs.

Phase A — manual invocation (committed)

• Both playbooks land as YAML under dark-factory/playbooks/ release_flow/. They are runnable from the Weft <PlaybookRunner> (ADR-070 §1) by an operator typing the PR number and project slug into the form.
• An operator validates the full flow by hand: open a test PR on buzzy-frontend, run preview-up from the runner, visit the preview URL, authenticate through Access, verify the stack is up, close the PR, run preview-down from the runner, verify nothing lingers (DNS record gone, tunnel ingress gone, Access app gone, Caddy site gone, docker-compose stack gone).
• Dry-run mode (ADR-072 §2.2) is the safety net — the first invocation of each playbook runs in dry-run, the delta is inspected, then the operator re-runs with dry_run: false.
• Exit criteria: one manual cycle (up → visit → down) with green on every sub-playbook's changed flag. A second cycle on the same PR with no code change reports every sub-playbook changed: false on the up step — idempotency verified.

Phase B — webhook-driven (committed)

• A small Go service, preview-webhook-receiver, lives in dark-factory/cmd/preview-webhook-receiver/. It is a single HTTP endpoint POST /webhooks/github that verifies the HMAC (§6), records the delivery id, maps the event to a playbook invocation (pull_request.opened|reopened|synchronize| ready_for_review|review_requested → preview-up; pull_request.closed → preview-down), and invokes the playbook via the ADR-068 runtime's internal API (POST /playbook-runs with the input form filled).
• The service is deployed the ADR-072 way: a deploy-service playbook invocation places it behind Access (policy: only the GitHub webhook source IP ranges, cross-checked with the HMAC) on a dedicated hostname (github-webhooks.gyrum.ai), reverse- proxied by Caddy, backed by a systemd unit, receiving the webhook secret via ADR-069 vault resolution.
• A GitHub repository webhook is configured (via a playbook — github-repo-webhook-upsert, new) on every repo that any project's projects/*.yaml lists under repos:.
• The reaper cron lands: an hourly systemd timer that queries the preview registry, finds previews with up_at + 72h < now(), and invokes preview-down for each. Same invocation path as the webhook.
• Exit criteria: opening a PR on buzzy-frontend automatically produces a preview URL within 120 seconds; the URL is posted as a comment on the PR by the webhook-receiver (to save reviewers a lookup). Closing the PR tears the preview down within 60 seconds. A PR open-and-forgotten is reaped within 1 hour of its 72h anniversary.

Phase C — /ops/previews UI (named horizon)

• A new screen in the Weft frontend (ADR-070 contract), under the SRE persona (ADR-065 persona IA), at /ops/previews.
• Reads the preview registry. Renders a table: preview URL, PR link, project, triggering repo, opened-at, expires-at, reviewers, size-on-disk. Each row has a "tear down now" button that invokes preview-down through the runner.
• Live-updates via the SSE channel (ADR-068 §5): a preview-up completing anywhere in the fleet pushes a row; a preview-down removes it.
• Exit criteria: an SRE can answer "what previews are live right now?" from one screen, including RAM pressure and time- until-reap. Tear-down from the UI produces the same audit trail as the webhook-driven or manual path. No bypass of destructive-step approval — the UI button goes through the same ADR-069 §6 approval.

Phase D — Playwright against the preview URL (named horizon)

• The AI review flow (gyrum-review-pr, ADR-067-ish review gate) gets a new step: when a preview URL is live for the PR, run the Playwright suite (the shape of distill-gyrum-ai/e2e-integration/tests/) against the preview URL instead of against a local stack.
• The Playwright agent signs in via an Access service token (provisioned by a playbook, short TTL, scoped to the preview's domain).
• Test results post back as a PR comment — green = ✓, red = the failing test name + link to the Playwright trace.
• Exit criteria: every multi-repo PR inside a project automatically gets cross-browser E2E evidence on its review, without any reviewer having to run anything locally.

Phases A and B are committed to this cycle. Phases C and D are named horizons: they are in this ADR because they are the natural evolution of A+B, not because we commit to shipping them on a specific date. The ADR-073 re-review at T+90 days (§13) reassesses whether C and D are due.

Capacity planning — one Hetzner box

The first-order question: how many previews can we host concurrently?

Our Hetzner box is a CX52 (16 vCPU, 32GB RAM, 320GB SSD). Production services on it today: weft.gyrum.ai stack (~4GB), buzzy.gyrum.ai stack (~3GB), observability stack (~5GB), Postgres primary (~4GB), and headroom (~4GB system). That leaves ~12GB free, of which we reserve 4GB as a safety margin. 8GB available for previews.

A typical preview (Buzzy today — frontend + backend + Postgres + Redis + Caddy) consumes ~2GB RAM. A preview for a heavier project (Focus — three containers + shared Postgres) consumes ~3GB.

Phase 1 cap: 4 concurrent previews (worst-case: 4×2GB = 8GB). This is the soft cap the webhook-receiver enforces. A fifth preview-up request queues (§below) until a slot opens.

Queue policy: FIFO. First PR opened is first PR served. The webhook-receiver has a preview_queue table with (pr_number, project_slug, enqueued_at, status) rows and a tiny worker loop that pops the head when the active-preview count drops below 4. The webhook-receiver posts a comment on queued PRs — "preview queued, position N in line" — so reviewers know the delay is a capacity-shape problem rather than a failure.

Overflow handling: if a preview has been up for > 24h AND the queue depth is > 0, the webhook-receiver invokes preview-down on the oldest-up-at preview and the queue advances. This overrides the 72h cap downward to 24h under capacity pressure. The preview whose owner loses their environment gets a PR comment — "preview torn down to make room; push a commit or re-open to restore". The owner always has recourse: pushing a commit re- queues them at the tail.

Monitoring: the preview_registry emits a Prometheus gauge preview_active_count and a gauge preview_queue_depth. The ADR-055 observability stack scrapes both. An alert fires when preview_active_count == 4 AND preview_queue_depth > 2 for > 15 minutes — that's the signal that 4 is too few and we need either a second box or a harsher reaper.

When 4 stops being enough: two mitigations, in order of preference. First, drop the 72h default to 48h if many previews sit idle late in their window (reduces average RAM footprint). Second, provision a second preview-dedicated box (a CX31 at ~€16/ month adds another ~10GB preview budget). The second box is introduced via a new preview-host-provision playbook and the compose-target-host selection in preview-up step 8 becomes a scheduler call rather than a hard-coded ops-vps. Neither mitigation is in scope for this ADR; both are named.

Cost model

Cloudflare Tunnel is free at our scale (the free tier allows unlimited tunnels; bandwidth is free). Cloudflare Access is free up to 50 users on the free plan, which covers the Gyrum team plus external reviewers many times over. DNS is free. Hetzner RAM is fixed — we are already paying for the 32GB; preview workload consumes slack rather than incremental cost. Docker images are built once per push by the existing CI (cost already sunk); the preview pulls them. The only incremental cost is disk — ~5GB per preview for image layers + Postgres volume, which at four concurrent is 20GB on a 320GB disk. Rounds to zero.

Total preview incremental cost: ≈ zero. If/when we outgrow the CX52, the second-box cost is € 16/month. This is the cheapest review-app story we could credibly ship; it exists only because ADR-020 committed to the on-VM model and ADR-072 shipped the primitives.

Consequences

Easier.

• Every multi-repo PR is tested in context, automatically. The reviewer opens the preview URL, clicks around, and sees the PR's code running end-to-end against the PR's schema against the PR's assets. The "merge to main to find out" muscle atrophies.
• Review evidence is a link, not a checkout. Reviewers paste the preview URL in their review comments; the evidence is reproducible by the author, the other reviewers, and the reviewer's future self.
• AI reviewers get a URL. Phase D's Playwright-against-preview is the specific unlock: the AI reviewer does not need a local stack spin-up, which means AI review is no longer gated by "is the local fleet happy today". A remote, cheap, always-fresh environment is exactly what a headless reviewer needs.
• New SRE onboarding drops. "How do I test a frontend + backend PR?" is now a one-line answer — "open the preview URL in the PR description, sign in through Access" — rather than a ten-minute checkout-and-compose.
• Cross-project coordination is visible. The name-match rule means that if a PR is labelled feat/dark-mode on both buzzy-frontend and buzzy-backend, the preview runs both of them, and breakage between the two surfaces on the preview URL before breaking on main. A feature that touches three repos gets three repos' worth of shared evidence in one place.

Harder.

• Webhook latency is real. PR open → preview URL takes ~90 seconds end-to-end (image pull dominates at ~60s, the rest is CF propagation + Access app creation + Caddy reload + wait- ready). An operator who opens a PR and immediately looks for the preview URL waits. We document the expected timing in the webhook-receiver's PR comment so the expectation is set: "preview coming up, ~90s."
• State drift. If an operator manually edits the preview's Cloudflare Access app in the UI (say, to add an external collaborator), the next preview-up run — triggered by a push on the PR — recomputes the policy from reviewers.emails + ['*@gyrum.ai'] and overwrites the manual addition. This is the ADR-072 §11 drift problem; it is not new, and the same rule applies: if you clicked the UI, you owe a PR (to the reviewers list, or to the playbook itself). We document this in the preview UI's "share with external reviewer" affordance once Phase C lands.
• Teardown discipline matters. If a preview-down playbook crashes between steps — say, between delete_access and delete_dns — the Access app is gone but the DNS record lingers. The reaper cron (Phase B) is the safety net: it periodically (hourly) reconciles the preview registry against the Cloudflare state and invokes preview-down again on any preview whose registry row says "up" but whose <slug>-prN.preview.gyrum.ai returns NXDOMAIN-via- origin-down. Zombie DNS + Tunnel entries without a registry row are a separate sweep; a follow-up preview-zombie-reap playbook (out of scope for this ADR) handles that cleanup.
• Capacity is a first-class concern. Four concurrent previews is not "unlimited". A team week where five reviewers each have two open PRs is a queue event. We explicitly chose to ship the queue rather than provision for a never-queue capacity, because FIFO + a visible PR comment is a better first-ship than capacity-for-the-tail.
• Secrets management is now more surface. Every preview stack has its own per-preview .env containing generated DB passwords and a JWT secret. These live on the Hetzner box under /var/lib/gyrum/previews/preview-<slug>-pr<N>/. The teardown rm -rfs the directory but a reading-before- teardown from a compromised account would expose the preview's secrets — low-value targets (short-lived, gated, no prod data) but surface nonetheless. Mitigation: the per-preview secrets are never referenced from anywhere except the preview's own compose file, and the file-mode on the .env is 0600 owned by the runtime worker user.

What we sign up to maintain.

• The two playbooks (preview-up, preview-down) with all the idempotency, dry-run, and audit guarantees of ADR-072 §2. Every change to either playbook is a PR, reviewed, CI-checked for shape conformance.
• Two new sub-playbooks in the ADR-072 catalogue: cloudflare-access-app-delete and cloudflare-tunnel-route- delete. These are straightforward destructive siblings of the existing upserts and follow the §3 shape.
• The webhook-receiver service. A ~500-line Go service that does HMAC verification, delivery-id dedup, and playbook invocation. Owned like any other gyrum service: versioned, observability-wired (ADR-053), behind Access, deployed via deploy-service.
• The preview registry. A small table in the dark-factory app DB plus a thin service API. Referenced by the webhook, the reaper, the UI (Phase C). We resist the urge to make this a separate microservice; it is a table and a couple of handlers.
• The 72h + 24h-under-pressure cadence. These numbers were chosen deliberately (§5, §8). If they turn out wrong in practice — previews being reaped during an in-progress review, or previews surviving too long and bloating capacity — the review cadence (§13) re-tunes them. We track them as parameters, not as ADR amendments.
• The name-match convention. The rule "same branch name on every repo in the project" is a team discipline — a documented convention in docs/shared/preview-environments.md (a follow-up doc) that every contributor learns. A first- time failure (a PR opened on one repo with a branch name that collides with an unrelated feature on a sibling repo) is a retro item, not a bug in the playbook.

Alternatives considered

• Kubernetes-based preview environments (Tekton + Argo, or Jenkins X). Rejected. The k8s preview-env pattern is mature and battle-tested — Argo CD + an ApplicationSet generator per-PR is a well-trodden path. But Gyrum is not Kubernetes- native (ADR-020 commits to VMs + systemd + Caddy + Docker). Adopting k8s for the preview story means either running a control-plane cluster (all of the k8s ops overhead for exactly one use case) or paying for a managed control plane (external dependency for an internal concern, and a paywall whose cost scales with cluster size). The preview use case alone does not justify becoming a k8s shop. If we become a k8s shop for other reasons (unlikely per ADR-020 but reserve-the-right), the preview environment work done here — name-match, per-PR namespacing, Access-gated URL — ports to ApplicationSet generators cleanly.

• Vercel (frontend only) plus some backend story. Considered. Vercel's preview story for the frontend is genuinely excellent — every PR gets a preview URL automatically, zero config. The reasons we reject it:

  1. It doesn't cover the backend. A Buzzy PR that touches both buzzy-frontend and buzzy-backend needs both running against each other. Vercel runs the frontend; the backend is our problem. We would end up with two preview systems — Vercel's frontend, our own backend — and the glue (pointing the Vercel preview frontend at the self-hosted backend preview) is exactly the multi- repo coordination Gordian knot we are trying to solve once.
  2. Third-party billing surface. Vercel charges per build-minute above a free threshold. Four PRs a day at ~2 build-minutes each is ~240 minutes/month per repo. Across five product repos, that's 1,200 minutes — past the free tier. The annual cost is non-trivial and it scales with PR rate — exactly the wrong cost shape for a review primitive.
  3. Loss of the control surface. ADR-072 committed us to "every infra mutation is a playbook". A preview environment that is half-Vercel, half-Gyrum is a preview environment where half of the state lives in a vendor's database and the other half in playbook_run_events. The audit story splits; the rollback story splits; the drift story splits.

  Vercel is excellent at the shape of preview environments it solves, and we do not adopt it because our shape is "multi- repo, backend included, on our VMs, in our audit log". The escape hatch: if our preview capacity becomes a serious bottleneck and Vercel-for-frontend-only materially accelerates the review loop, we revisit — the decision boundary is "backend still on our VMs, Vercel proxied for frontend only". We do not cross that boundary now.

• Render.com's preview environments. Considered. Render does frontend and backend review apps, has PR webhooks, has Access control (HTTP basic auth; not as good as Cloudflare Access). Rejected for the same reason as Vercel (migration off Hetzner) plus: Render's preview cap on the free tier is low and paid previews are billed per-container-hour, which scales with PR rate in exactly the wrong direction. And: we would be forced to migrate the production stack to Render to get preview parity, which is a much larger decision than this ADR.

• GitHub Actions environments (GitHub-hosted preview infrastructure). Considered. GitHub offers ephemeral environments tied to Actions, with environment protection rules and secrets per environment. Rejected because:

  1. Gated behind GitHub's capacity and IP allowlisting. The preview environment would run on GitHub-hosted runners or self-hosted runners that are GitHub- discoverable, neither of which is a stable home for a stack that reviewers need to visit in a browser.
  2. Not self-hosted. ADR-020 is again the line. A preview environment that only exists inside the GitHub Actions runner is not a URL reviewers can hit from their browsers unless we proxy it out, at which point we are running the proxy ourselves, at which point we might as well run the stack ourselves.
  3. Worse audit story. Actions-log audit is per-run, per-workflow — not a unified fleet audit. Our playbook playbook_run_events gives us one place to look; Actions audit is another.

• Docker-compose on the reviewer's laptop (current status quo). Rejected because it is the status quo and the status quo is exactly what this ADR is trying to move past. Local compose is fine for the author's own inner loop; it is not a review primitive because the reviewer is not the author.

• Shared staging environment rather than per-PR previews. Rejected. A single shared staging environment is what we have today (the production-adjacent Hetzner slice); reviewing on it means reviewers step on each other's pushes. The per-PR-ness is the innovation — each review is isolated from every other. Shared staging also means destructive tests (migrations, seed rewrites) are dangerous in exactly the way per-PR isolation removes.

• A single "long-lived preview" per project, updated from the PR branch. Rejected. This is a half-step: the preview exists per-project rather than per-PR, so it still serialises reviewers of different PRs on the same project. The cost of going full per-PR is marginal (same playbooks, smaller queue) and the benefit is much larger (parallel review).

• Opt-in previews (operator must click a button on the PR to get a preview). Rejected. Opt-in means some PRs have a preview and some don't, which means reviewers have to check "does this one have one?" on every review. The cost of a preview is low enough (~2GB RAM, auto-torn-down) that opt-out (every PR gets one unless disabled) is the right default. A PR that wants to disable the preview (because the changes are docs-only and a preview adds no signal) carries a no-preview label; the webhook-receiver respects it. This keeps the default pro-evidence.

Related

• ADR-020 — shared infra vs per-product isolation; this ADR operates inside the shared-infra boundary (one Hetzner box, one CF account) and introduces per-PR application isolation, not per-PR infra isolation
• ADR-053 — frontend observability; the preview-webhook- receiver, the preview registry, and the preview stack (inside each preview) emit structured logs and metrics to the same observability stack as production
• ADR-055 — observability stack container pinning; the preview stack pulls pinned observ stack image tags, same as production
• ADR-059 — project-first IA, project YAML with repos: list; the name-match rule (§1 of the pattern) iterates projects/<slug>.yaml .repos to find sibling branches
• ADR-061 — extracted gyrum-projects library; the preview-up playbook reads project YAML through this library
• ADR-065 — persona-aware IA; the /ops/previews screen (Phase C) is scoped to the SRE persona
• ADR-067 — playbooks unified primitive; both preview-up and preview-down are kind: release_flow playbooks following ADR-067's front-matter, relations, and retro conventions
• ADR-068 — playbook runtime architecture; the subplaybook, http, ssh, and shell executors used here are ADR-068 Phase 4 primitives
• ADR-069 — playbook security; destructive: true on preview-down triggers ADR-069 §6 implicit approval; the webhook secret lives in the ADR-069 vault; scoped tokens per the ADR-069 §3 allow-list pattern
• ADR-070 — playbook UI contract; <PlaybookRunner> is the surface through which Phase A manual invocations run
• ADR-071 — product name Weft; the <product>.gyrum.ai subdomain pattern (weft.gyrum.ai, now preview.gyrum.ai) is the same namespacing pattern
• ADR-072 — infrastructure as playbooks; this ADR is a direct consumer of the ADR-072 catalogue (cloudflare- dns-upsert, cloudflare-tunnel-route-upsert, cloudflare-access-app-upsert, caddy-site-upsert) and names two new destructive siblings (cloudflare-access- app-delete, cloudflare-tunnel-route-delete) as additions to the catalogue
• Fleet E2E compose pattern — distill-gyrum-ai/ e2e-integration/ is the docker-compose shape the per-preview compose file generalises (same image pulls, same inter-service wiring, new per-PR namespace on the compose project name, ports, volumes, networks). ADR-075 promotes that same harness to the fleet's cross-product branch-testing service; its Phase 2 --branch override and the ADR-073 per-PR preview are complementary (CI plane vs Ops plane), and ADR-073 Phase D names ADR-075's Playwright suite as its scenario source when Phase D ships
• Dockerfile-per-repo cohort — the in-flight PR set that ships production Dockerfiles for buzzy-frontend, buzzy-backend, distill-gyrum-ai, et al is the precondition that makes "preview runs the prod image" meaningful rather than aspirational

Review cadence

Re-review every 90 days (next: 2026-07-22) against:

• Adoption — what fraction of PRs across project-tracked repos have a preview URL posted in their PR description? Target: > 90% within 30 days of Phase B.
• Latency — PR-open → preview-URL-posted. Target: p50 < 90s, p95 < 180s. Regressions get a bug, not an ADR.
• Capacity — queue depth trend. Target: queue depth = 0 more than 95% of working hours. If queue depth > 2 for > 15 minutes on any given week, raise the conversation about the second preview box (§8).
• Teardown hygiene — reaper-invoked teardowns vs. webhook- invoked vs. manual. Target: < 5% of teardowns are reaper- driven (a healthy ratio means most PRs get merged or closed within 72h, which is what review latency should look like anyway).
• Idempotency violations — preview-up runs that report changed: true on the second run with no code change. Target: zero. Each one is a bug to file against the relevant sub-playbook.
• Access policy drift — occasions on which someone edits the preview's Access app in the CF UI. Each is a retro item. Drift > 1/month → the UI-editing affordance is wrong and we address it.
• Phase C / Phase D due? Are the named horizons now overdue, under-pressure, or still fine to defer? Each cadence review answers this deliberately, not implicitly.

Stale flag on the Owner dashboard once last_reviewed on this ADR passes 90 days.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one