ADR-075: Fleet-E2E as a cross-product branch-testing service

Status: Proposed Date: 2026-04-24 Related: ADR-020 (shared infra vs per-product isolation), ADR-053 (frontend observability — first fleet-wide test surface), ADR-059 (project-first IA — multi-repo projects), ADR-067 (playbooks unified primitive), ADR-073 (preview environments per feature branch), ADR-074 (CI/CD plane separation), ADR-078 (trigger-driven orchestration), ADR-080 (warp as durable trigger queue)

Decision (one paragraph)

We promote distill-gyrum-ai/e2e-integration/ from "distill's own end-to-end suite" to Fleet-E2E — the fleet's cross-product branch-testing service, a single named harness that can boot any combination of sibling-repo branches against the fleet's production Dockerfiles and run the Playwright suite that owns cross-service regressions. The single surface accepts a --branch <repo>@<ref> flag from three triggers — local (e2e-run from any checkout), CI dispatch (repository_dispatch type:fleet-e2e from a sibling PR), scheduled (nightly cron against every product's main) — and rebuilds only the overridden services from the named refs, falling back to pinned ghcr.io/gyrum-labs/*:fleet-e2e-latest images for the rest. Each participating product owns its slice of the contract via tests/e2e-contract.json (which scenarios it must pass, which tags it owns, which playbooks it participates in); the harness reads the contract and runs only the relevant subset when a given branch is overridden. The harness stays in distill-gyrum-ai for now (it is already there, it works); a Phase 5 promotion to its own gyrum-labs/fleet-e2e repo is named but not committed. Security and cost are bounded by three rules — run strictly on the CI plane (ADR-074; no Ops creds, no Ops tailnet), concurrent-run cap of 2 on the dev machine with overflow queued via the ADR-080 warp primitive once it lands, and no shared secrets between sibling branches and the harness (each branch's container gets its own test-only env). Results post as an advisory check on the sibling PR — not a merge gate — until Fleet-E2E is demonstrably reliable (we revisit the gating rule at T+90 days). Phased rollout: Phase 1 (shipped) pinned images only; Phase 2 (this ADR's trigger) --branch override and local e2e-run CLI; Phase 3 GitHub Action sender + PR comment; Phase 4 scheduled nightly across all products, drift report; Phase 5 (horizon) extract to own repo.

Context

distill-gyrum-ai/e2e-integration/ shipped over the last three days — PRs #53, #58, #59 on distill-gyrum-ai, README and TUTORIAL merged, six scenarios green on CI. The harness is a docker-compose stack that boots postgres + ai-research + distill + ai-frontend from production Dockerfiles and drives a real Chromium browser through cross-service user journeys. Three triggers exist in .github/workflows/e2e-integration.yml: pull_request (on distill PRs), repository_dispatch type:fleet-e2e (receiver-ready; no sibling sends today), and workflow_dispatch (manual, accepts ai_research_ref / ai_frontend_ref inputs). The receiver-ready but sender-blocked state is the load-bearing gap this ADR closes.

The harness today has a narrow self-description. Its README leads with "fleet harness" and its compose file hard-codes sibling clones of ai-research and ai-frontend two directories up. A reader arriving at the repo for the first time asks "is this distill's test suite or something else?" and the honest answer is "it started as distill's, but it's the only place the fleet is tested in a browser, and the only place where a cross-service regression is caught before merge." The test code does not know which product is driving the run. The compose file does not know which branch it is testing. The Playwright scenarios exercise behaviours that none of the three repos owns alone — the playbook-run spec asserts the full round-trip UI → distill → ai-research → shell → UI. The surface is already the fleet's; the name is not.

Three converging pressures force the decision now:

ADR-059 shipped project-first IA. Projects now span multiple repos (Buzzy = buzzy-frontend + buzzy-backend; Focus = three repos; distill + ai-research + ai-frontend are the reference triple). Every multi-repo PR inside a project has exactly the regression class Fleet-E2E catches and nothing else does. Without a named cross-product harness, each project reinvents the pattern badly.
ADR-073 (preview environments) and ADR-074 (plane separation) just landed. ADR-073 spins preview URLs per PR on the Ops plane; Fleet-E2E exercises test scenarios on the CI plane. The two are complementary and their boundary needs to be explicit before reviewers start wondering whether the Playwright suite belongs on the preview URL or on its own stack. (Answer: both, eventually — see § Relationship to ADR-073.) ADR-074's CI plane is where Fleet-E2E lives by construction; this ADR makes that explicit and forbids any Ops-plane drift.
The dispatch receiver is ready; the sender is missing. The workflow's repository_dispatch path exists and has been manually tested (gh api .../dispatches -f event_type=fleet-e2e). Neither ai-research nor ai-frontend currently sends. Until the fleet is named as a service, no repo knows it is supposed to send; once it is named, the sender is a six-line workflow addition. The gating step is documentation — this ADR.

Why a separate ADR and not an amendment to ADR-073 or ADR-074

ADR-073 is about the runtime surface — a PR-scoped full-stack environment, URL, DNS, Access, teardown. Fleet-E2E is about the test surface — scenarios, scenario ownership, who runs them, where they run, what they gate. They touch (Phase D of ADR-073 names Playwright-against- preview as a named horizon; §9 of this ADR reconciles the two) but they answer different questions and have different owners.

ADR-074 is about plane isolation. Fleet-E2E is a CI-plane consumer of the rules ADR-074 codifies. This ADR cites ADR-074's rules (no Ops creds, no shared filesystem, ephemeral runner) and applies them to the harness, but it does not change them.

What this ADR is not

It is not a commitment to extract Fleet-E2E to its own repo. Phase 5 is a named horizon, not a scheduled migration. It is not a change to what the Playwright suite tests (the six scenarios stand). It is not a superseder of the existing e2e-integration/ README or TUTORIAL — those stay as the developer-facing docs; this ADR is the decision record behind them. It is not a merge-gate introduction for sibling repos — Fleet-E2E result is advisory on sibling PRs until we prove reliability.

The promotion shape

Today, the harness targets distill's own build against pinned images of its siblings. Tomorrow, the same compose file accepts per-service branch overrides at the command line, rebuilds those services from source, and leaves every other service on the pinned image. One harness, three triggers, n products.

 ┌─────────────────────────────────────────────────────────────────────┐
 │                       Fleet-E2E (one harness)                       │
 │                                                                     │
 │   inputs:                                                           │
 │     --branch ai-frontend@feat/foo      (optional, any product)      │
 │     --branch ai-research@feat/bar                                   │
 │     --branch distill@pr-123                                         │
 │     (unflagged products → ghcr.io/*:fleet-e2e-latest pinned image)  │
 │                                                                     │
 │   behaviour:                                                        │
 │     1. read each product's tests/e2e-contract.json                  │
 │     2. build/pull images per flag                                   │
 │     3. docker compose up (healthcheck-gated)                        │
 │     4. run only the scenarios the contracts claim                   │
 │     5. upload playwright-report/ + post result                      │
 └─────────────────────────────────────────────────────────────────────┘
         ▲                       ▲                         ▲
         │ local                 │ CI (dispatch)           │ schedule
         │ e2e-run               │ repo_dispatch           │ cron
         │ --branch ...          │ type:fleet-e2e          │ nightly,
         │                       │ from sibling PR         │ main of all
         │                       │                         │ products

The three triggers

Local. A developer from any checkout types:

e2e-run --branch ai-frontend@feat/foo --branch ai-research@feat/bar

The e2e-run CLI (installed via gyrum-devtools) resolves distill-gyrum-ai/e2e-integration/ as the harness location, clones or updates sibling checkouts to the named refs, invokes the existing make up && make test && make down with compose env vars populated per flag, and prints the Playwright report path on completion. Single invocation, no cd'ing, no manual clone juggling.

CI dispatch. A PR opens on ai-frontend. The repo's pr-ci.yml includes a fleet-e2e-dispatch step:

- uses: peter-evans/repository-dispatch@v3
  with:
    token: ${{ secrets.FLEET_DISPATCH_TOKEN }}
    repository: gyrum-labs/distill-gyrum-ai
    event-type: fleet-e2e
    client-payload: |
      { "branch_overrides": { "ai-frontend": "${{ github.event.pull_request.head.sha }}" },
        "source_pr_url":     "${{ github.event.pull_request.html_url }}" }

distill-gyrum-ai/.github/workflows/e2e-integration.yml accepts the payload, populates override env vars, boots the stack, runs the scenarios the contracts claim, and posts the result back to the originating PR as a comment (not a check that blocks merge — see §Gating).

Scheduled. A daily cron at 03:00 UTC fires the harness against main of every product listed in the fleet manifest. A drift report is posted to #fleet-e2e on internal chat: green stays silent, red names the scenario + the last-green SHA + a link to the artifact. This catches contract drift introduced by merges that individually passed their own CI (a classic two-repo lockstep bug) as well as environmental drift in the pinned base images.

Product-owned contract

Each product ships a declarative file at tests/e2e-contract.json:

{
  "product": "ai-frontend",
  "owns_scenarios": [
    "tests/catalogue-filter.spec.ts",
    "tests/version-footer.spec.ts"
  ],
  "participates_in": [
    "tests/playbook-run.spec.ts",
    "tests/playbook-hot-reload.spec.ts"
  ],
  "tags": ["ui", "persona:sre"],
  "preview_required": false
}

Three fields, one purpose — let Fleet-E2E know, for any given branch override, which scenarios must run. On a --branch ai-frontend@feat/foo invocation, the harness reads ai-frontend/tests/e2e-contract.json, unions owns_scenarios + participates_in, and passes that list to npx playwright test. A frontend-only change does not pay the cost of running the ai-research-heavy scenarios; a research-only change skips the UI-only scenarios.

Why contracts live in the product repo, not the harness. The product owns which scenarios its branches must pass — the harness does not know the domain. A new UI feature in ai-frontend lands with the matching contract entry in the same PR; the sender knows the scenario list is current because it is next to the code. If the contract lived in the harness, every frontend PR would need a paired harness PR, and the harness would become the bottleneck the monorepo people warn about.

Contract schema enforcement. The harness validates contracts at boot via ajv; an invalid contract fails the run with a clear error before the stack comes up. The schema ships with the harness.

Where the harness lives

For now: distill-gyrum-ai/e2e-integration/. It works. Relocating it is a cost without a benefit. What changes in Phase 2 is framing: the directory's README leads with "Fleet-E2E service — the fleet's cross-product branch-testing harness" rather than "distill's e2e tests". Every consumer-facing surface (the CLI name e2e-run, the #fleet-e2e channel, the PR comment preface, the dispatch event name) says Fleet-E2E. The distill-specific detail (it happens to live under distill-gyrum-ai/, the receiver workflow happens to run in that repo's Actions) is an implementation note, not the product.

A product's CLAUDE.md (devtools-maintained section) gains one line under "common commands":

e2e-run — test this branch against the fleet (Fleet-E2E — [ADR-075][]).

Onboarding is that line plus the CLI's --help. No per-product README fork, no sibling-of-harness copy of the compose file.

Phase 5 (named horizon). If Fleet-E2E pays its keep, extract to gyrum-labs/fleet-e2e with its own release cadence, its own owners, and its own contribution model. Triggers to reassess: (a) more than three products have contracts; (b) the harness ships a change that a non- distill engineer blocks on; (c) the compose file grows past what a single-product repo should carry. We revisit at T+90 days and at T+180 days; if none of the triggers fire, Phase 5 stays deferred.

Relationship to ADR-073 (preview environments)

Both ADR-073 and this ADR involve per-PR environments. They are orthogonal:

Question	ADR
Spin up a full-stack, Access-gated URL for the PR?	ADR-073 (preview environments)
What can I test against that URL (or against its own stack)?	ADR-075 (this ADR)

ADR-073's preview lives on the Ops plane — it uses Ops credentials (Cloudflare Access, the tunnel token), it runs under Weft's playbook runtime, it is production-adjacent. Fleet-E2E runs on the CI plane — ephemeral runner, no Ops creds, tears down on completion. They overlap in one place and one place only: ADR-073 Phase D names "Playwright against the preview URL" as a named horizon. When Phase D ships, Fleet-E2E's Playwright suite is the scenario source — the suite moves from "driving a CI-plane docker-compose" to "driving the Ops- plane preview URL through an Access service token". The scenarios do not change. The harness does not change. Only the BASE_URL and the auth handshake change.

Put another way: ADR-073 gives us the URL, ADR-075 gives us what to do with it. ADR-075's Phase 2 (this ADR's Phase 2) ships before ADR-073 Phase D, so for some time the two live side-by-side — Fleet-E2E runs its own CI-plane compose stack and (when Phase D lands) can also run against an Ops-plane preview URL for the same PR, with the harness switch being a --target preview|local flag. That is the integration seam; the ADRs do not need to merge.

Relationship to ADR-074 (plane separation)

Fleet-E2E is a CI-plane job, full stop.

It runs on a GitHub-hosted runner today (Actions free tier on public repos, $0.008/min on private — fits inside the ADR-074 cost envelope). Self-hosting trigger (ADR-074 §5) does not fire yet.
It never holds Ops credentials. The pinned ghcr.io images are pulled with the ephemeral GITHUB_TOKEN; sibling branch code is fetched with the workflow's read-only checkout token; no Cloudflare or production-database credential appears in any Fleet-E2E secret, env var, or workflow file. A PR on a sibling repo whose diff injects a malicious Dockerfile layer can break its own preview's tests and waste CI minutes — it cannot reach the Ops tailnet because it is not on the tailnet.
It never mutates Ops state. Fleet-E2E's dispatches are inbound (sibling PR → distill workflow); its outbound surface is the PR comment (via the ephemeral GITHUB_TOKEN) and the artifact upload. No repository_dispatch into the Ops plane, no HMAC call to the Weft runtime, no Caddy reconfiguration. Fleet-E2E observes Ops indirectly (when Phase D runs against a preview URL, the observation is through a public HTTPS surface, exactly like a user), never mutates.

This is not new rule-making. ADR-074 §3's three planes already forbid all of the above. This ADR cites them so that a future engineer tempted to "just run Fleet-E2E on the Ops box because it has Docker" has a paragraph to lose the argument against.

Security

The single non-trivial attack vector: a branch whose container image is arbitrary code under the PR author's control. Fleet-E2E builds that image from source. If the harness and the image share state, a PR can exfiltrate harness secrets or persist beyond the run.

Three mitigations, all of which fall out of running on a GitHub-hosted ephemeral runner (ADR-074 default):

Runner isolation. Every Fleet-E2E run is a fresh GitHub-hosted VM. The runner is destroyed at end-of-run. A malicious image cannot persist to the next run.
Resource limits. The compose file sets mem_limit and cpus on every service. A misbehaving image is killed before it can exhaust the runner.
No shared secrets. The harness itself has one secret — the FLEET_DISPATCH_TOKEN (GitHub PAT scoped to repository_dispatch) — and it lives only on the sender side. The receiver (Fleet-E2E) accepts the dispatch payload but does not re-emit the token. Sibling branches' containers get a test-only env (.env.test, checked in, non-secret by construction); they never receive production-shaped credentials.

If Fleet-E2E migrates to a self-hosted runner (ADR-074 §5 trigger), the runner is a dedicated Hetzner CX22 outside the Ops tailnet — ADR-074 §6 is the reference architecture, and this ADR inherits it without modification.

Cost

A single Fleet-E2E run: 2–5 minutes wall clock on the hosted runner (warm cache ~2 min, cold ~5 min). At current scale:

~15 PRs/day across the fleet × ~0.5 path-pass after e2e-contract.json filtering × 4 min avg = ~30 min/day → ~900 min/month.
Private-repo cost: ~$7.20/month. Well inside the ADR-074 "do not optimise" band.

The scheduled nightly adds 1 run/product/day × 4 products × 4 min = ~16 min/day → ~480 min/month → ~$3.84/month. Combined ~$11/month.

Concurrency cap on the dev machine (local e2e-run): 2 concurrent runs. Beyond that, the host's Docker daemon starts swapping and runs get flaky. Overflow is queued — Phase 2 uses a simple filesystem lockfile in ~/.gyrum/fleet-e2e/queue/; Phase 3 (once ADR-080 warp primitive lands) migrates to the warp durable queue for visibility and cross-checkout consistency. CI concurrency is bounded by GitHub's own per-repo limit (20 concurrent jobs), well above our rate.

Gating — advisory, not required

Fleet-E2E's result posts on the sibling PR as a comment + status check marked as non-required. Branch protection does not include fleet-e2e in the required-checks list. Until Fleet-E2E is demonstrably reliable — specifically, flake rate < 2% over 50 consecutive runs across at least two products — it is an advisory signal only. A red Fleet-E2E on a sibling PR is a strong prompt to investigate; it is not a merge block.

Why advisory and not required: the harness is one failure domain for four products. A flaky Fleet-E2E with a required check on every sibling PR stops the fleet's merge train the moment a dependency update breaks a scenario for reasons that have nothing to do with the PR. Adopting the gate too early puts the harness's reliability on the critical path for every merge; the blast radius justifies a reliability-first ramp.

We revisit the gating decision at T+90 days (§ Review cadence).

Phased rollout

Phase 1 — pinned images (shipped)

Status: done, merged 2026-04-21 → 2026-04-23.

distill-gyrum-ai/e2e-integration/ compose stack, Makefile, Playwright suite (6 scenarios).
CI workflow on distill PRs, plus repository_dispatch receiver and workflow_dispatch manual.
Sibling repos fetched at main by default; ai_research_ref / ai_frontend_ref inputs override for manual runs.

What Phase 1 does not have: branch-override CLI, sibling-side dispatch senders, e2e-contract.json files, scheduled runs, consumer-facing "Fleet-E2E" framing.

Phase 2 — `--branch` override + `e2e-run` CLI (this ADR's commit)

e2e-integration/docker-compose.yml accepts per-service branch override env vars (AI_FRONTEND_BRANCH, AI_RESEARCH_BRANCH, etc.). When set, the service builds from source at that ref; when unset, pulls ghcr.io/gyrum-labs/<product>:fleet-e2e-latest.
gyrum-devtools ships e2e-run, a thin wrapper that parses --branch <product>@<ref> flags, resolves the harness location, and invokes make up && make test.
Framing sweep: README and TUTORIAL re-titled; CLAUDE.md snippet added to product repos by gyrum-setup --sync-claudemd; workflow name changed to fleet-e2e.
Exit criteria: a developer running e2e-run --branch ai-frontend@feat/foo --branch ai-research@feat/bar from any checkout gets a green run in under 7 minutes cold.

This is the committed Phase. It is one tranche of implementation PRs against distill-gyrum-ai plus one tranche against gyrum-labs/ devtools; neither is this ADR's scope. This ADR is docs-only.

Phase 3 — dispatch senders + PR comment (committed)

ai-frontend/.github/workflows/pr-ci.yml and ai-research/.github/ workflows/pr-ci.yml each gain a fleet-e2e-dispatch step. The FLEET_DISPATCH_TOKEN secret (existing, scoped to repository_dispatch) provides the auth.
e2e-integration.yml gains a final step that posts a comment on the originating PR: result, run link, artifact link, scenario pass counts.
Exit criteria: opening a PR on any participating repo produces a Fleet-E2E comment within 8 minutes of push; comment text is actionable on its own (no artifact download needed for green; for red, names the failing scenario and links the Playwright report).

Phase 4 — scheduled nightly + drift report (committed)

e2e-integration.yml gains a schedule: trigger (cron: '0 3 * * *').
A fleet-manifest.json at distill-gyrum-ai/e2e-integration/ enumerates participating products + their default refs. The scheduled run iterates.
The drift report posts to #fleet-e2e with the shape named in §Trigger. Red opens an issue on the offending product's repo with the area:fleet-e2e label.
Exit criteria: a cross-repo regression that escapes per-repo CI surfaces in the morning drift report rather than in a production incident.

Phase 5 — extract to own repo (named horizon)

Trigger list (all three must fire before we commit): more than three products have contracts; at least one non-distill engineer has blocked on a harness change; the compose file is carrying services none of which distill depends on. If/when those fire, extract to gyrum-labs/fleet-e2e as a standalone repo with its own release cadence. Until then, Fleet-E2E keeps its current home.

Phase 5 is in this ADR so the T+90 / T+180 reviews have a clear question to answer; it is not a commitment.

Consequences

What becomes easier.

A multi-repo project PR (ADR-059 project model) gets cross-service browser evidence without the reviewer doing anything. The "works-on-my-machine" class of regression loses its cheapest path into main.
A sibling-repo engineer has exactly one command (e2e-run) to answer "does my branch break the fleet?". Today the answer is "run gh workflow run e2e-integration.yml with the right refs and wait", which nobody remembers.
Naming the harness lets us speak about it. Fleet-E2E gets its own channel, its own issue label, its own ownership rotation — all of which are cheap to do once the thing has a name and expensive to retrofit once four products are leaning on an anonymous directory.
The scheduled drift report turns a tacit "nothing broke since yesterday" into an explicit signal; silent drift becomes loud.

What becomes harder.

Four products' contributors need to keep tests/e2e-contract.json accurate. Stale contracts silently skip scenarios. We mitigate with schema validation at run time and a contract-coverage check in the review flow, but the muscle is new.
The dispatch sender adds ~30 seconds to every sibling PR's CI. That is the price of admission to Fleet-E2E; it is path-filtered to "this PR actually changes code that Fleet-E2E could exercise".
Debugging a Fleet-E2E failure on a sibling PR means a round trip across repos (the sender's PR → the receiver's workflow → Playwright report). We mitigate with artifact links in the PR comment and with the trace viewer; it is still more clicks than a single-repo failure.

What we sign up to operate, maintain, or revisit.

The harness becomes a fleet-wide dependency. A bad change to e2e-integration/ is a four-product outage. We carry it like other fleet deps: small PRs, review by someone other than the author, revert the moment flake rate crosses 5%.
tests/e2e-contract.json schemas evolve. We version the schema in-harness; a minor version bump is backwards-compatible, a major bump requires a deprecation window.
T+90 days: revisit gating (still advisory? or required?). T+180 days: revisit Phase 5 triggers.
The fleet-manifest.json drifts if products are added without a manifest entry. Scheduled-run includes a "known products not in manifest" warning.

Alternatives considered

Per-repo copy of the Playwright suite. Each product keeps its own e2e/. Lost because copies drift within two sprints, each product pays the full cost of the full stack for changes that touch one tier, and cross-repo regressions stay invisible.
Require preview URL (ADR-073) for every test, collapse Fleet-E2E into ADR-073 Phase D. Lost because ADR-073 previews need Ops-plane credentials (tunnel, Access) and a running Weft; a CI-plane harness that stands up a fresh compose stack in 2 minutes is cheaper, plane-safer, and runs before previews exist. When Phase D ships, Fleet-E2E gains a second --target preview mode; it does not replace the CI-plane compose.
Keep the harness distill-local and let other products use gh workflow run from their CI. Lost because naming matters. A harness that is distill's by identity gets distill's attention for maintenance and four products' attention for bug reports — the worst ratio. Naming it Fleet-E2E makes the cross-product ownership legible from the first commit.
Extract to gyrum-labs/fleet-e2e immediately (skip Phase 5 deferral). Lost because the repo's first owner would be a ghost — the harness works today because its maintainers also maintain distill's compose and cross-check against distill's Dockerfile. Premature extraction puts a coordination tax on changes that today are atomic. The deferral is explicit so the cost/benefit gets reassessed, not forgotten.
One contract file at the harness, enumerating every product's scenarios. Lost because the contract lives next to the code that makes the scenarios valid. Centralising the file makes product PRs require a harness PR for every scenario add/remove — we would build a queue where there does not need to be one.

Review cadence

T+30 days (2026-05-24): Phase 2 exit-criteria met? Flake rate over the first four weeks? Who ran e2e-run and what did they say?
T+90 days (2026-07-24): Gating decision — advisory → required? Only if the reliability bar (flake rate < 2% over 50 runs across ≥ 2 products) is met. If not, set the next review.
T+180 days (2026-10-24): Phase 5 triggers. Is any of the three firing? If not, Phase 5 stays deferred; if yes, open the extraction ADR.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one