ADR-074: CI/CD plane separation

Status: Accepted Date: 2026-04-23 Related: ADR-020 (shared infra vs per-product isolation), ADR-053 (frontend observability), ADR-055 (observability stack container pinning), ADR-069 (playbook security), ADR-072 (infrastructure as playbooks), ADR-073 (preview environments per feature branch)

Decision (one paragraph)

We separate the Gyrum compute surface into three planes — Dev, CI, Ops — and forbid any shared filesystem, shared credentials, or shared runtime between them. The Dev plane is the engineer's laptop running dev-all.sh; it sees local clones and never touches production code paths. The CI plane is ephemeral, per-run, and today runs on GitHub-hosted runners (Actions free tier for public repos, $0.008/min for private); it sees sibling-repo code + test fixtures and never holds Ops credentials or production data. The Ops plane is the Hetzner fleet running weft.gyrum.ai, buzzy.gyrum.ai, the main backends, and the observability stack; it carries live traffic and never runs CI test code. Each plane calls the next only through documented APIs (HTTPS with scoped tokens, ghcr.io image pulls, the ADR-072 playbook APIs), never through shared mounts or reused creds. We stay on GitHub-hosted runners until one of four explicit triggers fires (PR volume, cost, speed, Tailscale-only dependencies — §5); when a trigger fires we move to a dedicated Hetzner CX22 self-hosted runner, provisioned via an ADR-072 release_flow:ci-runner-provision playbook, Tailscale-joined for private-registry access, with zero reachability into the Ops tailnet. Migration is additive (self-hosted alongside GitHub-hosted; one runs-on: label flips per job), never a cutover. The three-plane rule is enforced at convention and at config, and breaking it is a security finding in review.

Context

Three things need CI runners today, and the list is growing:

Fleet E2E — the distill-gyrum-ai/e2e-integration/ compose stack runs Playwright against a boot-from-prod-images multi-service stack on every PR that touches the fleet. It pulls ghcr.io/gyrum- labs/distill-gyrum-ai:<sha> (plus the sibling-repo images) and exercises them end to end. One run today is ~6 minutes of runner time.
Persona review — the 3-persona + 1-advisory AI review flow (CLAUDE.md §Review) runs as a CI job that calls the Anthropic API with the PR diff and some context files. Token-bounded, ~2 minutes per persona, four personas in parallel.
Contract tests — every repo runs Go/TypeScript unit tests, a build, staticcheck/eslint, and gosec on each push. This is the classical per-repo CI; ~3 minutes per repo.

ADR-073's preview environments (Phase B) will add a fourth consumer (webhook → playbook-runtime → preview-up → docker compose up), but that runs on the Ops plane on purpose — a preview environment is production-adjacent by design, and its playbook-runtime invocation is an Ops mutation, not a CI step. It is not counted against the CI budget.

All three CI consumers today run on GitHub Actions hosted runners. The runner is spun up ephemerally on each job, the workflow clones the repo, pulls images from ghcr.io, and tears down on completion. Cost for public repos is zero; for our private repos inside gyrum-labs it is $0.008/minute on the standard Linux tier. At current PR volume (~15 PRs/day across the gyrum-labs fleet, ~20 workflow runs/day per repo including preview redeploys, ~6 minutes per run) we spend roughly $30/month on Actions minutes — well inside the "don't optimise" band.

The question this ADR answers: as CI load grows and as more services need private-network access (private GHCR mirror, private test DB, Tailscale-only services), when do we move to self-hosted runners, and how do we do it without compromising the Dev/CI/Ops plane boundary we implicitly rely on?

The question behind the question is the plane boundary itself. Today we implicitly rely on the fact that GitHub's runners run on GitHub's infrastructure, with GitHub's network, with no path into our ops tailnet. A careless move to self-hosted — "just stand up a runner on the ops box, it already has Docker" — would erase that boundary, putting untrusted PR code one sudo away from the production Postgres. The industry has enough post-mortems of exactly this shape (runner compromise → production breach) that we codify the rule before we need it, not after.

Why now — and not after we hit the triggers?

Two reasons. First, ADR-072 (infra as playbooks) and ADR-073 (preview environments) just shipped, and both are about to raise CI load (preview-up on every PR, cross-repo repository_dispatch signals firing fleet E2E on sibling PRs). The trigger-to-move is closer than it was six weeks ago. Second, the plane rules are cheap to write down today (we already follow them; this ADR just names them) and expensive to retrofit once a runner is already breaching them. The "don't mix CI and Ops" discipline is either free or very expensive — there is no intermediate cost.

A concrete scenario illustrates the retrofit cost. Suppose a future engineer, in a hurry, adds a self-hosted runner on the Ops box because "it's there" — actions/runner installed next to the Weft runtime, sharing the Docker daemon, sharing the Tailscale identity, sharing /var/lib/gyrum/. Every workflow that runs on it now has a reachability path to the production Postgres, to the Cloudflare production API token (resolved from the ADR-069 vault on the same box), to the SSH keys that Weft uses to reach sibling hosts, and to every other piece of Ops machinery. Unwinding that — moving the runner to its own VM, re-scoping credentials, rebuilding the Tailscale ACL — is weeks of careful work under the pressure of "we have to keep shipping". Writing down the rule now ("the CI plane is never on the Ops box") costs one ADR; enforcing it later after the retrofit mistake has been made costs an incident plus the weeks of unwind. The asymmetry is what forces the ADR to land before the trigger fires, not after.

What this ADR is not

It is not a commitment to self-host. It is not a procurement decision. It is not a migration plan with dates. It is a plane- separation rule that applies today (on GitHub-hosted runners) and that survives whatever runner substrate we choose later. The trigger list (§5) and the self-hosted architecture (§6) are "if and when", not "we will". The load-bearing output of this ADR is the three-plane contract (§3); the self-hosted detail exists so that when a trigger fires we are not designing from scratch in a hurry.

The three planes

 ┌────────────────────┐    ┌────────────────────┐    ┌────────────────────┐
 │      DEV plane     │    │      CI plane      │    │      OPS plane     │
 │  (engineer laptop) │    │  (ephemeral, per-  │    │  (Hetzner fleet,   │
 │                    │    │     run, isolated) │    │   live traffic)    │
 │  - local clones    │    │  - sibling repos   │    │  - weft.gyrum.ai   │
 │  - dev-all.sh      │    │  - fixtures        │    │  - buzzy backends  │
 │  - sqlite fixtures │    │  - ghcr.io pulls   │    │  - production Pg   │
 │  - no prod creds   │    │  - no Ops creds    │    │  - observ stack    │
 └─────────┬──────────┘    └─────────┬──────────┘    └─────────┬──────────┘
           │                         │                         │
           │  push to GitHub         │  pull images from       │  serve users
           │  (HTTPS, PAT)           │  ghcr.io (HTTPS)        │  (HTTPS)
           │                         │                         │
           │                         │  signal Ops via         │
           │                         │  playbook API           │
           │                         │  (HTTPS + HMAC)         │
           ▼                         ▼                         ▼
        github.com               ghcr.io                  user browsers

Three rules, non-negotiable:

No shared filesystem. The Dev plane's ~/.gyrum/ is not mounted on any CI runner. CI's workspace ($GITHUB_WORKSPACE) is not mounted on any Ops host. Ops's /var/lib/gyrum/ is not mounted on any CI runner. Each plane has its own filesystem namespace, period.
No shared credentials. A Dev token (gh auth login PAT on the laptop) is not re-used as a CI secret. A CI secret (GITHUB_TOKEN, the preview-webhook-receiver HMAC key) is not re-used as an Ops credential. An Ops credential (the Cloudflare production API token, SSH keys to the production box, prod DB password) never appears in a CI secret, a CI env var, or a CI workflow file. Every plane has its own credential surface, scoped to the minimum.
No shared runtime. The laptop's Docker daemon is not the CI runner's Docker daemon. The CI runner's Docker daemon is not the Ops host's Docker daemon. Each plane runs its own containers, its own processes, its own user accounts.

The planes call each other, but only through documented, authenticated, rate-limited APIs: the Dev plane pushes to GitHub (HTTPS, PAT), the CI plane pulls images from ghcr.io and pushes built images back (HTTPS, GITHUB_TOKEN), the CI plane signals the Ops plane via cross-repo repository_dispatch or via the ADR-072 playbook-runtime HTTP API (HTTPS + HMAC), the Ops plane serves live user traffic (HTTPS). No plane mounts another plane's disk. No plane shares a process space with another plane.

This is the contract. Everything in §4 and §6 is about enforcing it in practice.

Where does the preview environment fit? ADR-073 preview environments live on the Ops plane by construction — the preview is production-adjacent, it runs under Weft's playbook runtime, and it uses Ops creds (scoped, per-preview, short-lived) to mutate Cloudflare. The preview is not CI. The CI plane's relationship to the preview is exactly the same as the CI plane's relationship to production: it can observe (via the public URL, after Access auth) and it can dispatch (via repository_dispatch to trigger a fleet E2E run against the preview URL), but it never mutates the preview's infrastructure directly. Phase D of ADR-073 (Playwright against preview) is a CI-plane consumer of an Ops-plane product. The seam is the HTTPS URL, not a shared disk.

Where does the persona-review job fit? It is a CI plane job. It runs on the same runner substrate as contract tests and fleet E2E, calls the Anthropic API with repo-scoped context, and posts a PR comment with the findings. It reads the PR diff (via GITHUB_TOKEN), reads a small slice of the repo (the files changed plus their neighbours), and calls api.anthropic.com with a token scoped to the persona-review use case. That token is a CI secret; it is not an Ops credential and cannot be repurposed as one.

Today — GitHub Actions hosted runners

We run CI on GitHub's managed Linux runners. The setup per repo is one or two workflows under .github/workflows/ — typically a build.yml (lint + test + build + push image) and a workflow or repository_dispatch target that orchestrates the fleet E2E.

What we get.

Zero ops cost. No runner to patch, no host to reboot, no disk to clean up. GitHub owns the whole surface; we own zero infrastructure.
Rock-solid isolation. Every job gets a fresh VM, which is torn down on completion. A job cannot see another job's filesystem. A malicious dependency in one PR cannot persist anywhere that another PR would see.
Scales automatically. 1 PR/day or 100 PRs/day, GitHub scales the runner fleet. We never hit a capacity wall at current load.
Cache-friendly for small caches. The actions/cache@v4 action preserves Go module caches, npm caches, and Docker layer caches across runs at a cost of a few seconds of download time per run.
Attested builds. actions/attest-build-provenance and OIDC-based ghcr.io auth give us provenance without running our own signer.

What we pay.

Billed minutes on private repos. $0.008/min Linux, $0.016/min Linux 4-core, $0.04/min Linux 8-core. At current volume we run at ~$30/month. This scales linearly with PR rate.
Layer cache indirection. Docker layers live in the Actions cache (not in our own registry mirror), so a cold run re- downloads the base layers from Docker Hub or GHCR. Each run pays a ~15-second image-pull tax.
No access to private networks. A hosted runner cannot reach Tailscale-only services, a private GHCR mirror inside our VPC, or a production-lookalike database on the Ops plane. We route around this today by exposing everything CI needs via ghcr.io (public or GITHUB_TOKEN-authed).
Runner IP churn. The runner's egress IP changes every run, so we cannot use IP allowlisting on downstream services. (We use per-run OIDC tokens or per-job GITHUB_TOKENs instead, which is the right answer anyway.)

This is the baseline. It is good enough, right now, for everything we do.

When to self-host — four explicit triggers

We move CI off GitHub-hosted runners only when at least one of these four triggers fires. Any one is sufficient; none of them is "it would be nice to have".

PR volume > 40/day across the fleet OR Actions cost > $50/month. At $0.008/min and ~5 min/run, $50/month is about 6,000 job-minutes, or ~1,200 runs, or ~40 PRs/day with 30 runs each across repos. Below this line, the $50 is cheaper than the engineering time spent on a self-hosted runner. Above, the break- even tips.
Speed matters more than cost. A cached layer on a persistent self-hosted runner is ~10× faster than a re-pulled layer on a fresh GitHub VM (observed rough number from other teams' published benchmarks; needs re-measurement against our own images once we land). If the review loop is waiting on CI latency and reviewers are context-switching as a result, self- hosted is the answer even at the same or higher $/month.
Need access to Tailscale-only services. If we stand up a private GHCR mirror inside our VPC (to eliminate the ~15s image-pull tax), a private test database with production-shaped fixtures, or any other Tailscale-only dependency for CI, a GitHub-hosted runner cannot reach it. Self-hosted, Tailscale- joined, is the only answer.
Need scheduled runs that exhaust GitHub minutes. If we commit to e.g. a nightly cross-browser matrix (Chrome + Firefox
- WebKit + mobile Chrome + mobile Safari) of the full fleet E2E suite, that's potentially hours of runner time per night, per environment, against a monthly budget. Self-hosted at €4/month is effectively uncapped.

If none of these fires, we do not self-host. We resist the urge to self-host for aesthetic reasons ("we run everything else on our boxes, we should run CI there too") or pre-emptive reasons ("we might hit the trigger some day"). The trigger is the trigger.

Anti-trigger: cost alone is not enough. We will not move a low-volume workload to self-hosted just to "save $10/month" — the engineering-ops cost of a Hetzner runner dwarfs $10/month. The trigger is $50/month or one of the other three; we do not treat $10 as a mini-trigger.

Self-hosted architecture — when we get there

When a trigger fires, here is what we stand up. This is the shape, not the config; the config lives in the ci-runner-provision playbook (a follow-up to this ADR, scoped via ADR-072).

Dedicated Hetzner CX22 (2 vCPU, 4GB RAM, 40GB SSD, €4/month). Not shared with the Ops CX52. Physically separate VM, separate Tailscale identity, separate SSH keys, separate sudoers.
Provisioned via release_flow:ci-runner-provision — the ADR-072 pattern. The playbook:
- Creates the Hetzner VM (via the Hetzner Cloud API, scoped token).
- Joins it to Tailscale with a tag:ci-runner ACL tag.
- Installs Docker, the actions/runner package, and a systemd unit that runs the runner as an unprivileged runner user (no sudo, no Docker root, rootless Docker or Podman for the build sandbox).
- Registers the runner with GitHub using a scoped registration token for the gyrum-labs organization.
- Labels it with self-hosted, fleet-e2e so jobs that want it opt in via runs-on: [self-hosted, fleet-e2e].
- Configures the actions/runner to tear down its working directory after every job (runner ephemeral-job mode) to eliminate cross-job state bleed.
Tailscale-joined, tag:ci-runner. The ACL permits the runner to pull from any private GHCR mirror we stand up, and from a future read-only production-shape test database (if we ever need one). The ACL explicitly denies reach into the Ops tailnet: no SSH to ops-vps, no access to the production DB, no access to the Weft playbook-runtime API on the Ops plane. The CI plane stays isolated by Tailscale ACL, not by convention.
Zero Ops credentials. The runner has no Cloudflare API token, no SSH key for the production box, no production DB password, no production HMAC secret. If a future CI job legitimately needs to reach Ops (we cannot think of a real case today), it does so via a scoped, short-TTL, audited token fetched at job start from an OIDC-authenticated exchange — the same pattern ADR-069 §3 uses for playbook secrets.
Same workflows. A workflow moves from hosted to self-hosted by flipping one line: runs-on: ubuntu-latest → runs-on: [self-hosted, fleet-e2e]. No workflow rewrite. The workflow still pulls ghcr.io/gyrum-labs/<repo>:<sha>, runs docker compose up, runs Playwright, and tears down — it just does these things on a box whose Docker image cache persists across runs.
Runs ephemeral containers. The runner itself is long-lived; the work it does is short-lived. Every job runs inside a container sandbox (either rootless Docker or Podman), so a job cannot escape into the runner's host filesystem. The runner's working directory is wiped between jobs.

Failure mode we accept. A compromised runner (e.g. a malicious dependency exploits a 0-day in the runner sandbox) still gets Tailscale network access with tag:ci-runner. Because the ACL forbids the Ops tailnet, the blast radius is CI plane + public internet + the private GHCR mirror. That is strictly smaller than "the whole tailnet", which is what "share the Ops box for CI" would cost us. We accept CI-plane blast radius as the price of self- hosting.

Failure mode we do not accept. A runner that can reach the production DB. A runner that has the Cloudflare API token. A runner that can ssh ops-vps. Each of these is an explicit ACL violation that review must catch.

Why a dedicated VM, and not just a container on an existing box? A container on the Ops box shares the host kernel with production services; a container escape (the canonical runner-sandbox-escape shape) gives the attacker the entire host. A container on a dev laptop is gone when the laptop sleeps. A dedicated VM — with its own kernel, its own network identity, its own sudoers, its own Tailscale ACL tag — is the smallest isolation primitive that survives the compromise shapes we care about. The €4/month is cheaper than the half-day of argument about whether container isolation is enough; we spend the money and stop arguing.

Why one box, not a pool? Pools introduce scheduling, leader election, quorum, and the operational surface of a distributed system for a workload that fits comfortably on one 4GB VM at our current scale and next-scale projection. When one box stops being enough (queue depth on the runner > N for more than a week, or runner CPU pegged > 80% for more than an hour a day), we add a second box labelled self-hosted, fleet-e2e and GitHub routes jobs to either. That is still not a "pool" in the distributed- systems sense; it is two peers, both registered, both ephemeral- job-mode, both ACL-restricted the same way. The scaling story remains linear until we see evidence it cannot be.

Why the actions/runner agent and not Gitea Actions / a self- managed orchestrator? GitHub Actions workflows are what we write today; the actions/runner agent is the only way to execute those workflows on our own compute without rewriting them. Alternative orchestrators (Gitea Actions, Woodpecker, Drone) would force a migration of workflow language on top of a migration of runner substrate — two changes at once, for a problem that neither solves better. We keep the workflow surface stable and change only the substrate.

Image pipeline — the glue (independent of where CI runs)

The pipeline that connects repos is the same whether CI runs on GitHub-hosted or self-hosted runners:

Per-repo CI builds and pushes to GHCR on every push to a feature branch and to main. Tags are <ref-sha> (immutable, content-addressed) and latest (convenience pointer for main tip).

# .github/workflows/build.yml (shape)
- uses: docker/build-push-action@v5
  with:
    tags: |
      ghcr.io/gyrum-labs/${{ github.event.repository.name }}:${{ github.sha }}
      ghcr.io/gyrum-labs/${{ github.event.repository.name }}:latest

Fleet E2E compose consumes pinned images via the FLEET_E2E_IMAGE_* environment variables the distill-gyrum-ai/e2e-integration/compose.yaml already reads. Each sibling repo's image is pinned by SHA in the workflow that triggers fleet E2E.
Cross-repo dispatch. When a sibling repo's PR merges (or when its CI completes on a PR branch we want to test against), the sibling's workflow sends a repository_dispatch to distill-gyrum-ai with types: [fleet-e2e] and client_payload: { ai_research_ref: <sha>, trigger_repo: <name>, trigger_pr: <N> }. The receiving workflow runs the fleet E2E with the pinned sibling SHA.
Works the same everywhere. Hosted or self-hosted, the pipeline is image-pull + compose-up + Playwright. The only difference is where the pull happens and whether the layer cache is hot or cold.

This shape is the stable contract. Moving a workflow from hosted to self-hosted is a one-line change to runs-on: (and possibly adding a cache configuration tuned for the persistent layer cache). Everything else — the triggering, the image tags, the compose file, the Playwright suite — is identical.

Image provenance. Every image pushed to GHCR by CI is signed via actions/attest-build-provenance (OIDC-based, no long-lived signing key). The fleet E2E compose verifies the attestation on pull (via gh attestation verify in a workflow step before docker compose up). This gives us an attestation chain from the source commit to the running preview: "this image was built by GitHub Actions from this repo at this SHA" is provable, not asserted. The attestation survives the hosted → self-hosted migration unchanged; the OIDC identity of the workflow is the signer, and that identity is the same whether the runner is GitHub-hosted or self-hosted. A self-hosted runner cannot forge an attestation from a different SHA because it does not hold the OIDC signing key; the key is minted per-job by GitHub's OIDC provider.

Image retention. GHCR retains images indefinitely on the free tier for public repos. For private repos under the gyrum- labs org, retention follows the org's GHCR settings; we keep the last 30 days of <ref-sha> tags plus every main tip tag indefinitely. Old SHA tags are swept by a scheduled playbook (out of scope for this ADR; see release_flow:ghcr-retention- sweep named-horizon). This keeps the registry bounded without deleting the SHAs that preview environments (ADR-073) and persistent PR branches may still reference.

Security + isolation rules — non-negotiable

These rules hold whether CI runs on GitHub-hosted or self-hosted runners. They are the enforcement side of the three-plane contract (§3).

CI plane NEVER gets Ops credentials. No Cloudflare production API token, no production SSH key, no production DB password, no production HMAC secret in any CI secret, CI env var, or workflow file. If a workflow file references one, that is a security finding in review, blocks merge, and is fixed before merge.
Per-PR secrets are scoped to the minimum. GITHUB_TOKEN (automatic, per-job, PR-scoped) is fine — it's what GitHub gives us to authenticate to GHCR and post PR comments. Custom tokens (the fleet-webhook HMAC secret, any Anthropic API token for persona review) are stored as secrets.* with the narrowest repo/environment scope GitHub offers, and are never echoed in logs.
Self-hosted runners run as unprivileged users. No sudo, no root Docker, no world-writable workspace. Rootless Docker or Podman for the build sandbox. The systemd unit that runs the actions/runner executes as a runner user with a locked-down shell.
Jobs run in containers only. A self-hosted runner does not execute raw shell scripts directly on the host; every job command runs inside a container image (either the repo's own build image or a pinned ubuntu:24.04). This prevents a workflow from poisoning the runner's host filesystem between jobs. The actions/runner ephemeral-job mode is the enforcement (it wipes the workspace between jobs); the container is defence-in-depth.
Public repos clone freely. No PAT required, anonymous clone is fine.
Private repos use a read-only PAT scoped to gyrum-labs. When CI needs to clone a sibling private repo (rare — most of our cross-repo work is via pinned images, not source clones), it uses a PAT with repo:read on the gyrum-labs org only. The PAT is stored in secrets.GYRUM_LABS_READ_PAT, rotated every 90 days via a playbook (same shape as the webhook-secret rotation in ADR-073 §6).
No workflow runs on pull_request_target with repo secrets. The pull_request_target trigger runs in the context of the base branch and has access to secrets; a malicious PR could exfiltrate them. We use pull_request (runs in the context of the PR branch, no access to sensitive secrets) everywhere. Any workflow that legitimately needs pull_request_target (e.g. posting labels on untrusted PRs) splits the work: the untrusted part runs on pull_request with permissions: read- all, contents: read; the trusted part runs on a separate workflow triggered by workflow_run with the minimum necessary scopes.
Secrets never appear in workflow logs. GitHub redacts secrets.* in logs automatically; we do not disable this. We also do not construct secrets by concatenation or base64-encode them in workflow steps in a way that would bypass the redaction.

These rules are enforced by review (a security-focused reviewer reads every new workflow for these patterns) and partially by convention (no shared template yet). A future reusable-workflow library (gyrum-labs/ci-workflows) will codify the compliant patterns so most repos inherit the rules rather than re-implementing them.

Worked examples of the rules, in both directions.

Allowed:

A workflow that pulls ghcr.io/gyrum-labs/distill-gyrum-ai:${{ github.sha }}, runs it in a compose stack, and exercises it with Playwright. The image is public-read-within-org; GITHUB_TOKEN authenticates the pull. No Ops credential is used.
A workflow that calls api.anthropic.com for a persona-review pass, using secrets.ANTHROPIC_API_KEY_PERSONA_REVIEW. The key is scoped to the persona-review billing account, is unique to CI, and is never used outside the persona-review workflow.
A workflow that posts a comment on the PR with the fleet E2E results, using the default GITHUB_TOKEN. The token's pull- requests: write permission is scoped per-workflow.
A workflow that fires repository_dispatch to distill-gyrum- ai with types: [fleet-e2e] and a pinned SHA, using a PAT stored as secrets.FLEET_DISPATCH_PAT. The PAT is scoped repo:read + repository_dispatch: write on gyrum-labs only.

Not allowed:

A workflow that SSHes into ops-vps to tail a log or run a deploy. SSH into the Ops plane is an Ops mutation; it belongs in a playbook, not a CI workflow.
A workflow that reads the Cloudflare production API token to validate that a DNS record exists. DNS validation from CI is allowed — but via dig against a public resolver, not via the Cloudflare API with a production token.
A workflow that reads the production Postgres connection string "just to verify the schema". Schema verification in CI runs against a fresh Postgres container in the compose stack, not against production.
A workflow that uses a PAT scoped to "all repositories" when it only needs repo:read on one repo. Over-scoped tokens are a finding.

These examples are the concrete shape reviewers check against. The reusable-workflow library (when it lands) inherits them.

Cost model at each scale

Three scales. Each shows what we pay on GitHub-hosted vs. what we would pay self-hosted (one CX22, amortized €4/month). Runner-minute assumptions: 5 min/run average.

Scale	Volume	GitHub-hosted cost	Self-hosted cost	Cheaper?
Current	~20 PRs/day, ~400 runs/month, 2,000 min/month	~$16/month	~€4/month + ops time	GitHub (ops time dominates)
Medium	~40 PRs/day, ~800 runs/month, 4,000 min/month	~$32/month	~€4/month + ops time	Close; speed tiebreaker
High	~100 PRs/day, ~2,000 runs/month, 10,000 min/month	~$80/month	~€4/month + ops time	Self-hosted (by a margin)

Two caveats on the table. First, "ops time" is the real cost on the self-hosted side and it does not appear as a line item: a Hetzner runner adds ~30 minutes/month of patching, monitoring, runner-agent upgrades, disk cleanup. That amortises across many months if the runner is stable, but the first-month cost of standing it up is a full day of engineering (provision playbook, Tailscale ACL, review, measurement). Second, the table assumes we do not pay for the 4-core or 8-core GitHub runner tiers. If we move fleet E2E to 4-core (to halve runtime), the GitHub cost doubles and self-hosted becomes a tiebreaker one scale earlier.

Reading the table backwards: at current scale, the monetary case for self-hosting is marginal and dominated by ops overhead — do not move. At medium scale (where we are within a year if ADR-073 Phase B ships on schedule), revisit. At high scale, self-host.

Migration path — when we move

The migration is additive. No cutover, no "flag day". The phases:

Phase 1 — add self-hosted alongside GitHub-hosted.

Run the ci-runner-provision playbook. The runner registers with GitHub Actions, labelled self-hosted, fleet-e2e.
Zero workflows change. Every job still runs on ubuntu-latest. We now have a runner sitting idle.
This is the "dry run" of self-hosted: we can ssh into the runner (via Tailscale), verify Docker is rootless, verify the ACL forbids Ops reach, verify the runner is healthy.
Exit criteria: runner is green in the GitHub Actions "Runners" UI for 72 hours with no jobs routed to it.

Phase 2 — selectively route jobs to self-hosted.

One workflow flips to runs-on: [self-hosted, fleet-e2e]. We pick the highest-value candidate: whichever job has the longest image-pull phase, because self-hosted's persistent layer cache is where self-hosted shines. Fleet E2E is the obvious first candidate.
We keep all other jobs on ubuntu-latest. A failure of the self-hosted runner does not break the fleet; those other jobs still run on GitHub-managed infrastructure.
We measure. Image pull time, total job time, cache hit rate, ops incidents (runner disk full, runner clock drift, runner agent update broke the daemon). Two weeks of data.
Exit criteria: the migrated job runs at least 20% faster on self-hosted than on hosted, with no self-hosted-specific failures that did not also affect hosted.

Phase 3 — monitor, tune, potentially retire GitHub-hosted for that job family.

If Phase 2 is a win, we migrate more jobs in the same family (contract tests for repos whose image-pull time is high, then persona review jobs if their runtime is dominated by model API latency — though it almost certainly is not, so they stay hosted).
We keep hosted as a fallback: any job with runs-on: [self- hosted, fleet-e2e] can be flipped to ubuntu-latest in one line if the self-hosted runner is down.
Exit criteria: the self-hosted runner carries the majority of minute-weighted CI load; GitHub-hosted is our fallback, not our primary.

Rollback. One config flip. Every runs-on: [self-hosted, fleet-e2e] reverts to runs-on: ubuntu-latest, and CI is back on GitHub-hosted. We can do this in under five minutes fleet- wide (one commit per repo, or a scripted gh command that edits workflow files in bulk). The runner itself can stay up idle — or be torn down by the paired release_flow:ci-runner- decommission playbook.

Observability during and after migration. The runner emits three metrics to the ADR-055 observability stack: ci_runner_ status (0/1), ci_runner_queue_depth (jobs waiting), and ci_runner_job_duration_seconds (histogram). An alert fires on ci_runner_status == 0 for > 10min so we notice a down runner before reviewers do. A dashboard under Observability → CI shows the time series alongside the hosted-runner cost trend (pulled from the GitHub billing API on a daily cron) so we can see the cost-vs-speed tradeoff in one place. This is the same dashboard pattern ADR-055 §4 defines; we inherit it rather than invent a new one.

Per-job routing. The runs-on: [self-hosted, fleet-e2e] label selector is GitHub's native mechanism; we do not layer a router on top. If a job is mis-routed (e.g. it landed on self- hosted but should have stayed hosted for security reasons), the fix is a label change in the workflow file, not a routing-layer config change. Keeping routing in the workflow file — where anyone reading the workflow can see where it runs — is the transparency discipline we commit to.

Consequences

Easier.

Right now: zero ops cost. We ship on GitHub-hosted and the only infrastructure we own is the .github/workflows/ files. No runner to patch, no Tailscale ACL to maintain, no disk to clean up.
Instant value. Every repo in the gyrum-labs fleet already has this today; no migration needed to stay on the happy path.
The move-trigger is explicit, not vibes. When somebody proposes "let's self-host CI", the answer is "which of the four triggers has fired?". If the answer is "none, but it would be nice", the answer is "no, and here's why". This prevents a class of infrastructure-by-accretion that most teams suffer.
Self-host is real work when earned. The ci-runner- provision playbook, the Tailscale ACL, the monitoring, the rotation — each is a real artefact. We do this work once, when the trigger has fired, not pre-emptively.
Plane isolation is enforced at config and convention. The three rules (§3) are simple enough that every reviewer can hold them in their head. A workflow that breaks them is caught in code review, flagged as a security finding, and fixed before merge.

Harder.

The plane boundary is a convention, not a firewall. A careless change to a workflow — adding secrets.PROD_SSH_KEY to a CI job — would breach the rule without the runner complaining. We rely on review to catch this. A future hardening step (explicit deny-list in a CODEOWNERS-enforced lint on .github/workflows/**) is named but not in scope.
Self-hosting, when we do it, is real operational work. A runner that falls over on a Sunday afternoon blocks merges. The ci-runner-provision playbook, the monitoring (Prometheus scrape of the runner's health endpoint, alerts on "runner offline > 15 min"), the rotation of registration tokens — each needs to be built, maintained, and owned.
Private-network CI creates a new surface. Once the runner is Tailscale-joined, it has network access to the tag:ci-runner ACL targets. If we ever stand up a private GHCR mirror inside the VPC, the runner can pull from it — and so could a compromised runner. The ACL and the rootless- Docker posture are what keep the blast radius bounded; a relaxation of either is a security regression.
Cost grows linearly on hosted, steeply at the low end. At 20 PRs/day we spend $16/month; at 100 PRs/day we would spend $80/month; above that, GitHub's higher-core tiers start being tempting and the cost grows super-linearly. The "GitHub is cheaper" argument is monotonic in the current direction; it flips at ~$50/month (the trigger).
Fallback on self-host still depends on hosted. If we eventually run most CI on self-hosted, a self-hosted outage means jobs queue up waiting. We mitigate by keeping ubuntu-latest as a one-line fallback (§9 Rollback), but an operator has to push the flip. A future "automatic fallback on runner offline" is doable but non-trivial and out of scope.

What we sign up to maintain.

The three-plane contract (§3). Every new workflow, every new CI job, every new self-hosted runner is reviewed against the three rules. Breaking a rule is a security finding, not a design question.
The trigger list (§5). When somebody proposes moving to self-hosted, we check the triggers. When a trigger fires, we move — not before.
The ci-runner-provision playbook (when we get there). Idempotent, dry-runnable, scoped-tokened, ADR-072-shaped. Rolls up, rolls down, and is the single source of truth for what the runner is.
The Tailscale ACL for tag:ci-runner. Reviewed whenever it changes. Denies Ops reach by default; additions are explicit and justified.
The image-pipeline contract (§6). ghcr.io/gyrum- labs/<repo>:<ref-sha> on every push. FLEET_E2E_IMAGE_* env var on every fleet E2E run. repository_dispatch with client_payload.ai_research_ref for cross-repo signals. These are stable and portable across hosted/self-hosted.

Alternatives considered

Run CI on the Ops box ("it already has Docker"). Rejected, with prejudice. The Ops box runs production services carrying live traffic; running untrusted PR code on the same kernel is the single worst CI-plane violation available to us. A compromised dependency in one PR could read the production Postgres socket, or exfiltrate the Cloudflare API token from the Ops environment, or simply consume enough RAM to evict a production container. This is not a theoretical concern — it is the canonical shape of a runner-compromise post-mortem. No trigger fires hard enough to justify this; there is no negotiable version of this decision.
Run CI on the dev laptop. Rejected. The laptop is the Dev plane, is not shared, goes to sleep when the engineer closes the lid, and has no stable identity. CI needs a stable endpoint and a persistent identity to receive webhooks, post status checks, and push images to GHCR — the laptop is none of these. The Dev plane is for the author's inner loop, not for the team's review gate.
Buildkite. Acknowledged. Buildkite's model ("bring your own runners, we orchestrate") is actually very close to the self- hosted pattern we would adopt in §6, with a better UI for pipe- line composition. Rejected because (a) GitHub Actions is already in the team's flow — the workflow language, the secrets management, the status checks, the review integration — and switching costs are real; (b) the self-hosted path on GitHub is adequate and shipping more CI surface area is a net-negative for operator cognitive load; (c) we have no trigger that Buildkite solves and GitHub self-hosted does not. The escape hatch: if we ever need multi-repo pipeline orchestration more sophisticated than repository_dispatch, revisit.
CircleCI / Travis / Jenkins. Acknowledged and rejected for the same reason as Buildkite — not in the flow, and GitHub Actions covers the use cases we have. Jenkins specifically would be a large new operational surface (we would own a Jenkins master, its plugins, its auth model) that buys us nothing GitHub does not already provide.
BuildJet (drop-in faster GitHub-hosted runners). Considered. BuildJet offers faster GitHub-Actions-compatible runners at a lower per-minute cost than GitHub's 4-core tier. Rejected for now because: (a) trigger #1 ($50/month) has not fired; (b) the speed gains are real but marginal compared to a persistent- layer-cache self-hosted runner; (c) we would introduce a new vendor relationship for a problem we can solve with €4/month of Hetzner when the time comes. If trigger #2 (speed) fires before the cost trigger, BuildJet is reconsidered as a halfway house between hosted and self-hosted.
Multi-region runners. Deferred. Running CI runners in multiple Hetzner regions (for resilience, or for running tests against region-pinned services) is a real question for later, not for this ADR. The one-box self-hosted pattern is the right first step; multi-region is an evolution.
GitHub-hosted larger runners ($0.016/min 4-core, $0.04/min 8-core). Acknowledged. Larger hosted runners are the right first step when a specific job is CPU-bound on 2-core. We reach for them before self-hosting, on a per-job basis, if a single job's runtime is the bottleneck and the job is infrequent enough that the 2-3× cost is tolerable. Fleet E2E at ~6 minutes on 2-core is not yet in this category; if it grows to ~15 minutes we reconsider.

ADR-020 — shared infra vs per-product isolation; the three- plane rule in §3 is the CI-specific refinement of ADR-020's isolation stance
ADR-053 — frontend observability; the self-hosted runner, when it lands, emits structured logs and a Prometheus /metrics endpoint (runner health, queue depth, job duration) to the same observability stack as production — scraped by the ADR-055 observability pinning
ADR-055 — observability stack container pinning; the CI plane pulls pinned observability images when running fleet E2E, same as the Ops plane
ADR-069 — playbook security; the CI-plane credential rules in §7 mirror ADR-069's scoped-token and secret-vault posture (narrow, short-lived, audited)
ADR-072 — infrastructure as playbooks; release_flow: ci-runner-provision is an ADR-072-shaped release_flow playbook composing sub-playbooks (hetzner-vm-create, tailscale-join, systemd-unit-upsert, actions-runner- register), introduced in a follow-up PR if and when a trigger fires
ADR-073 — preview environments per feature branch; preview-up is an Ops-plane mutation (not a CI step); this ADR's three-plane rule is what keeps preview-up from being confused with CI runner work
Fleet E2E TUTORIAL — distill-gyrum-ai/e2e-integration/ TUTORIAL.md (landing concurrently) documents the image- pipeline contract (§6) from the consumer side; this ADR documents it from the plane-separation side

Review cadence

Re-review every 90 days (next: 2026-07-22) against:

Which triggers are close? Monthly Actions spend vs. the $50/month trigger. PR volume vs. the 40/day trigger. Any new pressure to access Tailscale-only services. Any scheduled-run proposal that would exhaust hosted minutes.
Plane breaches. How many workflows tried to put an Ops credential in a CI secret in the last 90 days, and were caught in review? Zero is the target; > 0 means the rule needs better tooling (a lint, a reusable workflow template), not just better reviewers.
Workflow drift. How many repos are on a per-repo workflow file vs. a shared reusable workflow? The direction is toward reusable workflows in gyrum-labs/ci-workflows; this review checks progress.
Self-hosted readiness. If we have not self-hosted yet, is the ci-runner-provision playbook ready to run (or is it a cold-start-from-scratch job when the trigger fires)? Named: prepare the playbook one cycle before a trigger is expected to fire, not the week of.
Rollback tested. If we have self-hosted, has the one- config-flip rollback been exercised in the last 90 days (at least as a drill)? Untested rollbacks rot.

Stale flag on the Owner dashboard once last_reviewed on this ADR passes 90 days.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one