ADR-134: Runtime-base image as unit of isolation for CI + agent-execution runners

Status: Accepted Date: 2026-05-05

Context

The current self-hosted runner pool (gyrum-ci-runner-1, gyrum-ci-runner-2 — both Hetzner CAX Ampere ARM64 hosts) leans on the host as the unit of isolation. Toolchain (node, go, python), utility binaries (gh, jq, rg, fd), playwright OS deps (libnss3, libxss1, libasound2, …), and the gyrum-* devtools are all installed on the host via a bootstrap script. Adding a new runner means re-running that script — captured today as warp#1370 (per-repo SvelteKit checklist) and surfaced repeatedly through the warp self-hosted migration epic (warp#1171) and warp's Phase B cutover (warp#1363).

This worked fine while the runner pool's only consumer was CI. It does not survive contact with the planned agent-execution pool (forthcoming) for three structural reasons:

Heterogeneous workload. A CI job is "build + test + lint a single repo's tree". An agent job is "claim a warp ticket, do whatever the ticket says, push a PR" — which can include shelling out to any tool the agent decides it needs. A host bootstrap that pre-installs every conceivable tool is a maintenance treadmill; a host that doesn't is a permission-denied wall.
Isolation. Multiple agents running concurrently on the same host need stronger boundaries than chmod and unix users provide. An agent job that misbehaves (writes to /tmp carelessly, leaves daemons running, mutates the runner's git config) leaks into every subsequent job on the same host.
Reproducibility. A runner host that has drifted from the canonical bootstrap (manual apt install to chase a flaky build, a one-off npm install -g for a debugging session) is hard to detect and harder to reset. The fleet's auditability story stops at "the host was bootstrapped from the script at some point" — which is not a story.

Operator (jon) endorsed on 2026-05-05 the architectural shift this ADR captures: the unit of isolation moves from the host to a hardened Docker image. Each agent run, and each CI job once migrated, spawns a fresh container from the image. The runner host's job is to manage container lifecycle — claim work, run the container, ship the output, terminate the container — not to be the toolchain.

Decision

Adopt ghcr.io/gyrum-labs/runtime-base:v<X> (published to GitHub Container Registry — GHCR) as the canonical container image for both the agent-execution pool and (over time) CI workflows. The image is hardened (read-only-rootfs, non-root user, dropped capabilities, network egress through a logging proxy, cosign-signed-and-pinned-by-digest), multi-arch (ARM64 + x64), and contains the full toolchain a fleet job is allowed to assume. Runner hosts install only docker + the actions-runner agent; everything else lives inside the image.

The same model retroactively applies to CI workflows — they migrate from runs-on: [self-hosted, linux, arm64] directly executing on the host to a containerised wrapper that spawns the same runtime-base image. The bootstrap script's apt-deps approach (warp#1370) is superseded over time as Phase 4 of this ADR's phasing lands.

Image contract

The contract a consumer (CI workflow, agent-execution job) can rely on the image to satisfy:

Base layer. Debian 12 (bookworm). Multi-arch image; docker manifest resolves to linux/arm64 or linux/amd64 at pull time per the runner host's platform. The choice between Debian 12 and Ubuntu 22.04 is settled on Debian 12 for its smaller default surface and more conservative apt-default policy; deviations require an ADR amendment, not a Dockerfile edit.
Pinned toolchains. node@22, go@1.23, python@3.12, plus utility binaries (gh, jq, rg, fd, bash >= 5, curl, git >= 2.40). Every version is pinned by full version string in the Dockerfile (FROM golang:1.23.4-bookworm, RUN curl -fsSL https://nodejs.org/.../node-v22.11.0-linux-arm64.tar.xz). Toolchain bumps are PRs against the image repo, not silent latest-tag drift.
Pre-installed OS deps for browser tooling. The playwright OS shim list captured in warp#1370 (libnss3, libxss1, libasound2, libatk1.0-0, libcups2, libxkbcommon0, …) is apt install-ed at image-build time. npx playwright install --with-deps becomes a no-op at runtime — the deps are already present, the browsers fetch into the bind-mounted cache. No more per-job apt install.
gyrum-* devtools binaries baked in. Built from the canonical devtools repo at image-build time and installed under /usr/local/bin/. The image's published version pins to a specific devtools commit SHA (the image's LABEL gyrum.devtools.sha=... exposes which). Fleet-wide devtool bumps land via a coordinated image rebuild, not via an unsynchronised host-by-host re-bootstrap.
Non-root user gyrum-runner (uid 1000). The image's default USER is gyrum-runner. No sudo. No setuid binaries beyond what the base distro requires. The container runs without --privileged, drops ALL capabilities at run time, and re-adds only NET_BIND_SERVICE if the workload genuinely needs it (rare; most agent jobs do not).
No persistent state inside the container. Anything a job needs to keep across the container's lifetime is written to a bind-mounted host directory: /cache (read-only, content-addressed — see the runner-cache-contract ADR cross-link below), /git-mirror (read-only, pre-fetched git mirror of the target repo), /secrets (read-only, short-lived, scoped to this job), /output (read-write, the only writable mount; rsynced back by the runner agent after termination). /tmp is a tmpfs and disappears at container exit.
Network egress through a logging proxy. The container is run on an isolated network with HTTP_PROXY / HTTPS_PROXY / NO_PROXY env vars pointing at a host-resident logging proxy (mitmproxy or equivalent). Every dependency fetched by npm install, go mod download, pip install, apt install (rare — pre-installed at build time) flows through the proxy and is auditable. A container that tries to bypass the proxy via direct DNS to *.npmjs.org fails because the isolated network has no route off-host except via the proxy.
Image signed with cosign. The build pipeline signs the image with the fleet's cosign key on publish. Workflows pin the image by @sha256:<digest>, not by :vX tag. A compromised tag rewrite can't redirect a workflow because the workflow holds the digest, not the tag.

Per-job container lifecycle

The per-job lifecycle, from the runner host's perspective:

Runner agent on host claims work (warp ticket OR GitHub Actions queue)
  ↓
docker run --rm \
  --user gyrum-runner \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --network <isolated-net-with-proxy> \
  -v /var/cache/runtime/<repo>/<workload>/<lockhash>/:/cache:ro \
  -v /var/cache/runtime/git-mirrors/<repo>:/git-mirror:ro \
  -v /var/lib/runtime/secrets/<job-id>:/secrets:ro \
  -v /var/run/runtime/output/<job-id>:/output \
  ghcr.io/gyrum-labs/runtime-base@sha256:<digest>
  ↓
Container does its work, writes to /output
  ↓
Runner agent rsyncs cache-worthy output back to /var/cache/runtime/.../<lockhash>/
  ↓
Container terminates; host removes /var/run/runtime/output/<job-id> scratch

The host's responsibilities reduce to: keep docker running; keep the actions-runner agent registered; keep the cache directories and git mirrors fresh; pull the pinned image digest before each run; rotate secrets bind mounts. None of "install node", "install playwright deps", "install gyrum-* binaries" appear on the host's runbook anymore — they are the image's responsibility.

Hardening dimensions

The hardening posture, named so an auditor can check each one:

Read-only root filesystem. --read-only flag on every run; only tmpfs /tmp and the explicit bind mounts are writable.
No persistent state by default. --rm removes the container at exit; tmpfs /tmp discards scratch. Anything to keep is in /output or a cache mount.
Cache mounts content-addressed. /cache and /git-mirror are read-only and indexed by lockfile hash + repo SHA per the runner-cache-contract ADR (sister cross-link below). A poisoned cache from one job does not contaminate the next.
Secrets via short-lived bind mounts, not env vars baked into the image. /secrets/<job-id>/ is populated immediately before run, removed immediately after. No secret ever appears in docker inspect, no secret is part of the image layers, no secret is recoverable from a cached image pull.
Image rebuild cadence. Weekly auto-rebuild on a cron (picks up base-distro CVE patches without a human in the loop) plus on-demand rebuild on toolchain version bump (PR against the image Dockerfile triggers a rebuild). The cron rebuild is itself a playbook fire so it is auditable through the same /operate surface as any other fleet event.
Cosign-pinned by digest. Workflows hold ghcr.io/gyrum-labs/runtime-base@sha256:<digest>, not :latest or :vX. Tag rewrites do nothing.
Egress through logging proxy. Container's outbound HTTP(S) traffic is logged at the proxy; direct egress is blocked at the network layer. A surprise upstream dependency change is visible in proxy logs.
No privileged mode. --privileged is never set. Workloads that genuinely need it (rare; usually a smell) take the audited operator-override path, not a default-on flag.
Non-root user. gyrum-runner (uid 1000), no sudo, no setuid escape paths beyond stock distro binaries.
Drops ALL capabilities by default. --cap-drop ALL. Workloads that need a capability re-add it explicitly (--cap-add NET_BIND_SERVICE); the default is no capabilities, the deviation is the visible thing.

Consequences

Easier

Agent-runner pool provisioning. A new agent-execution host's bootstrap reduces to "install docker, install the actions-runner agent, point at the image digest". The image is the host-bootstrap state. Scaling the pool is a Hetzner cloud-init template, not a 200-line shell script.
Shared isolation model. CI workflows and agent jobs live inside the same image with the same hardening posture. Reasoning about "what can a job touch" reduces to "what's in the image, what's bind-mounted, what's behind the proxy" — one model, two consumers.
Image rebuild benefits all consumers. A toolchain bump or a CVE patch is one PR against the image Dockerfile; every CI workflow and every agent job picks it up on next pull. Today the same change requires a fleet-wide bootstrap-script-rerun coordination problem.
Auditable egress. The logging proxy makes the dependency surface visible in a way apt install on a host never was.
Per-job cleanliness. --rm + read-only-rootfs means a misbehaving job cannot leave the host in a worse state than it found it. Today a job that mutates ~/.npmrc or leaves a daemon running affects the next job on the same host.

Harder

Image build pipeline is a new thing to maintain. Dockerfile, multi-arch buildx, cosign signing, GHCR publishing, weekly rebuild cron, on-bump-trigger. None of these exist today. The cost is real and lands in Phase 0.
Per-job docker-run cold-start. ~2-3s of overhead per job (image pull from local cache, container start, mount setup) replaces ~0s of overhead today. Offset substantially by the eliminated per-job apt-install (~10-30s for playwright deps), eliminated per-job npm-install-from-cold (~30-90s), and eliminated per-job browser download (~5-15s).
Cache contract is now load-bearing. Today caches are nice-to-have; once the toolchain lives inside an immutable image, the bind-mounted caches are how dependency download cost stays low. The runner-cache-contract ADR (sister cross-link) carries this weight.
Operator mental model has to update. "ssh to runner-1 and check what node version is installed" stops being meaningful — the answer is "whatever's in the pinned image digest the last job ran". The fleet docs (sister: warp#1171) update to point at image inspection, not host inspection.

What we sign up to operate

The image build pipeline. Dockerfile, multi-arch buildx config, cosign signing key + rotation, GHCR publish workflow, weekly rebuild cron. Phase 0 of the phasing below.
The image's content as a versioned API. Toolchain pin bumps are PRs with a regression check (does CI still pass against the new image?). The image's LABEL set carries gyrum.devtools.sha, gyrum.toolchain.node, gyrum.toolchain.go, etc. so a workflow can interrogate which versions it ran against.
The cache-population side-channel. When a fresh image lands and the cache is cold, the first job pays the full dependency-download cost. A weekly "warm the cache" job populates the common cases (gyrum-labs/* repos' package-lock.json and go.sum hashes) so the first real job hits warm caches.
The proxy's log retention policy. Egress logs are sensitive (URLs reveal what packages a project depends on). Retention is bounded; access is audited; the proxy itself runs under the same hardening posture as the runner host.
The migration roadmap. Each CI workflow migrated from "host-direct" to "runtime-base container" is a PR with a verification step; the deprecation of the apt-deps bootstrap path (warp#1370) is the terminal milestone.

Phasing

The migration is phased so each step is independently revertable and the fleet keeps shipping:

Phase 0 — image build pipeline. Dockerfile in a new repo (or a new directory under devtools — Phase 0 picks the home), multi-arch buildx, cosign signing, GHCR publish on merge to main. First image tagged v0.1.0. Acceptance: docker pull ghcr.io/gyrum-labs/runtime-base:v0.1.0 works on a clean host, the image runs node --version / go version / gh --version and reports the pinned versions.
Phase 1 — agent-execution pool uses runtime-base from day one. New runners spun up under the agent pool launch containers from the image. No host-installed toolchain on these hosts. Acceptance: an agent claims a warp ticket, runs the work inside the container, pushes a PR, terminates the container — all on a host whose apt list --installed mentions only docker and actions-runner.
Phase 2 — migrate one CI workflow to runtime-base. Pick a low-risk workflow whose isolation requirements are already container-shaped (warp's contract.yml runs in docker-compose for its postgres dep, so it is a natural fit). Switch from runs-on: [self-hosted, linux, arm64] directly to a containerised wrapper that spawns the runtime-base image, and verify CI still passes. Acceptance: contract.yml green on the new shape across at least 5 PRs.
Phase 3 — migrate remaining CI workflows. As each workflow is verified working under the new shape, flip it. Workflows that have host-specific assumptions (privileged docker socket access, host-network needs) get explicit overrides documented in their workflow file rather than special-cased in the host bootstrap.
Phase 4 — deprecate the host-bootstrap apt-install path. warp#1370's per-repo SvelteKit bootstrap checklist is superseded. The runbook for "add a new runner host" reduces to: install docker, install the actions-runner agent, register against the pool. The image is the toolchain.

Alternatives considered

Stay with host-installed toolchain; harden the bootstrap script. Considered. Rejected because the failure modes named in Context (heterogeneous agent workload, isolation, reproducibility drift) are properties of the host-as-isolation-unit choice, not of bootstrap-script quality. A more rigorous bootstrap script does not change which surface the agent's mistakes land on.
Use a VM per job (Firecracker / Kata / Hetzner cloud-init-per-job). Considered. Rejected on cold-start cost (VM boot is ~5-30s, not 2-3s), on operational complexity (VM lifecycle management is a substantial new system), and on YAGNI — the isolation gap between a hardened container and a VM is small relative to the gap between a host-shared environment and a container. If the threat model later requires VM-grade isolation (operator note: untrusted-tenant agent jobs), this ADR is supersedable; the per-job container shape remains valid against the inner shell of the VM.
Use Nix-shell / devbox / asdf for per-job toolchain isolation. Considered. Rejected on isolation depth — a Nix shell shares the host kernel, host filesystem, host network namespace, and host processes with every other job on the host. The reproducibility win is real but the isolation win is absent, which is the load-bearing failure for the agent-execution pool.
Use systemd-nspawn instead of docker. Considered. Rejected on tooling familiarity (the fleet already operates docker via warp's compose-deploy runbooks; nspawn is a parallel toolchain with no current operator) and on multi-arch build ergonomics (buildx is mature for multi-arch; nspawn's image-building story is rougher).
Use a single image per workload type rather than one shared image. Considered. Rejected on substrate count — a per-workload image means N images to rebuild on a CVE patch, N pipelines to maintain, N digests to pin in workflows. The shared-image approach trades a slightly larger image (~1.5GB vs ~500MB per specialised) for one rebuild pipeline. If the shared image's size becomes a bottleneck (slow first-pull cold-starts), the layered split — base layer (OS + utilities), language-specific layers (node, go, python) — is an in-place refinement, not a rearchitecture.

Cross-link

ADR-136 — runner-cache contract for CI workflows + agent-execution pool (sister ADR). Defines the eight-principle on-disk shape of the bind-mounted /cache and /git-mirror directories — content-addressed, lockfile-keyed, arch-tagged, security-boundary-scoped. This ADR's per-job lifecycle assumes that contract.
warp#1370 — per-repo SvelteKit bootstrap checklist. Superseded over time as Phase 4 of this ADR lands.
warp#1171 — fleet-wide self-hosted migration epic. Phase 2 of this ADR is its eventual closure.
warp#1363 — warp Phase B cutover. The first migration that exposed the per-host-deps friction this ADR responds to.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one