ADR-136: Runner-cache contract for CI workflows + agent-execution pool

Status: Accepted Date: 2026-05-05 Related: warp#1384 (first concrete instance — PR#233 in flight), warp#1370 (per-repo SvelteKit checklist), warp#1363 (Phase B self-hosted ARM64 runners), warp#1171 (fleet-wide self-hosted migration epic)

Context

Phase B of the self-hosted runner migration (warp#1363, PR#228) flipped warp's CI to self-hosted ARM64 runners. The follow-on ticket warp#1384 (PR#233, in flight tonight) added per-workflow caching for node_modules and ~/.cache/ms-playwright so the second-and-onwards runs avoid the 90-second cold-install penalty. In doing so it surfaced a small set of decisions that look local to "the warp CI" but are not — every other repo on self-hosted runners will hit the same questions, and the future agent-execution runner pool (the L2 executor's runtime layer, sister to but distinct from CI) will hit them too.

Operator (jon, 2026-05-05 prime-time) endorsed lifting the warp#1384 patterns into a portable contract: same cache-key shape, same idempotency rule, same restore-keys posture, same security-boundary discipline across CI runners and the agent-execution pool. Without this ADR each workflow author re-derives the rules from first principles (or worse, copies the warp shape without understanding it), and the agent-runner pool ships with a different cache mental model than CI — guaranteeing a class of "works on CI, breaks in production agents" bugs.

The cost of getting cache shape wrong is not "slightly slower CI" — it is silent corruption (cross-arch reuse, stale-deps poisoning), invisible test flakes (partial-restore that left a half-installed node_modules), and cross-tenant leakage (two projects sharing a cache namespace, one poisons the other). These failure modes don't surface in normal logs; they surface as "the test passes locally and on a fresh runner, fails sporadically on a warm one". Locking the contract before more workflows adopt caching is much cheaper than retrofitting.

Decision

Every cache used by a runner — CI workflow or agent-execution pool — MUST obey the eight principles below. The principles apply equally to actions/cache@v4 (today's CI substrate), to host-local cache directories (the future agent-runner default), and to any third-party cache action a workflow author reaches for. The principles are the contract; the substrate underneath is an implementation detail.

1. Hash content, not branch names

Cache keys MUST be derived from the content that determines the cache contents — the lockfile hash, requirements.txt hash, go.sum hash, Dockerfile hash, or equivalent. Branch-keyed caches are wrong in both directions: different branches with the same dependency set fail to share (waste), and the same branch with different dependency sets at different points in time poison each other (silent breakage).

Concrete: warp#1384 keys its node_modules cache on ${{ hashFiles('**/package-lock.json') }} and its playwright cache on the same. A feat/foo branch and feat/bar branch with identical lockfiles share. A main branch whose lockfile changed yesterday gets a fresh key today and a clean install — not a poisoned restore from yesterday's deps.

2. Always include arch in the key

Cache keys MUST include the architecture (and, where relevant, the OS) the cache was built on. node_modules carries arch-specific natives (sharp, esbuild, libvips bindings); chromium binaries are arch-specific by definition; compiled Go binaries are obviously arch-tagged; Python wheels carry arch-tagged C extensions. Cross-arch cache reuse is silent corruption — the build succeeds, the runtime crashes deep inside a native module.

Concrete: warp#1384 keys node_modules as node-modules-${{ runner.arch }}-${{ hashFiles('**/package-lock.json') }}. Phase B's ARM64 runners and any residual x86_64 runners cannot share, by construction. The future agent-runner pool (mixed-arch by design — ARM64 graviton + x86_64 throughput) inherits the same key shape.

3. Make install steps idempotent; always run them

Install steps (npm ci, playwright install, pip install, go mod download) MUST be idempotent and SHOULD always run, regardless of cache-hit state. The "always run" pattern is more robust against partial-restore silent failures (a cache that restored 90% of a directory and left the rest stale is harder to detect than a cache miss that triggers a fresh install).

The exception: a non-idempotent install command that is expensive on hit (e.g. npm ci is a pure linker step that wipes node_modules and re-links — running it on a hit is wasteful, not unsafe). For those, gating on cache-hit != 'true' is acceptable. The bar: gate only when the install command's "no-op on hit" cost is substantial AND the command is verifiably safe to skip.

Concrete: warp#1384 gates npm ci on cache-hit != 'true' (linker step, safe to skip on hit, wasteful otherwise). It does NOT gate playwright install — playwright's install is a no-op on hit (it sees the binaries are already in ~/.cache/ms-playwright and returns in milliseconds), so the safer "always run" pattern applies. The pattern is "default to always-run; gate only with justification".

4. `restore-keys` is a footgun by default

Partial-key fallback (restore-keys in actions/cache@v4, equivalent in host-local caches) MUST NOT be enabled by default. Partial-restore risks installing an incompatible cache from a stale dependency set — the cache was built when package-lock.json had a different shape; restoring it produces a node_modules that doesn't match today's lockfile, and the subsequent npm ci might fix it but might also leave residual files the linker doesn't touch.

The simpler shape: cache miss → fresh install → save. Slightly slower on key-changes (every lockfile bump pays a full install), structurally safer (no possibility of "cache from two days ago partially restored").

Restore-keys MAY be enabled when partial-restore is provably safe for the workload — typically read-only caches (a downloaded toolchain that doesn't change) or caches whose install step is robust against pre-existing junk (pip install --upgrade on a partially-populated site-packages is reasonable; npm ci on a partially-populated node_modules is not). The bar is "the install step would produce the same final state regardless of what the partial restore left behind".

Concrete: warp#1384 does not use restore-keys. The follow-up question "should we?" is answered no — the few-seconds saving on lockfile-bump days is not worth the silent-corruption risk on the much more common "deps are stable, key matches exactly" path.

5. Cache at the host level when you own the host

actions/cache@v4 pays a network round-trip to GitHub's CDN — typically 5-10 seconds on a warm cache, more on a large one. When the runner host is owned by the fleet (self-hosted CI runners, agent-execution pool nodes), a host-local cache directory at /var/cache/<pool>/<workload>/<key>/ is instant.

The decision: the future agent-runner pool defaults to host-local caching. CI workflows MAY stay on actions/cache@v4 for now — one less moving part during the Phase B / Phase C migration, and the GitHub-cache UX (eviction, observability) is well-understood. Migration of CI to host-local is deferred to measurement (open question 2).

Concrete: agent-runner pool nodes will mount /var/cache/agent-runner/ as a fleet-managed volume; workloads address it via /var/cache/agent-runner/<workload>/<key>/. CI workflows continue using actions/cache@v4 until measurement justifies the migration cost.

6. Bound cache size with a cron, not vibes

Unbounded caches grow until disk-full. Every cache substrate MUST have an explicit eviction policy:

actions/cache@v4: GitHub Actions evicts on its own (10GB per repo, LRU). No additional cron required.
Host-local caches: a cron job evicts entries older than a documented threshold OR enforces an LRU (Least Recently Used) size cap. Concrete shape: find /var/cache/<pool> -mtime +30 -delete for time-based, or a small LRU-eviction script for size-based. The choice is per-workload.

The bar: every host-local cache directory has a documented eviction policy and a scheduled job that enforces it. "We'll clean it up if we hit disk-full" is not a policy.

Concrete: the agent-runner pool ships with /etc/cron.daily/agent-runner-cache-evict running an LRU-eviction script with a per-workload size cap (default 5GB per workload, overridable). No CI workflow currently needs this — actions/cache@v4 covers it.

7. Don't share caches across security boundaries

Two projects MUST get two cache namespaces, even if their dependency sets overlap. Otherwise a poisoned cache (corrupted on disk, tampered with by a malicious dependency, restored from a compromised key collision) leaks across projects.

For CI: actions/cache@v4 is repo-scoped by default — the boundary is automatic. The principle holds: do not engineer cross-repo cache sharing without an explicit ADR.

For host-local (agent-runner pool): the cache path MUST include the project / tenant identifier. /var/cache/agent-runner/<tenant>/<workload>/<key>/ — never /var/cache/agent-runner/<workload>/<key>/ shared across tenants. Multi-tenant agent runners are the failure mode this principle is engineered to prevent: a poisoned cache in tenant A's namespace cannot affect tenant B's runs.

Concrete: the agent-runner pool's path scheme includes <tenant> as a mandatory prefix; the runtime refuses to mount a cache path missing the tenant segment. Cross-tenant cache sharing is a separate, ADR-gated decision.

8. Never let a restore failure block work

A cache restore failure (network error, corrupted cache, unreadable directory) MUST fall through to a fresh install transparently. The runner does not abort; it logs the failure and proceeds as if the key had missed.

For actions/cache@v4: this behaviour is the default. Cache restore failures are non-fatal; the workflow continues and a subsequent "save" step writes a fresh entry.

For host-local caches: the workflow MUST handle the failure explicitly. The canonical shape:

[ -d "$CACHE" ] && cp -a "$CACHE/." node_modules/ || npm ci

The || branch fires on cp failure as well as [ -d ] failure — any path where the cache could not be honoured ends in a fresh install. No silent half-restore, no abort.

Concrete: the agent-runner pool's cache helper (the gyrum-runner-cache wrapper, deferred to open question 1) implements this fall-through internally so workload authors get it for free; until then, host-local cache uses MUST hand-roll the && copy || install pattern.

Consequences

Positive

One mental model across CI and agent runners. A workflow author who learned the rules on warp's CI applies them unchanged to the agent-runner pool. No "in CI we do X but in agents we do Y" footnotes.
Predictable cold-start behaviour. Cache misses are predictable (key-change forces fresh install) rather than surprising (partial-restore left unknown state). Debug time on cache-related test flakes drops because the failure mode is "fresh install ran, here are its logs", not "what state was the cache in".
Simpler debugging. Every cache failure mode reduces to two cases — "key matched, restore happened" or "key missed (or restore failed), fresh install ran". No third case to reason about.
Cross-tenant safety baked in. Principle 7 makes cross-tenant cache sharing a deliberate choice (requiring its own ADR), not an accident of path naming. The multi-tenant agent-runner pool is the case this principle is engineered to protect.

Negative

Every new workflow author has to think about the cache key shape. The principles are not magic — an author writing a new CI workflow must understand "hash content not branch", "include arch", "default to always-run installs". The architectural cost is paid up-front. Mitigated by linking this ADR from the per-repo CI checklist (warp#1370) and from the future gyrum-runner-cache wrapper's docs.
Cache-storage cost grows with the number of distinct lockfile hashes in flight. Each package-lock.json change creates a new cache entry; long-lived branches with diverging lockfiles each carry their own. Mitigated by principle 6's eviction cron and by actions/cache@v4's 10GB-per-repo LRU.
restore-keys discipline rules out a small win on lockfile-bump days. Branches with frequent dep churn pay a full install every key change; some teams will want partial-restore. The bar (principle 4: "provably safe for the workload") is the right one — a workload that genuinely supports partial-restore can opt in with documented justification, not by default.
Host-local caching for the agent-runner pool means the pool ships with a cache-eviction cron from day one. This is not optional; principle 6 is a hard requirement. The pool's bootstrapping playbook owns the cron install.

What we sign up to operate

The principles in this ADR — every new caching workflow is reviewed against them; persona reviewers (per ADR-115) cite this ADR's principle numbers in findings.
The agent-runner pool's host-local cache substrate, including the eviction cron and the tenant-scoped path scheme.
The cross-link from the per-repo SvelteKit checklist (warp#1370) to this ADR's principles, so per-repo CI authors hit the rules at the moment they author cache config.
Periodic review of cache effectiveness (cache-hit rate, cold-start time) — the data feeds open question 2 (whether to migrate CI to host-local).

Alternatives considered

Bake the cache rules into a single shared GitHub Action (gyrum/cache@v1) used by every workflow. Rejected on substrate-coupling. A shared action ties every repo's cache to the action's version — bumping the action requires a PR in every repo. The ADR-as-contract approach is shape-portable: the agent-runner pool, which is not running GitHub Actions at all, applies the same rules without depending on the action.
Document the rules per-repo (per-project CLAUDE.md cache section). Rejected on duplication. Twenty repos × one cache section each = twenty places to update when the rule shape changes. The ADR is the single source of truth; per-repo docs link to it (warp#1370 already does).
Defer the contract until the agent-runner pool ships. Rejected on the "lock the schema before the first writer" principle (sister to ADR-133's reasoning). Without this ADR, warp#1384's patterns become "the warp shape" rather than "the fleet shape", and the next repo to add caching either copies the warp shape without understanding it or re-derives slightly different rules. The cost of locking the contract now is one ADR; the cost of unlocking it after three repos have diverged is migrating three repos.
Allow restore-keys by default; warn rather than forbid. Rejected on failure-mode shape. The failure mode of partial-restore is silent corruption — the build succeeds, the test runs, and a subset of test runs flake on residual state. "Warn rather than forbid" trades a known cost (slightly slower fresh installs on key-change days) for an unknown one (debug-time on cache-related flakes). The known cost is the right trade.
Default the agent-runner pool to actions/cache@v4-equivalent CDN caching. Rejected on latency budget. The agent-runner pool's per-fire latency budget is ≤ 30 seconds for routine fires; a 5-10 second cache-restore round-trip is a third of that. Host-local caching is the only substrate that meets the latency target. CI workflows have no such target (a CI run is a minute-scale activity), which is why CI staying on actions/cache@v4 is acceptable.
Combine this ADR with the runtime-base image ADR (sister swarm). Rejected on scope. The runtime-base image ADR is about "what binaries ship pre-installed on a runner host"; this ADR is about "how a runner caches per-workload state across runs". They overlap on "runner host" but the decision spaces are distinct: a base image is immutable per fleet release; a cache is mutable per workload run. Two ADRs, cross-linked, are the right shape.

Open questions / future work

Wrapper script gyrum-runner-cache --key <hash> --path <dir> --or-run <install-cmd> that abstracts the host-local cache pattern (principle 5 + principle 8's fall-through) and is callable from BOTH CI workflows and agent runners. Out of scope for this ADR; file as a follow-up Warp ticket once the agent-runner pool needs it.
Local-cache-on-runner-host migration for CI workflows (currently using actions/cache@v4). Defer to measurement: if CI runs become bottlenecked on cache-restore time (the 5-10 second CDN round-trip), file a follow-up. Until then, the simplicity of staying on the GitHub-managed substrate outweighs the latency win.
Cross-tenant cache sharing for agent runners, where two tenants with identical dependency sets could share. Per principle 7, this is forbidden by default; if a future workload genuinely needs it (e.g. a fleet-wide compiler toolchain that every tenant pulls), it requires its own ADR with the security analysis spelled out.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

References

warp#1384 — first concrete instance (PR#233, in flight tonight)
warp#1363 — Phase B self-hosted ARM64 runners (the migration that surfaced the question)
warp#1370 — per-repo SvelteKit checklist (will reference this ADR)
warp#1171 — fleet-wide self-hosted migration epic (the umbrella)
ADR-115 — principle-aware PR reviewers (the surface that cites this ADR's principles in findings)
ADR-134 — runtime-base image as unit of isolation for CI + agent-execution runners (sister ADR — the bind-mounted /cache and /git-mirror directories named in ADR-134's per-job lifecycle implement this ADR's contract)