← Decisions operate mode →

Decisions

ADR-163: Self-hosted-only fleet CI — runs-on contract + audited override + 1-week soft-warn rollover

Phase 0 of EPIC [warp#1721](https://warp.gyrum.ai/items/1721). Operator (jon, 2026-05-07 ~01:30Z) named the structural shape: "we're using our own runners. Ideally we always use ourown runner only ultrathink". The fleet…

#163

ADR-163: Self-hosted-only fleet CI — runs-on contract + audited override + 1-week soft-warn rollover

Status: Proposed Date: 2026-05-07

Related: warp#1722 (this ADR's ticket, Phase 0 / SH0.1), warp#1721 (parent EPIC: self-hosted-only fleet CI), warp#1406 (rollover precedent: title-format-gate 1-week soft-warn), warp#1486 (sister structural gate: docs-validate auto-regen), warp#1551 (sister structural gate: sitemap-as-source-of-truth), warp#1463 (sister fleet-rollout: docs-validate template-bump-then-cascade), warp#1658 (composes: CPX21 amd64 runner, Phase 2 prereq), ADR-117 (module guidelines tier — applies to code; this applies to CI workflow YAML), ADR-157 (sister gate semantics), ADR-159 (sister bidirectional CI gate).

Context

Phase 0 of EPIC warp#1721. Operator (jon, 2026-05-07 ~01:30Z) named the structural shape: "we're using our own runners. Ideally we always use ourown runner only ultrathink". The fleet bill audit at session start surfaced 7 of 12 active workflows still running on ubuntu-latest — ~800 min/day at peak (devtools ci.yml ~285, dark-factory docs-validate.yml ~180 pre-flip, warp web-e2e.yml + contract.yml ~354, ai-frontend release.yml ~35). The CAX runner pool already provisioned (gyrum-ci-runner-1 CAX21, gyrum-ci-runner-2 CAX31) costs ~€18/mo combined and can absorb the bulk of the load; the marginal CPX21 amd64 runner (warp#1658, Phase 2) closes the architecture gap for the small subset of workflows that genuinely need amd64 native (notably ai-frontend release).

Tonight's swarm flipped 3-4 workflows by hand (warp#1620 ai-frontend release reverted on QEMU tax, warp#1621 dark-factory docs-validate landed, warp#1622 warp web-e2e + contract landed). The remaining gap stays open the moment a new workflow ships without the author thinking about runs-on:. GitHub Actions documentation, copy-pasted templates, and YAML scaffolders default to ubuntu-latest; nothing in the current substrate refuses that default.

The pattern to import is the one already pinned by warp#1486 (docs-validate, the always-runs gate that retired the audited --skip-protection-check per-PR pattern), warp#1551 (sitemap-validate, ADR-157), and warp#1582 (fleet-host-validate, ADR-159): commit canonical truth — here the [self-hosted, linux, <arch>] runs-on convention — and run a bidirectional CI gate that fails on drift, with a 1-week soft-warn rollover and an audited override for genuinely-edge cases. Per parent EPIC: "Phase 0 ADR MUST land first; cascades into substrate phases." This is that ADR.

Decision

The self-hosted-only fleet CI contract has six pinned components: the runs-on rule itself, the audited-override comment syntax, reason-string validation, matrix detection, soft-warn rollover policy, and gate location. Every fleet repo's .github/workflows/**/*.yml MUST conform after the rollover window; bypasses go through the same audited-override pattern (per feedback_admin_override_with_audit.md) that every other fleet rule uses.

1. The runs-on rule

Every job's runs-on: field MUST reference one of two canonical fleet labels:

runs-on: [self-hosted, linux, arm64]   # CAX runner pool (CAX21/CAX31)
runs-on: [self-hosted, linux, amd64]   # CPX runner pool (Phase 2 / warp#1658)

GitHub-hosted labels (ubuntu-latest, ubuntu-22.04, macos-latest, windows-latest, etc.) are refused unless the job carries an audited override (component 2). The two canonical labels are exhaustive for v1; if a future arch lands (e.g. linux/riscv64), an additional label joins this list via successor ADR.

2. Audited-override comment syntax

A workflow author may opt out per-job by annotating the YAML with a single-line comment IMMEDIATELY above the runs-on: line:

jobs:
  some-job:
    # audit-override-self-hosted-only: GitHub-hosted macOS support needed for the cross-platform integration test (no macOS self-hosted runner today; tracked in warp#NNN).
    runs-on: macos-latest

The comment marker is audit-override-self-hosted-only: followed by a free-form prose reason. The IMMEDIATELY-above-the-line constraint is a regex anchor — a blank line or unrelated comment between the override and the runs-on: line invalidates the override and the gate fails. This mirrors ADR-157's and ADR-159's comment-anchor convention so authors who have already learned one already know this one.

3. Reason-string validation

The override reason must:

  • Be ≥ 50 characters (same prose-quality threshold ADR-157 and ADR-159 use; the friction is the point).
  • Not match the placeholder pattern (case-insensitive): TODO, FIXME, xxx, placeholder, stub, temporary, temp, tbd. The list is closed; expansions ride a successor ADR.
  • Reference a warp ticket (warp#NNN substring match) where the structural follow-up is tracked. The ticket is the lifeline that keeps the override from becoming a permanent escape hatch.

The gate fails-with-explanation on each rule violated, naming the file, line, and the rule that failed. The reason is appended to ~/.gyrum/findings/findings.jsonl via the existing audit pattern so overrides accumulate where every other audited override accumulates.

4. Matrix detection

If runs-on: references a matrix variable (runs-on: ${{ matrix.os }}), the gate inspects the matrix definition and fails if any entry is a GitHub-hosted label without a per-entry override. Per-entry overrides use the same comment-above-the-line pattern, scoped to the matrix array element:

strategy:
  matrix:
    os:
      - [self-hosted, linux, arm64]
      # audit-override-self-hosted-only: macOS slot needed for cross-platform smoke (warp#NNN).
      - macos-latest

The contract is uniform across runs-on:-direct and matrix-indirect cases — same comment marker, same anchor rule, same 50-char + warp-ticket validation. Implementations that do not yet handle matrices fall back to the single-runs-on: shape and surface the matrix case as an unsupported-yet warning.

5. Soft-warn rollover (sister to warp#1406)

The first time a fleet repo adopts the gate (detected by absence of marker file .gyrum/self-hosted-only-rollover-began), the gate runs in soft-warn mode for 1 calendar week: it computes drift and logs each violation as a workflow annotation but exits 0 so the PR is not blocked by drift the operator did not yet know to fix. The marker file is written by the workflow on first run with an ISO-8601 timestamp; the workflow reads the timestamp on subsequent runs and switches to hard-fail when more than 7 days have passed. Same shape as ADR-157, ADR-159, and warp#1406's title-format-gate rollover; the convention is shared so operators only learn it once.

6. Gate location and fleet rollout

The gate lives at .github/workflows/self-hosted-only-validate.yml per repo, devtools-templated and fleet-rolled-out via gyrum-setup — the same template-bump-then-cascade pattern warp#1463 uses for docs-validate. The workflow runs on every PR that touches .github/workflows/**, runs on [self-hosted, linux, arm64] itself (dogfood), and shells to a bash validator under devtools/lib/self-hosted-only-validate.sh. Phase 1 / SH1.1 ships the validator and the template; this ADR pins only the contract the validator reads.

Consequences

What becomes easier. New workflow YAML defaults to the fleet's own runner pool by structural rule, not by author memory. The bill-audit failure mode (drift creeping back the moment someone copies a ubuntu-latest snippet from a GitHub blog post) becomes a CI failure with a clear actionable message at PR time. Genuine edge cases (macOS test, Windows binary builds, transient amd64 gap during Phase 2) ride the override lane with a reason ≥ 50 chars and a warp ticket — costly enough to discourage routine use, cheap enough to not block legitimate work. The rollover window means day-1 adoption never breaks ordinary work; the one-week soft-warn period accumulates real-state context and the eventual hard-fail diff is small.

What becomes harder. Every fleet repo touching .github/workflows/** runs a new gate; PR latency grows by the gate's runtime (target <30s, mirroring docs-validate). The 50-char + warp-ticket floor on override reasons is friction by design. The marker-file convention (.gyrum/self-hosted-only-rollover-began) is one more file the adopting repo commits, but the convention is shared with the sister gates. Phase 1's bash validator must handle YAML's syntactic flexibility (quoted/unquoted scalars, flow vs block sequences, anchors) without resorting to a full YAML parser inside a shell script — the gate is structural enough that a 90% solution that catches the common cases and flags the residue as "could not parse, override required" is acceptable for v1; coverage gaps ride a successor ticket.

What we sign up to operate. The workflow lives in each adopting repo's .github/workflows/; rolling it out fleet-wide is the same shape as warp#1463. The 1-week rollover is fleet-uniform; bumping the window requires a successor ADR. The override audit log lives at ~/.gyrum/findings/findings.jsonl alongside every other audited gate; overrides accumulate there and are reviewed during fleet-wide audit sweeps. The contract this ADR pins applies to CI workflow YAML; an analogous principle applies to code via ADR-117 (module guidelines tier).

Alternatives considered

  • Hard-fail from day 1 (no rollover). Lost: the first PR after adoption blocks on drift the operator did not yet know to commit. Soft-warn rollover lets the gate ship without breaking ordinary work; the precedent is warp#1406, ADR-157, ADR-159.

  • Ban GitHub-hosted labels entirely; no override lane. Lost: real edge cases exist (macOS-only tests, Windows binary builds, transient amd64 capacity gaps). An override lane with a 50-char audited reason is the right friction — costly enough to discourage routine use, cheap enough to not block legitimate work. The friction is the point.

  • Pre-commit hook instead of CI gate. Lost: pre-commit hooks are bypassable (git commit --no-verify), and the bidirectional check (matrix detection, marker-file rollover state) is a CI shape. Author-time hooks are a useful pre-flight but not a substitute. CI is where this gate runs.

  • Fold this into the existing docs-validate.yml. Lost: docs-validate runs on docs/ADR/sitemap PRs; this gate runs on workflow YAML PRs. Different paths, different validators, different per-repo template bumps. Folding produces a multi-purpose workflow that grows by accretion. Sister-gate-but-separate is the established pattern.

  • No matrix detection in v1; ship single-runs-on: only. Lost: matrices are a common shape (cross-platform test suites). Skipping them means agents author matrix workflows that bypass the gate by accident. The 90% solution (catch common matrix shapes, flag complex ones for override) is acceptable.

  • Allow ubuntu-latest with a one-line audited reason but no warp ticket. Lost: tickets are the lifeline that keeps overrides from becoming permanent escape hatches. The audit log accumulates overrides; without a ticket reference, follow-up work is invisible. Mandating the ticket reference is the same shape warp#1486 uses for cross-repo PAT rotation runbooks.

Out of scope

  • Phase 1 substrate (the self-hosted-only-validate.yml template + devtools/lib/self-hosted-only-validate.sh validator + 4 test cases) — separate ticket SH1.1.
  • Phase 2 amd64 runner provisioning (CPX21 + GH Actions registration + label [self-hosted, linux, amd64]) — already filed as warp#1658, composes here.
  • Phase 3 fleet sweep (devtools ci.yml + pr-gate.yml flip; ai-frontend release.yml re-flip post-Phase-2) — separate cleanup tickets SH3.1 / SH3.2.
  • Phase 4 retire ubuntu-latest entirely (flip soft-warn → hard-fail across the fleet) — deferred SH4.1.
  • Cross-fleet visibility dashboard (fleet-wide override audit-log view) — deferred.

Supersedes: the implicit "any runs-on: ships" assumption baked into gyrum-setup's workflow scaffolds and gyrum-create-repo's starter templates pre-2026-05-07. Superseded by: {{leave blank until a later ADR reverses this one}}