ADR-102: Agent fleet — bootstrap a fresh box into a warp-claiming worker in one script

Status: Accepted Date: 2026-04-25 Related: ADR-074 (CI/CD plane separation — establishes the per-service deploy autonomy that lets a fresh worker pull and run any repo without coordination), ADR-077 (warp as session-independent agent coordination layer — names warp as the AI-worker inbox and defines the claim/heartbeat/complete protocol every worker in this ADR obeys), ADR-097 (PM-in-factory — the operator-facing surface where worker state and queue depth are observed), ADR-098 (epics with structured requirements + claim-returns-context-bundle — the shape of work a worker pulls when it claims), ADR-100 (factory growth boundaries — the principle that operator-facing surfaces consolidate and runtimes federate; agent workers are federated runtimes), ADR-101 (host inventory and roles — the manifest shape that declares an agent-worker host's identity and capabilities; this ADR is the role-composition forward-reference ADR-101 names).

Context

The fleet has had warp as its system-of-record AI-worker inbox since ADR-077 and has had structured PM, epics, and a claim-returns-context-bundle protocol since ADR-097 and ADR-098. What it has not had is a worker — a process running somewhere that does what the protocol invites: poll for ready items, claim one, work it, ship a PR, mark it complete, repeat. Today every Claude Code session re-discovers warp at session start, claims an item or two by hand from whichever machine the operator happens to be at, and stops the moment the operator closes the laptop. The protocol is session-independent; the workers, in practice, are not.

The operator's framing, captured verbatim from the 2026-04-25 session: "i run this on a separate machine install the factory and ask it to chew through the bugs". The shape that fits is a headless agent worker — a process on a fresh Hetzner box (or any box) that runs the same loop a Claude Code session would, but unattended, against a declared set of lanes (repos × kinds × priorities) the operator has authorised it to take. The principle is named in ADR-077 ("warp is session-independent") and in the operator memory feedback_warp_is_design_reference.md; the implementation has been deferred for three sessions running. This ADR names the cut and the contract so a Hetzner box bootstrapped tomorrow morning is a useful fleet member by lunch.

There are three reasons to do this now rather than defer again. First, the Hetzner provision-and-destroy story landed last week (ADR-088, ADR-089) which means a fresh box is a 90-second gyrum-provision-host away — the marginal cost of a worker is now host cost, not setup time. Second, queue depth at warp has crossed the threshold where one operator's by-hand claiming is the bottleneck (40+ ready items at session start on 2026-04-24; the operator spent the first hour of the session triaging rather than building). Third, the fleet's CI gates and review gates are deterministic enough (ADR-074 plane separation, ADR-097 review pipeline, ADR-099 brief-as-PR) that an unattended worker shipping PRs through them is not a quality risk — the gates catch the same things they catch for human-driven PRs. Naming the worker contract now means tomorrow's box ships PRs through the same gates today's session does.

The cut this ADR does not take, equally important: AI-driven lane selection. The worker is told which lanes to claim from via WARP_LANES env at boot; it does not decide for itself which work to take based on queue analysis or operator priorities. Lane selection is an operator decision; the worker is the runtime that executes it. If a future workload demands dynamic lane selection, a follow-up ADR re-opens the question; until then, declarative lanes is the contract.

Decision

A fresh Hetzner box (or any Linux host) is turned into an agent worker by one bootstrap script that installs the gyrum devtools, registers the worker with warp, and starts a long-running daemon that polls the warp queue for items in its declared lanes, claims one, runs it through the existing gyrum-implement → gyrum-review-pr → gyrum-complete-pr workflow, and reports back. The cut from today's flow is additive — every existing operator-driven session keeps working unchanged because workers and operators both speak the same warp protocol; the worker is just an unattended caller.

The five pieces:

Bootstrap script gyrum-bootstrap-worker.sh — one curl-able shell script that runs on a fresh box and produces a registered, running worker. The script: installs git, gh, node, go, Claude Code; clones ~/.gyrum/devtools and links the gyrum-* commands onto PATH; writes ~/.config/gyrum/control-plane.env from operator-supplied secrets (WARP_URL, WARP_TOKEN, GH_ADMIN_TOKEN, ANTHROPIC_API_KEY); calls POST warp.gyrum.ai/api/v1/workers to register the worker and receive a worker id; writes a systemd unit for gyrum-worker-loop that runs as the host's primary user; starts the unit. Idempotent — re-running on a partially-bootstrapped box completes the missing steps and is a no-op when the box is already a worker.
WARP_LANES env contract — a comma-separated list of lane filters the worker is authorised to claim from, in the shape <repo>:<kind>:<priority> where any segment may be * to mean "any". Examples: gyrum-labs/distill-gyrum-ai:bug:* (every bug in distill, any priority), *:docs:low (every low-priority docs item across the fleet), gyrum-labs/ai-frontend:*:high,gyrum-labs/warp:*:high (high-priority work in two specific repos). The worker's poll for ready items applies the lane filter as the WHERE clause on the /api/v1/items?status=ready query; items not matching any lane are invisible to the worker. Lane is a declarative authorisation surface — the operator changes lanes by editing the env file and systemctl restart gyrum-worker-loop, not by editing code.
Worker registration endpoint POST /api/v1/workers — warp gains a new resource. The body carries { hostname, lanes, claude_model, registered_at, agent_key_scope }; the response carries { worker_id, heartbeat_interval_seconds }. The worker keeps its worker_id in ~/.gyrum/worker-id and includes it as a header (X-Warp-Worker-Id) on every claim/heartbeat/complete call so warp can attribute work to a worker for the observability surface. Workers heartbeat against POST /api/v1/workers/{id}/heartbeat every heartbeat_interval_seconds (default 60) regardless of whether they are mid-claim — a worker with no heartbeat for 5× the interval is marked stale and its in-flight items are released back to ready. The auth is the existing warp agent-scoped key from ADR-077; no new key shape is introduced.
gyrum-worker-loop daemon — the long-running process the systemd unit runs. Pseudocode for one iteration: heartbeat → list ready items in my lanes → if zero, sleep 30s and loop → claim the oldest → cd into a fresh /tmp worktree (via gyrum-start-work) → run gyrum-implement against the item's brief → gyrum-review-pr → gyrum-complete-pr → mark warp item complete with the merged PR URL → loop. Errors at any step: capture the failure mode, mark the warp item blocked with the reason, post the failure to the daily session log via the existing dark-factory writer, loop. The loop never exits except on SIGTERM; on SIGTERM it finishes the current item, releases the claim, marks itself stopping, and exits cleanly.
Observability surface in /pm/agents — the factory gains one route under the existing PM tree (per ADR-097's "operator-facing PM affordances live at gyrum.ai/pm/*" rule): a list of every registered worker, its lanes, its current claim (if any), its heartbeat age, its lifetime PR count, its blocked-item count, and a "drain" button that sets the worker's drain flag (the worker finishes its current item, releases the claim, exits, systemd does not restart). The page reads from GET /api/v1/workers and the existing item endpoints; it does not introduce a new auth surface — the same localStorage['warp.token'] from ADR-097 carries the read.

The cost model is named explicitly so operators know what they are signing up for: each worker's marginal cost is host + Claude API. Host is ~$5/month for the smallest Hetzner box that comfortably runs Node + Go + a Claude Code session (CX22). Claude API is variable and is the dominant cost — at the workload's actual shape (one PR every 20-40 minutes, average 200K input + 30K output tokens per PR including review iteration) the per-PR cost is roughly $1.50-$3.00 against Claude Opus pricing as of 2026-04. A worker that ships 30 PRs a day is roughly $50-$90/day in API spend; the host is rounding error. Operators provision workers explicitly and watch the spend in the existing Anthropic console; this ADR does not introduce per-PR cost tracking inside warp (out of scope, see below).

Three rules follow from the cut and are non-negotiable:

A worker never bypasses the gates a human PR goes through. No --admin, no --no-verify, no direct gh pr merge. The worker calls gyrum-complete-pr exactly the same way a human session does; the same review gates, structural checks, and merge strategy apply. A worker whose PR fails review marks the warp item blocked and moves on — it does not retry indefinitely against the same failure.
A worker only claims items in its declared lanes. WARP_LANES is the authorisation boundary; the worker must filter at claim time, not at item-list time. An item that becomes invisible to the worker mid-flight (lane removed, item re-tagged) is released cleanly on the next heartbeat.
The bootstrap script is the only supported way to stand up a worker. Hand-installing devtools and starting a loop produces a worker that drifts from the contract within one devtools release. The script is the contract; the contract is the script.

Consequences

What becomes easier:

Parallel autonomous work across the fleet. The operator names a queue of bug tickets, lane-filters them onto two or three workers, and walks away — the workers chew through the queue overnight, ship PRs through the same gates, and the operator wakes to a session log enumerating what landed. The framing the operator named ("ask it to chew through the bugs") becomes literal.
Cheap horizontal scale. A second worker is one bootstrap-script run away on a second Hetzner box. The marginal coordination cost is zero because warp's claim protocol already serialises ownership (SELECT … FOR UPDATE SKIP LOCKED from ADR-077); two workers in the same lane race-free, three workers in the same lane race-free, N workers in the same lane race-free.
Operator's first-hour-of-session triage time goes to zero. Today's session opens with the operator reading the warp queue and deciding which items to claim by hand; after this ADR ships, the workers have already drained the high-confidence work overnight and the operator's session opens against the items that genuinely need human judgement. The session log writer already exists (per the daily-log block in CLAUDE.md), so the worker's PR-shipping events flow into the same log the operator reads at session start.
Bug-fix throughput stops being session-bound. Every bug-tagged item in a worker's lane lands within its target_date window without an operator being awake; the operator's attention shifts up the stack to architecture, naming, and review of work the worker did.

What becomes harder:

Claude API cost becomes the fleet's dominant operating expense. Today the fleet's operating spend is hosts + GitHub + a small Anthropic line; after this ADR ships, the Anthropic line dominates by an order of magnitude. The cost is bounded by (workers × 24h × PRs-per-hour × cost-per-PR); at three workers and 2 PRs/hour each, roughly $200-$400/day. Mitigation: the operator chooses how many workers to run and which lanes they cover; the cost is visible in the Anthropic console; a per-PR cost tracking inside warp is named as out-of-scope below specifically because the existing console is sufficient for the current scale.
Queue depth tuning matters in a way it didn't before. Today's queue is poll-driven by the operator; the operator sees ten items, claims two, leaves eight for the next session. After this ADR ships, the queue is drained continuously; an operator who adds ten items during the workday sees them gone by morning whether they are ready for autonomous work or not. Mitigation: the lane filter is the right knob — items the operator wants reserved for human review get tagged human-only (a kind no worker is lane-authorised for) at create time; worker lanes default-exclude human-only. The convention is named in this ADR's Decision section so the next time someone adds a worker they know to set the lanes accordingly.
A misbehaving worker can ship 30 broken PRs before anyone notices. The gates catch the broken PR (review fails, structural checks fail, merge does not happen), but the loop logs the failure and moves on; a worker stuck in a failure mode that the gates catch consistently is loud in the logs and silent in the merge stream. Mitigation: the /pm/agents observability surface paints a worker's blocked-item rate; a worker whose blocked rate climbs above (say) 40% over its last ten claims surfaces a red badge so the operator's /pm glance catches it. The drain button is the immediate response; root-cause investigation happens after the drain.
Deployment of devtools updates becomes a fleet operation. Today a devtools change reaches the operator's laptop the next time they git pull ~/.gyrum/devtools; after this ADR ships, the change has to reach every worker too. Mitigation: the bootstrap script links ~/.gyrum/devtools to the upstream git repo and the worker loop runs git pull against it at the top of every iteration; a devtools push is a fleet-wide rollout in the time it takes the longest in-flight item to complete.

What we have signed up to operate:

A gyrum-bootstrap-worker.sh script under ~/.gyrum/devtools/scripts/ that turns a fresh Hetzner box into a worker; idempotent, single-command invocation via curl.
A gyrum-worker-loop binary (or shell daemon) under ~/.gyrum/devtools/bin/ that implements the poll-claim-work-complete loop; lifecycle managed by systemd; survives the box reboot.
A new POST /api/v1/workers registration endpoint and a new workers table in warp (columns id, hostname, lanes, claude_model, registered_at, last_heartbeat_at, current_item_id, lifetime_pr_count, blocked_item_count, drain_requested).
A /pm/agents route in gyrum-labs/ai-frontend that lists workers and exposes the drain button; reads via the same localStorage['warp.token'] flow ADR-097 names.
A documented WARP_LANES grammar in the warp env block of CLAUDE.md so an operator (or a future me) reading a worker's env knows exactly what each lane authorises.

What we revisit:

If queue depth tuning becomes a recurring operator pain (signal: workers running idle for hours while items pile up in lanes none cover), AI-driven lane selection comes back as a follow-up ADR. Until then, declarative lanes is the contract.
If per-PR cost tracking becomes necessary (signal: the operator can't reconcile the Anthropic invoice against the warp PR stream), warp gains a cost_estimate_usd field on the item-complete payload and the /pm/agents page paints lifetime spend per worker. Out of scope today.
If a single Hetzner region becomes a reliability bottleneck (signal: workers all go down together when a region has an incident), cross-region distribution becomes a follow-up. The bootstrap script already works in any region; the deployment topology is an operator choice, not a contract.

Alternatives considered

GitHub Actions workers. Run the loop as a self-hosted runner attached to a private repo, with a workflow that polls warp on a cron and acts. Lost on the same billing failure mode the fleet just hit two weeks ago (Actions minutes burn opaque, the meter ticks even when the runner is idle, the fleet's budget got eaten by a polling loop nobody could see). Also lost on the operational shape — a self-hosted runner is a worker by another name, with worse observability (Actions UI doesn't surface the warp claim) and a tighter coupling to GitHub's availability. Rejected because the failure mode is recent and named.
Kubernetes job runner. Stand up a small k8s cluster on Hetzner; each warp item becomes a Job; the Job's container runs the implement → review → complete flow; node autoscaling absorbs queue depth. Lost on complexity unjustified at the workload's actual scale — k8s is the right answer for thousand-pod workloads, not for three-worker workloads, and the operational surface (etcd, control plane, network policies, pod-identity-to-warp auth) is several orders of magnitude larger than the systemd-on-a-box surface. The whole shape of this ADR is "one script, one box, one daemon, done"; introducing k8s would invalidate the shape. Re-open if the fleet grows to dozens of workers and the systemd-per-box approach has measurable scale pain. Until then, the simpler shape wins.
Manual operator-spawned sessions. Status quo. Lost on the operator's framing — the user explicitly named that they want to "install the factory and ask it to chew through the bugs", which is a description of what the status quo is not. Status quo also loses on the queue-depth bottleneck named in §Context: one operator's hands cannot keep pace with the queue's growth, and the gap widens every session. Rejected because the gap is the problem this ADR exists to close.
One worker that handles every lane via priority queueing. Single daemon, no per-worker lane filter, picks the highest-priority ready item across the fleet. Lost on the authorisation boundary — every worker has the same Claude API key and the same gh token, so any lane the worker can see is a lane it can ship PRs to; "every lane" means "the worker has commit access everywhere", which is exactly what the lane filter exists to constrain. Lost also on horizontal scale — one worker is one Claude session is one in-flight PR at a time; the cost of running N workers covering disjoint lanes is exactly N times the cost of one worker, with N times the throughput. Rejected because lane-as-authorisation is the right cut and N independent workers is the right scale shape.

Out of scope

AI-driven lane selection. The worker takes its lanes from WARP_LANES; it does not decide for itself which lanes to claim based on queue analysis or operator priorities. Re-open as a follow-up ADR if and when declarative lanes hit a measurable pain wall.
Cross-org federation. Workers belong to one warp instance; an item created in another org's warp is invisible. The federation question is a real one for a multi-org future but does not exist in the current scope.
Auto-scaling on queue depth. The number of running workers is an operator decision; the system does not spin up new Hetzner boxes when the queue grows or destroy them when the queue shrinks. The provision/destroy primitives from ADR-088 and ADR-089 make the autoscaler implementable, but the scaling policy itself is deferred.
Per-PR cost tracking inside warp. The Anthropic console is sufficient at the current workload scale; baking cost into the warp item-complete payload is real schema and UI work that is not justified until the operator can no longer reconcile the invoice against the PR stream by eye.

Supersedes: none. Amends: ADR-077 (extends the warp coordination protocol with a worker-registration endpoint and a X-Warp-Worker-Id header on claim/heartbeat/complete calls; the underlying claim/heartbeat/complete contract is unchanged). Composes with: ADR-097 (the /pm/agents observability surface lives under the existing /pm/* route tree), ADR-098 (the worker consumes the claim-returns-context-bundle response shape verbatim — no special-casing for headless callers), ADR-100 (the worker is a federated runtime; the operator-facing observability is one route under the consolidated UI shell). Superseded by: {{leave blank until a later ADR reverses this one}}