ADR-160: gyrum-fleet — Go library + CLI for fleet ops (supersedes bash-script shape)

Status: Proposed Date: 2026-05-06 Related: warp#1189 (parent EPIC, direct-ssh forbidden — substrate decision), warp#1582 (sister Phase 0: host-state-as-truth — gyrum-fleet reads hosts.yaml), warp#1646 (sister: fleet SaaS chrome — frontend half of dogfood pattern), warp#1054 (pattern model: recent_deploys.go — interface + http impl + 60s cache), warp#1451 (auth: gyrum-go shared library), warp#1178/1179 (existing bash primitives to migrate), warp#1254 (audit precedent: warp-add stdout/stderr/exit contract)

Context

Phase 0 of EPIC warp#1189 ("direct ssh/sudo access forbidden; all access via scripts"). The EPIC is correct in structural intent — fleet operations should be named, audited, idempotent, agent-callable — but was filed in the 2026-05-04 era when bash was the assumed implementation. Operator (jon, 2026-05-06 22:11Z) named the structural correction: "it'll be a library and go and tdd and full test, bsah isn't qa'ed". This ADR locks the shape before Phase 1 substrate ships; Phase 1 ships the Go library skeleton plus the first command, Phase 2+ ports each existing bash script incrementally.

Tonight's scripts/hetzner-status.cjs bug demonstrates the cost of the bash-script shape: showContainers() (lines 194-219) runs ssh ${env.VPS_HOST} docker ps against a single env-var-named host, then prints the result as if it represented the whole fleet. The inventory dashboard built on it told us warp-04 was empty when DNS shows warp.gyrum.ai live there. One line of test (the function takes a host list, not an env var) would have caught it; the bash shape ships without tests, so the bug hid until production lied. The same failure mode is latent across every fleet bash script. The cure is the same shape applied across tonight's other Phase 0 ADRs: canonical Go implementation, structurally enforced, dogfood-able. For fleet ops, that means a Go library and CLI with TDD and ≥80% coverage gate.

Decision

gyrum-fleet is a single Go binary plus library, shipped from a new repo gyrum-labs/gyrum-fleet, with mockable provider interfaces (GitHub, Hetzner, Ansible), TDD-required tests with ≥80% coverage gate, audit-by-construction (every command writes server_access_log row + findings.jsonl entry), authentication via the gyrum-go shared library (warp#1451) under a new token scope fleet-ops, and an incremental migration plan that preserves existing bash scripts until each Go equivalent is verified in production. The shape has six parts: repo location, command surface, provider mediator pattern, audit shape, auth, and migration.

1. Repo location — new gyrum-labs/gyrum-fleet

A new repo, not a sub-package of an existing Go repo. Clean dependency boundary: nothing in warp/ or dark-factory/ should import fleet-ops code, and nothing in gyrum-fleet should depend on warp or dark-factory beyond their published HTTP APIs. The repo carries its own CI (ARM64 self-hosted runners, matching the fleet), its own release cadence, and its own go.mod root. Same shape as gyrum-labs/devtools and gyrum-labs/warp already follow.

2. Command surface skeleton

Five command groups plus break-glass: gyrum-fleet runs cancel|list|rerun (GitHub Actions ops), hosts provision|decommission|inventory (Hetzner ops; inventory cross-checks hosts.yaml from warp#1582), runners register|deregister|cleanup (self-hosted runner ops; absorbs warp#1178), bench provision|destroy (ad-hoc bench server lifecycle), and emergency-ssh (the EPIC-named break-glass with --reason ≥ 50 chars and full audit log). The verb-noun grouping mirrors gh's surface and keeps tab-completion legible at the fleet's scale.

3. Provider mediator pattern (sister to warp#1054 recent_deploys.go)

Each external provider sits behind an interface plus an http/exec implementation plus a mock. Tests use the mock; production uses the http impl; the 60s cache layer lives in the http impl per recent_deploys.go's precedent. This is what makes the library testable — unit tests never hit GitHub or Hetzner, integration tests wrap the same interfaces with a recording fixture, and the regression-prevention gate gyrum-review-pr enforces is meaningful because it covers the logic not the network.

type GitHubClient interface {
    CancelRun(ctx context.Context, repo string, runID int64) error
    ListRuns(ctx context.Context, repo string, status string) ([]Run, error)
    RegisterRunner(ctx context.Context, host, label string) error
}

type HetznerClient interface {
    Provision(ctx context.Context, name, hetznerType, sshKey string) (Server, error)
    Decommission(ctx context.Context, name string) error
    List(ctx context.Context) ([]Server, error)
}

type AnsibleRunner interface {
    Run(ctx context.Context, playbook string, hosts []string, vars map[string]string) (Result, error)
}

4. Audit by construction

Every command writes three audit surfaces in lock-step. (a) A server_access_log row (schema per warp#1189's EPIC body) BEFORE executing — intent recorded even if the op fails or the process is killed. (b) A second row AFTER, with exit code, duration, and output tail. (c) A mirrored entry to ~/.gyrum/findings/findings.jsonl for cross-session visibility. The stdout/stderr/exit contract follows warp#1254's warp-add precedent: stdout is one parseable line on success, stderr carries diagnostics, exit codes are uniform across the binary. This is the same defense-in-depth shape as warp#1189's EPIC: the audit log is the structural record, findings.jsonl is the agent-readable mirror, and stdout/stderr are the human-readable mirror.

5. Auth — gyrum-go/auth + new fleet-ops scope

Authentication routes through gyrum-go/auth (warp#1451) and reuses warp's token-mint endpoint (warp#1443/1453). A new token scope fleet-ops is the single bearer required to invoke any gyrum-fleet command; the binary reads it from GYRUM_FLEET_TOKEN. Provider-side credentials (the GitHub PAT for gh ops, the Hetzner token for hcloud ops) live on the runner host, never in agent context — gyrum-fleet reaches out from the runner and the agent never sees the raw provider tokens. Same shape as recent_deploys.go already follows for GH_FLEET_TOKEN.

6. Migration — incremental, never Big Bang

Existing bash scripts (gyrum-runner-register, gyrum-runner-cleanup-workspace, gyrum-bench-provision, gyrum-bench-destroy, the rest of warp#1178/1179's primitives) are PRESERVED. Each gets a Go replacement shipped one command at a time; the bash version is retired only after the Go equivalent has been verified in production for at least one fleet operation. The migration table is a child ticket of this Phase 0 ADR; no individual port is in scope here. Same shape as warp#1620's gradual flip strategy (per workflow), not a single cut-over.

Consequences

What becomes easier. Every fleet operation has one canonical entry point with structural audit, mockable tests, and an agent-callable surface. The hetzner-status.cjs-class bug becomes structurally hard to ship — the test that takes a host list lives next to the function, the mock returns multiple hosts, and the unit refuses to merge below 80% coverage. Operators and agents call the same binary; the audit log answers "what changed on runner-2 last week?" with a SQL query rather than per-shell-history archaeology.

What becomes harder. A new repo to maintain (CI, release cadence, dependency hygiene). Every existing bash script eventually has a Go port to write, review, and verify in production before retiring its bash predecessor — that is multi-week work spread across Phase 2+. Provider-SDK choice (Hetzner SDK vs raw HTTP, GitHub go-github vs raw HTTP) is deferred to Phase 1 but constrained by the mediator pattern: the interface is the contract, the SDK is an implementation detail.

What we sign up to operate. The new repo follows the same gyrum-setup-propagated CI shape as devtools and warp; release tagging, branch protection, and the persona review gate apply identically. The fleet-ops token scope joins the existing agent and admin scopes in warp's token-mint UI; rotation cadence and audit accountability follow warp#1451's pattern. Phase 1 lands the skeleton plus runs cancel (the simplest read-write command, smallest blast radius); Phase 2+ ports per the migration table above.

Alternatives considered

• Keep bash; add bats test suite. Lost: bash test frameworks exist but the fleet has zero current investment in them, the assertion vocabulary is poorer than Go's, and the mockability of ssh/gh/hcloud calls is a per-script invention rather than a language-level interface. The first integration mistake (a script that takes one host but is called with many) is exactly the failure mode the EPIC retires; Go's interface contract makes it a compile-time question, not a runtime surprise.

• Sub-package of gyrum-labs/devtools instead of a new repo. Lost: devtools is the agent-toolchain repo (CLI wrappers, hooks, templates); fleet-ops is provider integration with a different dependency graph (Hetzner SDK, GitHub SDK, Ansible exec). Co-locating would couple devtools' release cadence to fleet-ops' provider-SDK churn. Clean repo boundary keeps both moving at their own pace.

• Rust or TypeScript instead of Go. Lost: Go is the fleet's existing back-end stack (warp, deploy-manifest, recent_deploys.go), the team has the most muscle memory in it, and the provider SDKs (hcloud-go, go-github) are mature and idiomatic. TypeScript would force a Node runtime on every runner host (already true for some surfaces but not a target); Rust would re-fight battles already won in Go.

• Single-binary with all commands as subcommands of gyrum. Lost: the gyrum-* binary surface is intentionally fragmented (gyrum-start-work, gyrum-review-pr, gyrum-complete-pr) so each binary has clear ownership and a small blast radius for bugs. A monolithic gyrum binary would couple fleet-ops releases to devtools releases and grow the test surface unboundedly.

• Defer the audit-log to Phase 2+; ship Phase 1 without it. Lost: audit-by-construction is the EPIC's structural intent. A Phase 1 binary that doesn't log is a binary that has to be retrofitted, and retrofits routinely miss code paths. Audit lands in Phase 1 alongside the first command, so every subsequent command inherits the pattern by copy-and-modify rather than re-deriving it.

Out-of-scope

• Implementing the library itself — Phase 1 child ticket.
• Migrating individual bash scripts — Phase 2+ child tickets per the migration table above.
• Hetzner SDK choice (hcloud-go vs raw HTTP) — decided in Phase 1 against concrete needs; the mediator pattern abstracts over either choice.
• A web UI for fleet ops — that is warp#1646's territory (fleet SaaS chrome, frontend half of the dogfood pattern); gyrum-fleet is CLI plus library only.
• Direct-ssh detection cron and break-glass session-recording UI — Phase 4+ children of warp#1189.

Supersedes: the bash-script implementation shape implied in warp#1189's original EPIC body; the structural intent (named, audited, idempotent, agent-callable) is preserved. Superseded by: {{leave blank until a later ADR reverses this one}}