ADR-178 — Library / CLI / UI three-layer pattern with infra-agnostic providers

• Status: Proposed (2026-05-09)
• Driver: Operator
• Scope: System-wide. Applies to every operation in the Gyrum fleet that mutates state (provision, decommission, deploy, scale, register, deregister, etc.) — not just runner lifecycle.

Context

Gyrum's operational logic today is scattered. Provisioning a runner involves:

1. Editing a YAML manifest in gyrum-catalog/runners/.
2. Editing an inventory file in dark-factory/ansible/.
3. Firing an ansible playbook (via gyrum-fire-playbook) that shells out to config.sh over SSH.
4. Minting a registration token via the GitHub API.
5. (Sometimes) ad-hoc shell when the playbook hits a stale-vault gate or a missing-binary on a runner.

Decommissioning is worse: there's no playbook for the unsuffixed base agent, so today (2026-05-09) the operator's "please unprovision gyrum-ci-runner-1" became a sequence of:

gh api -X POST /orgs/.../actions/runners/remove-token       # mint
ssh root@host 'systemctl stop ...; svc.sh uninstall;
                config.sh remove --token <reg>; rm -rf /opt/...'
git rm runners/<name>.runner.yaml                            # catalog
sed -i '/<name>/d' ansible/inventory.ini                     # inventory
gh pr create ...                                             # x2 PRs
gh api -X PUT .../branches/main/protection ...               # bypass broken gate

Every operator who learns one of these workflows has to re-learn it on the next host shape, the next provider, the next workflow. The system has no single Go function that says "decommission a runner". The same is true of every other lifecycle verb across the fleet — deploys, hosts, secrets, agents.

This ADR records the operator-set fundamental: the system shall operate like AWS — every action exposed via library + CLI + HTTP/UI, with a single source of truth. Fundamental, not runner-specific.

Pattern already partly established

A pre-flight check (2026-05-09) shows the three-layer chassis is already designed in for runners, just not extended to lifecycle verbs:

This ADR codifies and extends the pattern; it does not invent it. The hole is the lifecycle layer (pkg/runners/lifecycle/ or similar) — public Go API per verb, replacing today's ansible-playbook subprocess + ad-hoc shell. The chassis + provider registry stay; the lifecycle package is the new caller.

Decision

Adopt a three-layer pattern for every fleet-mutating operation, with pluggable per-infrastructure providers:

Layer 1 — Library (Go package)

The single source of truth. A pure Go package per domain, public API per verb. Every fleet operation has a corresponding library function. The library is THE implementation; it is not a thin wrapper over scripts. Where it shells out to existing tools (ansible-playbook, config.sh, terraform, kubectl), it does so as implementation detail — callers see only the Go API.

Public functions emit structured events (so callers — CLI, UI — can stream progress). Functions are idempotent: re-running with the same inputs after a successful run is a no-op. Functions return typed errors with a stable taxonomy (operators can branch on errors.Is(err, lifecycle.ErrAlreadyExists)).

The library NEVER assumes a specific UX (no fmt.Println, no prompts, no os.Exit). It receives a context, returns events and a result. CLI / UI is responsible for rendering.

Layer 2 — CLI (Go binary)

Operator-facing entry point. Wraps the library 1:1: every CLI subcommand calls one library function. Streams library events to stdout/stderr in a human-readable shape.

Existing CLI extends, not new binary: gy-catalog already exists at gyrum-catalog/cmd/gy-catalog/. Lifecycle verbs land as subcommands:

gy-catalog runner run <name>           [--from-manifest <path>] [--count N]
gy-catalog runner terminate <name>     [--keep-manifest]   # full delete
gy-catalog runner stop <name>                              # keep VM, pause agent
gy-catalog runner start <name>                             # resume stopped VM
gy-catalog runner scale <name> <count>
gy-catalog runner agent add <name>
gy-catalog runner agent remove <name>-<letter>
gy-catalog runner list                 [--json]
gy-catalog runner describe <name>

Same pattern when other domains land (gy-catalog deploy ..., gy-catalog secrets ...).

CLI flags map directly to library options. No flag is implemented in the CLI but missing from the library. If the library can't do it, the CLI can't either.

Layer 3 — HTTP server / UI

Web surface (warp /runners, /deploys, /projects, etc.) and its underlying HTTP server both call the library directly. The HTTP endpoint imports the same Go package the CLI uses; emits SSE events back to the browser so the UI shows progress in real-time. No business logic lives in the HTTP handler beyond "call the library function and stream the result"; same rule as the CLI.

UI buttons that today fire a playbook will fire a library call through the server. The /runners page's future "Decommission" button = POST /api/v1/runners/{name}/decommission → handler calls lifecycle.Decommission(ctx, name, opts) → SSE events back.

Pluggable providers (infra-agnostic)

Each library function dispatches to a per-infrastructure implementation via an interface. Today the fleet runs on Hetzner; tomorrow may include AWS, GCP, or bare-metal. Adding a provider = implementing the interface, registering it, and shipping a manifest schema extension. The library + CLI + UI surfaces don't change.

package lifecycle

type Provider interface {
    Provision(ctx context.Context, m Manifest) (*HostHandle, error)
    Decommission(ctx context.Context, h HostHandle) error
    HealthCheck(ctx context.Context, h HostHandle) (*Health, error)
    // …
}

func Register(name string, p Provider)         // hetzner / aws / gcp / …
func For(manifest Manifest) (Provider, error)  // dispatch by manifest.provider

The manifest declares which provider to use:

provider: hetzner   # or: aws, gcp, baremetal, …
provider_config:
  server_type: cax31
  datacenter: fsn1

The library reads the manifest, picks the provider, calls provider.Provision(...). The CLI / UI never knows which provider was selected.

Same pattern applies to every domain, not just runners:

• pkg/deploy/lifecycle/ — deploy a project to whatever infra hosts it
• pkg/secrets/lifecycle/ — rotate a secret across whatever vault holds it
• pkg/dns/lifecycle/ — manage DNS records via whatever registrar
• etc.

Consequences

What gets better

• Single source of truth per operation. The library function is the canon. Bugs fixed once propagate to CLI + UI. New features land via library API.
• Operator UX. gyrum-runner decommission <name> becomes one command. Same operator who's used to AWS CLI feels at home.
• Auditability. Library emits events; CLI + UI stream them to a single audit log. Today's ~/.gyrum/admin-merges.log shape extends to every domain.
• Provider portability. Hetzner-only assumptions in the ansible role become hetzner-provider implementations in the library; AWS provider can land alongside without rewriting the calling code.
• Testability. Library functions test directly via Go tests; CLI + UI need only thin integration tests. Today's bash + YAML
  • ansible mix is hard to unit-test.

What changes

• Existing bash + ansible-playbook + YAML invocations become library calls. The bash / ansible / YAML doesn't disappear immediately — the library shells to them where appropriate — but new functionality lands as Go.
• CLI scripts in ~/.gyrum/devtools/bin/ named gyrum-<domain> consolidate. gyrum-fire-playbook provision-runner becomes gyrum-runner provision. The bash CLIs continue to exist as legacy entry points; new CLIs are Go.
• Web UI buttons call library through HTTP server, not through ansible-playbook subprocess. Server emits SSE events for real-time progress.
• Manifest schema declares provider: <name> with provider_config: shape per provider. Today's runner manifest already has this shape; extending across domains follows the same pattern.

What this is NOT

• Not a Terraform replacement. Terraform is a provider for infra-state-management; the library calls it where useful.
• Not a kubernetes replacement. Same — pluggable provider.
• Not a "rewrite everything in Go". The ansible role for runner lifecycle stays; the library shells to ansible-playbook from Decommission()'s implementation. Replace incrementally.
• Not a one-quarter project. Phased delivery, one verb at a time. Phase 1 (warp#2146): runner provision + decommission.

Documentation language guideline

Operator directive (2026-05-09): library + CLI + UI + docs use a single consistent language so operators don't context-switch between them. Two rules:

1. Lifecycle verbs use AWS terms

AWS clusters its CLI verbs into a small number of shapes by resource type. The matrix:

Shape A — VM-like (compute lifecycle)

EC2 / Lightsail. Runners, hosts, anything that's a "machine you boot then destroy."

Critical distinction: stop ≠ terminate. stop keeps the VM, terminate destroys it. Matches AWS exactly.

Shape B — Resource (CRUD)

S3-objects / IAM / catalog manifests / secrets / DNS records — anything that's a record-shaped thing without a power state.

For Resource shape: delete means delete (no leftover infra behind). Different word from terminate only because the noun is different — there's no "VM kept dormant" case for a DNS record or a secret.

Shape C — Service rollout (deploy-shaped)

ECS / CodeDeploy / Lambda-versions — anything where a "release" is a discrete event with a rollout window.

Domain-shape mapping for the gyrum fleet

A new library MUST pick one of these three shapes. New verbs that don't fit a shape are a smell — either the noun is wrong (split the domain) or AWS has a precedent we're missing (look it up).

2. Nouns: separate the infra layer from the role layer

Examples (correct usage):

• "run a runner VM in fsn1"
• "terminate the runner VM" (= destroy the box; agent goes too)
• "stop the runner" (= keep the VM, pause the agent)
• "scale the runner to 3 agents" (= same VM, more agents on it)
• "the runner VM is paused" (= clear: VM exists, agent down)

Anti-patterns (WRONG):

• ❌ "delete the host" — delete isn't an AWS verb for EC2; host is overloaded
• ❌ "decommission the instance" — neither word matches the guideline
• ❌ "shut down the runner" — ambiguous (does the VM stay?)

This guideline applies to:

• Library function names (Run, Terminate, Stop, …)
• CLI subcommand names (gy-catalog runner run, … terminate, …)
• HTTP endpoint paths (POST /api/v1/runners/{name}/terminate)
• README + ADR + ticket text
• Operator-facing error messages
• Dashboard buttons + labels

Implementation

Driven by warp#2146 (Phase 1: runner library + CLI). Each future domain (deploys, secrets, etc.) lands its own library + CLI under the same pattern; AVR-177 governs the shape, the domain-specific tickets capture the work.

ADRs are the design contract; tickets are the implementation contract. A new domain library cannot ship without:

1. A README in <library-path>/README.md documenting the public API + provider interface (operator-facing contract).
2. A CLI binary at cmd/gyrum-<domain>/ with at least one verb wired through.
3. (When the UI consumes it) a warp HTTP endpoint calling the same library.

Provider plugins land in subdirectories (<library-path>/providers/<name>/); registering a new provider does not require editing the library's public API.

Cross-references

• warp#2146 — first implementation (gyrum-runner library + CLI for runner lifecycle).
• gyrum-catalog#65, #66 + dark-factory#743 (today's runner-1 decommission) — the manual shape this ADR replaces.
• ADR-074 — CI plane separation (today's plane boundary the runner library inherits).

• ADR-130 — areas/products taxonomy (the domain decomposition this pattern lives within).
• AWS SDK + AWS CLI — the reference implementation of this pattern at scale.
• Terraform — reference for the pluggable provider shape.

Status / next

Proposed pending operator review. On acceptance: warp#2146 proceeds as Phase 1; subsequent domains (deploy, secrets, dns, etc.) ride per-domain warp tickets, all governed by this ADR.