← Decisions operate mode →

Decisions

ADR-159: Fleet host-state-as-truth — hosts.yaml schema + bidirectional CI gate

Sub-ticket H0.1 of EPIC [warp#1582](https://warp.gyrum.ai/items/1582) (fleet host-state-as-source-of-truth). Sister to [ADR-153](./153-host-state-declarative-schema.md), which fills the role-manifest layer beneath each…

#159

ADR-159: Fleet host-state-as-truth — hosts.yaml schema + bidirectional CI gate

Status: Proposed Date: 2026-05-06

Related: warp#1657 (this ADR's ticket), warp#1582 (parent EPIC: fleet host-state-as-source-of-truth), ADR-153 (sister Phase-0 ADR: role-manifest layer beneath each roles[] enum value), ADR-101 (host inventory + roles axis this ADR pins as the gate's input), ADR-157 (sister structural pattern: bidirectional gate + 1-week soft-warn rollover for sitemap), warp#1486 (sister structural pattern: docs-validate auto-regen gate), warp#1551 (sister: sitemap-as-source-of-truth EPIC), warp#1646 (sister: fleet SaaS chrome EPIC), warp#1406 (rollover precedent: title-format-gate 1-week soft-warn).

Context

Sub-ticket H0.1 of EPIC warp#1582 (fleet host-state-as-source-of-truth). Sister to ADR-153, which fills the role-manifest layer beneath each roles[] enum value; this ADR locks the host-axis schema and the CI gate that keeps the declared inventory and the observed fleet bidirectionally honest.

Operator (jon, 2026-05-06) hit two structural failures the same evening that name the urgency:

First, the warp-04 lie. warp.gyrum.ai DNS resolves to 91.98.233.209 (warp-04) and the service responds healthily, but the inventory dashboard at factory.gyrum.ai/infra reported warp-04: 0 containers, 0 uptime. Root cause: dark-factory/scripts/hetzner-status.cjs:194-219 (showContainers()) SSHes only to a single host named in env.VPS_HOST rather than walking every fleet host; whichever host the env var pointed at was the only host whose containers got counted, every other host appeared empty. Fixing the script is a band-aid because the inventory is computed at runtime by querying provider APIs and SSH-ing to known hosts; any silent SSH failure or any host in a different Hetzner project produces the same lie.

Second, the invisible 9th host. 116.203.167.0 (static.0.167.203.116.clients.your-server.de) hosts hiphip.gyrum.ai, factory.gyrum.ai, distill.gyrum.ai, the apex gyrum.ai, and api.gyrum.ai — five DNS records pointing at a host that does not appear in the visible inventory. The dashboard reports 8 hosts; there are at least 9. An operator decision was nearly made on this bad data: an earlier brief recommended decommissioning warp-01 and warp-04 as "empty" and migrating gyrum-prod-1 to CAX without knowing what was deployed where.

Both are instances of one structural class: fleet host-state is computed from runtime probes against an undeclared population, drift accumulates silently, and surfaces only when a dashboard catches it (or fails to). The pattern to import is from warp#1486 (docs-validate) and warp#1551 (sitemap-validate, ADR-157): commit canonical truth in git, compute drift against the declaration, gate. The same shape applies one layer down — the host axis instead of routes or doc fragments — and retires the same failure mode under a different name.

Decision

A canonical hosts.yaml (committed at the dark-factory repo root) is the declared host inventory; the fleet-host-validate CI workflow runs three bidirectional checks (declared-but-not-reporting, reporting-but-not-declared, DNS-mismatch) on every PR touching infrastructure/** or hosts.yaml; each adopting repo gets a 1-week soft-warn rollover before the gate goes hard; an audited --skip-host-validate "<reason ≥ 50 chars>" flag exists for genuinely-unfixable cases; tagged-critical playbooks (deploy-project, fleet-health) carry a watchdog that auto-files a Warp ticket on N consecutive failures. The contract has four parts.

1. hosts.yaml v1 schema

The committed hosts.yaml (at the dark-factory repo root) declares every fleet host as an entry in a flat list. Required fields are name, public_ip, hetzner_type, declared_arch, role, and probe_required. Optional fields are declared_dns (an array of every gyrum.ai DNS record that points at this host, used by check 3 below) and notes (free-form, operator-facing). Worked example:

hosts:
  - name: gyrum-prod-1
    public_ip: 91.98.166.64
    hetzner_type: cpx32
    declared_arch: amd64
    role: prod-app
    probe_required: yes
    declared_dns:
      - warp.gyrum.ai
    notes: "27 containers as of 2026-05-06"
  - name: dark-factory-db
    public_ip: 178.104.45.69
    hetzner_type: cax11
    declared_arch: arm64
    role: prod-db
    probe_required: yes
    declared_dns: []

The role field joins into ADR-101's closed enum; the roles[] array on each host record (per ADR-101) further joins into ADR-153's role-manifest layer at dark-factory/host-roles/<role>.yaml. This ADR locks only the host-axis schema; the role-manifest layer beneath each role-name is the sister ADR's scope. probe_required: yes is the field check 1 reads.

2. Bidirectional CI gate (fleet-host-validate.yml)

Sister to warp#1486's docs-validate and ADR-157's sitemap-validate. Runs on every PR that touches infrastructure/** or hosts.yaml. Three checks, all fail-on-drift:

Check 1 — declared-but-not-reporting. For each entry in hosts.yaml with probe_required: yes, query the prober API for the host's most recent heartbeat; fail if no heartbeat within 5 minutes. This catches the warp-04 failure mode (declared host present in the inventory but actually invisible to the dashboard's runtime probe).

Check 2 — reporting-but-not-declared. Query the Hetzner Cloud API for every server in the project; SSH to every reachable host on the known fleet network; fail if any responding host is not declared in hosts.yaml. This catches the 116.203.167.0 failure mode (host hosting five gyrum.ai services but invisible to the inventory). The check uses the prober-collector's published state as its source of truth (cached server-side per the warp#1582 EPIC's "don't gate on prod-host network reachability from CI" pitfall); the workflow fetches the cached state, not raw network probes.

Check 3 — DNS-mismatch. Resolve every gyrum.ai DNS record (via the existing fleet DNS audit script); fail if any record's IP is not the public_ip of a declared host or the IP of a declared CDN. This catches the 116.203.167.0 failure mode from the DNS direction (five records pointing at an undeclared IP).

SAMPLE GATE FAILURE — check 2 (reporting-but-not-declared)
✗ fleet-host-validate: undeclared host responding to prober

  Hetzner API + SSH-reachability returned 9 hosts.
  hosts.yaml declares 8 hosts.
  Undeclared host: 116.203.167.0 (static.0.167.203.116.clients.your-server.de)
    DNS records pointing at this IP:
      - hiphip.gyrum.ai
      - factory.gyrum.ai
      - distill.gyrum.ai
      - gyrum.ai
      - api.gyrum.ai

  Fix: add an entry for 116.203.167.0 to hosts.yaml with the
       correct role + probe_required, OR decommission the host
       and clean up its DNS records, then re-run the gate.

The fail-on-drift contract is what makes the gate structural rather than documentary. A PR adding a host without a hosts.yaml entry fails check 2 on first run; a PR removing a host without cleaning up its DNS fails check 3; a PR shipping a host the prober cannot reach fails check 1. Each failure mode the warp#1582 EPIC names retires under a structural error, not a dashboard banner.

3. Soft-warn rollover (sister to warp#1406)

The first time dark-factory adopts the workflow (detected by absence of a marker file at .gyrum/host-validate-rollover-began), the gate runs in soft-warn mode for 1 calendar week: it computes drift and logs it as a workflow annotation, but exits 0 so the PR is not blocked by drift the operator did not yet know to commit. The marker file is written by the workflow on first run with an ISO-8601 timestamp; the workflow reads the timestamp on subsequent runs and switches to hard-fail when more than 7 days have passed. Same shape as ADR-157's rollover and warp#1406's title-format-gate rollover; the convention is shared so operators who already know one already know this one.

4. Critical-playbook watchdog

Tagged-critical playbooks (deploy-project, fleet-health, etc., per their playbook frontmatter tags: [critical]) carry a failure_threshold: N (default 5). An observer-claude per ADR-110 watches the last-N-runs of each tagged-critical playbook; on N-consecutive-failures it POSTs to the Warp item-create endpoint with tags: [observer:claude, urgent, pipeline-degraded, parent_playbook:<id>], files the ticket against gyrum-labs/dark-factory, and leaves the ticket for the on-call agent to claim. Auto-paging integration (PagerDuty webhook or similar) is out of scope for this ADR — separate ticket. Auto-rollback correlation (scanning recent PR merges for one that touches code related to the failing playbook) is Phase 3.2 of the parent EPIC, also out of scope here.

5. Override semantics — --skip-host-validate "<reason ≥ 50 chars>"

For genuinely-unfixable cases (e.g. cloud provider API temporarily 503-ing during the PR window, or a host transitioning between Hetzner projects mid-PR), gyrum-review-pr --skip-host-validate "<reason>" provides an audited bypass. The reason must be ≥ 50 characters (same prose-quality threshold as ADR-157's --skip-sitemap-validate) and is appended to ~/.gyrum/findings/findings.jsonl via the existing audit pattern. The override clears THIS PR's gate; it does NOT clear the underlying drift — the operator commits to a follow-up fix (either reconcile hosts.yaml, fix the prober, or unblock the API failure). The override is logged with the host names and check that drifted so the follow-up has the same context the override needed.

Consequences

What becomes easier. The two failure modes the EPIC retires (warp-04 underreport, 116.203.167.0 invisible) become structural errors a PR cannot accidentally ship past. Operator decisions about decommissioning or migrating hosts run against hosts.yaml rather than a runtime-computed inventory the dashboard might be lying about. New hosts enter the fleet through a PR that includes both the hosts.yaml entry and the prober install (per the EPIC's Phase 1 H1.3 ansible-role); the gate refuses to merge a half-shipped pair. Soft-warn rollover means day-1 adoption never blocks ordinary work; the gate accumulates real-state context during the week and the eventual hard-fail diff is small and auditable.

What becomes harder. Every fleet repo touching infrastructure/** now reads hosts.yaml from dark-factory (or its mirror) and hits a CI gate that depends on the prober-collector's cached state being available. The 50-char floor on override reasons is friction by design; the friction is the point. The marker-file convention (.gyrum/host-validate-rollover-began) is one more file the adopting repo commits; the convention is shared with ADR-157 and warp#1406 so operators only have to learn it once.

What we sign up to operate. The workflow lives in dark-factory's .github/workflows/ directory under the name fleet-host-validate.yml; rolling it out fleet-wide is the same shape as warp#1463's docs-validate fleet-rollout. The 1-week rollover is fleet-uniform; bumping it requires a follow-up ADR. The override audit log lives at ~/.gyrum/findings/findings.jsonl alongside every other audited gate; overrides accumulate there and are reviewed during fleet-wide audit sweeps. The watchdog observer is a routine-watching agent (Haiku-pinned per feedback_token_thrift_routine_agents_haiku.md) costing pennies per month per playbook.

Alternatives considered

  • Compute the inventory at runtime; no committed truth-file. What today's hetzner-status.cjs does. Lost: every failure mode the EPIC retires (silent SSH failure, host in different Hetzner project, undeclared DNS) returns under the runtime-drift name. Runtime inventory makes the dashboard the only check; the dashboard cannot fail a PR.

  • One-way gate (declared-but-not-reporting only, no reporting-but-not-declared check). Cheaper to ship. Lost: the 116.203.167.0 failure mode survives — a host invisible to the inventory but reporting healthily to its services would never trip a one-way gate. Bidirectional is the property the warp#1582 EPIC names; this ADR honours it.

  • Hard-fail from day 1 (no rollover). Lost: the first PR after adoption blocks on drift the operator did not yet know to commit. Soft-warn rollover lets the gate ship without breaking ordinary work; the precedent is warp#1406 and ADR-157.

  • Fold this schema into ADR-153 (role-manifest layer). Lost: the role-manifest layer answers "what should each role contain" and joins downward into per-resource records; this ADR answers "which hosts exist + what gates keep that list honest" and joins outward into DNS + provider APIs. Two ADRs at two layers preserve the 60–120-line scope per docs/shared/doc-standards.md and let each evolve independently.

  • Author-time hook (pre-commit) instead of CI gate. Lost: pre-commit hooks are bypassable (git commit --no-verify), and the bidirectional check requires querying the Hetzner API and the prober-collector — neither belongs in a pre-commit shape. CI is where this gate runs.

  • No override flag — every drift must be fixed at source. Lost: a transiently-503ing Hetzner API would block every PR until it recovered. The audited override with a 50-char reason floor keeps the escape hatch costly enough to discourage routine use without making it impossible.

Supersedes: none — extends EPIC warp#1582; sister to ADR-153 (Phase-0 role-manifest layer beneath each roles[] value). Superseded by: {{leave blank until a later ADR reverses this one}}