ADR-153: Declarative host-state schema — role-manifests, observed-state, drift-item, idempotency contract

Status: Proposed Date: 2026-05-06 Related: ADR-101 (host inventory and roles — defines gyrum-catalog/hosts/<name>.yaml with a roles[] enum; this ADR layers per-role manifests beneath each enum entry), ADR-102 (forward-ref for the role-playbook composition surface this ADR formalises), ADR-148 (idempotent bootstrap — same shape as the per-role idempotency contract this ADR locks), ADR-147 (bootstrap stage markers — referenced by the bootstrap-path section). Sister phase-0 ADRs of EPIC warp#1582: H0.1 hosts.yaml schema (extended here with the role-manifest layer), H0.2 critical-playbook watchdog (different surface, same drift-aware shape).

Context

Operator (jon, 2026-05-06) named the architectural shape behind warp#1581: four hosts (gyrum-ci-runner, warp-01, dark-factory-db, agent-worker-01) appeared on the operator board with missing-probe markers because nothing in the fleet declared what should be on each host. The reconciler proposed at H1.4 cannot be built before the schema it reconciles against. Sister-question on the same day: "are check and install the same with --dry-run that returns stats?" — structurally yes, and that property only holds if check + apply share a single representation of declared, observed, and drift state.

Three structural facts make the absence of a per-role manifest schema a near-term blocker:

1. ADR-101 ships the host axis but stops at the role-name boundary. hosts/warp-01.yaml declares roles: [edge, web, api, db]; nothing declares what edge means on disk. Every role today is implicitly defined by the ansible playbook that installs it (role-edge.yml, role-ci-runner.yml, etc.). That implicit shape has three failure modes the missing-probe incident surfaced: (a) the playbook can install a role and there is no machine-readable record of what the role contains afterwards; (b) two playbooks claiming the same role can drift over time and no diff catches it; (c) the operator UI cannot answer "is gyrum-prober actually installed on warp-01" without SSH-ing and running a checklist. The role-name-only catalog is the tip; the manifest layer beneath it is missing.

2. Check and apply diverge structurally without a shared drift representation. Today gyrum-validate-playbook answers "is the manifest valid"; nothing answers "does the host match the manifest". An operator running ansible-playbook --check gets a wall of "would change" lines but no structured diff to feed back into the catalog or into a Warp ticket. A subsequent apply gets a wall of "changed" lines and the relationship between the two is implicit. The reconciler at H1.4 needs a single drift-item shape that check produces, apply consumes, and the /infra/hosts UI renders. Without that shape, "are check and install the same with --dry-run" is true only by hand-waving.

3. The four missing-probe hosts are the canonical first failure. gyrum-prober is a tiny daemon (binary + systemd unit + config) that every fleet host should carry so the watchdog (H0.2 / ADR-149 cardinality-guard) has a heartbeat. It is the simplest possible role — three resources — and the four hosts not having it is the test case for the schema this ADR locks. If the schema cannot describe the prober cleanly, it cannot describe edge (Caddy + LE + ufw rules + vhosts) or db (postgres + pgbouncer + backup hooks).

The pattern to import is from ADR-101 and ADR-148: declarative manifest, schema-validated, version-controlled, idempotent applier, observed-state queryable, drift mechanically computable. ADR-101 already established the axis pattern (gyrum-catalog/<axis>/<name>.yaml + pkg/<axis>/ + cmd/validate-<axis>-manifest/); this ADR fills the per-role layer beneath each roles[] entry, on the same template.

Decision

Every role named in the ADR-101 enum carries a manifest at dark-factory/host-roles/<role>.yaml declaring its desired state as a list of typed resources. A reconciler (H1.4, future) computes drift as a list of drift-item records by comparing each declared resource against its observed-state query result, and applies the drift list idempotently. Check (--dry-run) and apply share the drift-item representation; the only difference is whether action_to_apply is executed or merely returned. A new host enters the system through a bootstrap path that pulls and runs the reconciler against its assigned roles before any other workload lands.

The shape of the cut:

1. hosts.yaml — role-list field is locked

ADR-101 already defines roles: [<enum>] per host record. This ADR pins the field as the join key into the role-manifest layer: every value in roles[] MUST correspond to a file at dark-factory/host-roles/<role>.yaml. validate-host-manifest extends to fail when a host names a role that has no manifest file. The closed-enum discipline of ADR-101 plus this referential integrity check makes the host → role-manifest graph statically verifiable in CI.

2. Role-manifest format — dark-factory/host-roles/<role>.yaml

A role manifest is a YAML document with a small fixed top-level shape and a flat list of typed resource records. The top level:

role: gyrum-prober           # MUST match the filename stem and the ADR-101 enum value
version: 1                   # bumps when the manifest semantically changes; reconciler keys cache by (role, version)
description: |               # one paragraph for the operator-facing /infra/hosts UI
  Tiny heartbeat daemon every fleet host carries so the watchdog can detect
  liveness loss before the missing-probe incident shape recurs.
requires: []                 # other role names this role depends on (must be declared on the same host); empty for prober
mutually_exclusive: []       # other role names that must NOT co-reside; empty for prober
resources:
  - { type: package, ... }
  - { type: file, ... }
  - { type: systemd_unit, ... }

The resources list is flat (not nested). Order within the list defines apply order; the reconciler honours it. Each resource record carries a type discriminator naming one of the closed resource-type set below, and the type-specific fields documented per type. The closed set is the schema's contract surface.

Closed resource-type enum (v1):

• package — apt/dpkg package. Fields: name (string), version (string, present for any installed version), state (enum present | absent, default present).
• file — file on disk. Fields: path (absolute), content (inline string OR source: <relpath> referencing host-roles/<role>/files/<relpath>), mode (octal string, e.g. "0644"), owner (string), group (string), state (enum present | absent).
• directory — directory on disk. Fields: path, mode, owner, group, state.
• systemd_unit — systemd service/timer/socket. Fields: name (e.g. gyrum-prober.service), state (enum started | stopped), enabled (bool), unit_file (path to the file resource that declares the unit; reconciler asserts the file exists in the same manifest).
• env — environment file managed via systemd EnvironmentFile=. Fields: path, vars: {key: value}, mode (default "0640"). Secrets MUST come from secret_ref: (resolved via the broker secret-fetch shape, ADR-111-adjacent); inline secret values are validator-rejected.
• command — escape hatch. Fields: run (the command), creates (path that exists iff the command has run; idempotency proxy), idempotent (bool, default true; if false the reconciler emits a warn-level drift note explaining why this resource is special).

Adding a new resource type is an ADR amendment, same discipline ADR-101 imposes on the role enum. The closed set exists so the observed-state query layer (§3) and the drift-item format (§4) have a finite shape.

3. Observed-state — what the reconciler queries from a host

Per resource type, the reconciler runs a deterministic query and reduces the result to a comparable observed-state record. The queries are documented as part of the reconciler's contract, not as ad-hoc shell:

• package → dpkg-query -W -f='${Status} ${Version}\n' <name>; reduce to {installed: bool, version: string}.
• file → stat + sha256sum; reduce to {exists: bool, mode: string, owner: string, group: string, sha256: string}. The reconciler computes the declared file's sha256 from content (or source) and compares.
• directory → stat; reduce to {exists: bool, mode, owner, group}.
• systemd_unit → systemctl is-active <name> + systemctl is-enabled <name>; reduce to {active: bool, enabled: bool, unit_file_present: bool}.
• env → read file + parse KEY=value; reduce to {exists: bool, mode, vars: {...}}. Secret-resolved values are compared by hash, not plaintext, so the drift-item never echoes a secret.
• command → check creates path's existence; reduce to {has_run: bool}.

The observed-state queries are read-only by contract: a --dry-run reconciler invocation MUST NOT modify host state. This is enforceable because the closed query set above is the only path from reconciler to host; any new query rides through an ADR amendment that re-asserts the read-only property.

4. Drift-item format — the shared check/apply representation

The drift-item is the unit of currency between check and apply. JSON-schema-precise:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://gyrum.ai/schemas/drift-item.v1.json",
  "type": "object",
  "required": ["host", "role", "resource", "declared", "observed", "action_to_apply", "severity"],
  "properties": {
    "host":     { "type": "string", "description": "host name from gyrum-catalog/hosts/<name>.yaml" },
    "role":     { "type": "string", "description": "role name from ADR-101 enum" },
    "resource": {
      "type": "object",
      "required": ["type", "id"],
      "properties": {
        "type":  { "enum": ["package","file","directory","systemd_unit","env","command"] },
        "id":    { "type": "string", "description": "stable identifier within the role-manifest, e.g. 'package:gyrum-prober' or 'file:/etc/gyrum/prober.yaml'" }
      }
    },
    "declared":        { "description": "the declared resource record from the role-manifest, type-shaped per §2" },
    "observed":        { "description": "the observed-state record per §3, or null when the resource is absent on the host" },
    "action_to_apply": {
      "type": "object",
      "required": ["verb"],
      "properties": {
        "verb":   { "enum": ["install","upgrade","remove","write","chmod","chown","start","stop","enable","disable","run","noop"] },
        "params": { "type": "object", "description": "verb-specific parameters; the reconciler executes verb(params) when not in dry-run" }
      }
    },
    "severity":        { "enum": ["drift","warn","error"] },
    "idempotent":      { "type": "boolean", "default": true },
    "discovered_at":   { "type": "string", "format": "date-time" }
  }
}

A reconciler run produces a drift_report envelope: {run_id, host, started_at, finished_at, dry_run: bool, items: [drift-item, ...], stats: {by_severity, by_verb, by_resource_type}}. The envelope is what the operator UI renders, what Warp tickets reference ({drift_report_id, item_index}), and what --dry-run returns to the caller. --dry-run does not execute any action_to_apply; non-dry-run executes them in items[] order and emits a second, post-apply drift-report whose items[] SHOULD be empty (the convergence check).

This is the shape that makes "are check and install the same with --dry-run" structurally true: they run the same code path, produce the same envelope, and differ only in whether action_to_apply.verb is executed.

5. Idempotency contract

Every action verb MUST be idempotent: applying the same drift-item twice in succession produces the same observed-state as applying it once. The reconciler enforces this by re-querying observed-state after apply and asserting convergence (the post-apply drift-report has zero matching items for the just-applied resource). A resource that cannot be expressed idempotently uses type: command with idempotent: false declared explicitly; the reconciler accepts this but emits a warn-level drift-item every run (so the deviation is visible, not silent). The default for command resources is idempotent: true; false is an opt-in escape hatch with a documented reason in the role-manifest's description.

This contract is what lets the reconciler run on a cron without harm: if nothing has drifted, every action is a noop; if something has drifted, the action converges.

6. Bootstrap path

A brand-new host gets the reconciler before it gets any other workload. The bootstrap path:

1. Cloud-init / kickstart phase (provider-side). The host comes up with git, curl, and a one-shot systemd unit gyrum-bootstrap.service that:
   • clones dark-factory shallow,
   • resolves the host's roles[] from a baked-in BOOTSTRAP_HOST_NAME env var by reading gyrum-catalog/hosts/<name>.yaml,
   • runs the reconciler against that role list with --apply.

1. Reconciler self-install (first reconcile run). The base host role-manifest (host-roles/host.yaml, declared in this ADR's scope as the universal-prerequisite role) installs the reconciler as a systemd timer (e.g. weekly check-run, on-demand apply via SSH). After the first run the cloud-init unit is no longer load-bearing.
2. Stage markers. The bootstrap writes ADR-147-shape stage markers (/var/lib/gyrum/bootstrap/<stage>.done) so a re-run is idempotent and the operator can see the bootstrap state without re-reading logs.

The bootstrap path is the answer to how does the host get the reconciler in the first place, and it sits beneath every role-manifest (the host role is the first thing every host carries; ADR-101's "base host role applied unconditionally" pre-existed; this ADR formalises that the base role is itself a role-manifest, not a special-case ansible playbook).

7. Worked example — host-roles/gyrum-prober.yaml

The canonical first role, declared inline so this ADR is concrete:

role: gyrum-prober
version: 1
description: |
  Tiny heartbeat daemon every fleet host carries so the watchdog can detect
  liveness loss. Three resources: the binary (apt package), its config file,
  and its systemd unit. This is the simplest role; if the schema cannot
  describe it cleanly the schema is wrong.
requires: []
mutually_exclusive: []
resources:
  - type: package
    id: package:gyrum-prober
    name: gyrum-prober
    version: present
    state: present

  - type: directory
    id: directory:/etc/gyrum
    path: /etc/gyrum
    mode: "0755"
    owner: root
    group: root
    state: present

  - type: file
    id: file:/etc/gyrum/prober.yaml
    path: /etc/gyrum/prober.yaml
    content: |
      heartbeat_interval_seconds: 60
      warp_endpoint: https://warp.gyrum.ai/api/v1/heartbeat
      host_name_source: /etc/hostname
    mode: "0640"
    owner: root
    group: gyrum
    state: present

  - type: systemd_unit
    id: systemd_unit:gyrum-prober.service
    name: gyrum-prober.service
    state: started
    enabled: true
    unit_file: /lib/systemd/system/gyrum-prober.service   # shipped by the apt package above; reconciler asserts presence after install

A check run on a host missing gyrum-prober produces a drift-report with four items (package install, directory create, file write, unit start+enable). An apply run executes them in order; a subsequent check run produces zero items. That round-trip is the test case for the reconciler at H1.4.

Consequences

What becomes easier:

• "What should be on this host" is a flat YAML, not tribal knowledge. Every role's contents are declared, validated, and version-controlled. The four missing-probe hosts become a one-PR fix: their hosts/<name>.yaml adds gyrum-prober to roles[], the next reconcile run installs the role, the post-apply drift-report is empty.
• Check and apply share one code path. --dry-run returns the drift-report envelope; non-dry-run executes the same items[] in place. The structural answer to "are they the same" is yes; the operator's intuition matches the implementation.
• Drift is mechanical, not interpretive. The drift-item is a JSON record with a closed verb set, not "the playbook says CHANGED". The operator UI iterates items[], the Warp ticket references {drift_report_id, item_index}, the reconciler's stats roll up cleanly per-severity / per-verb / per-resource-type.
• Idempotency is contract, not aspiration. Every action verb converges; the post-apply re-query proves it; non-idempotent escape hatches are explicit and visible. Cron-running the reconciler is safe by construction.
• Bootstrap is the same shape as steady-state. A new host's first reconcile run is structurally identical to its hundredth. There is no "install" path that diverges from "reconcile"; both produce drift-reports.
• The role enum gets observable depth. ADR-101's [edge, ci-runner, ...] was a name; this ADR makes each name a manifest a reviewer can read in one screen. New role proposals come with their manifest as the concrete artefact, not a prose description.

What becomes harder:

• Two manifest layers to maintain instead of one. hosts/<name>.yaml (host axis) and host-roles/<role>.yaml (role axis), joined by the role-name. We accept the coupling because the alternative — implicit role definitions in playbooks — is what got us here.
• The closed resource-type set is a narrow door. Six types cover the prober, the host base role, and the obvious cases for edge / db; new types (e.g. cron_job, caddy_vhost, ufw_rule) require an ADR amendment. We accept the friction; uncontrolled type expansion is the same failure mode the role enum's closed-set discipline exists to prevent.
• The reconciler is now load-bearing. H1.4 ships the implementation; until then this ADR's contract is theoretical. We accept the staging because the contract has to land first or the implementation has nothing to implement against.
• Secrets handling is now part of the role-manifest contract. env resources reference secrets via secret_ref: (broker-resolved, ADR-111); inline secrets are validator-rejected. Operators authoring role-manifests must use the secret-fetch shape. This is correct but adds onboarding cost.
• Backfilling existing hosts is real work. warp-01 carries [edge, web, api, db] today; each of those four roles needs a manifest written. We accept it because the alternative is keeping the implicit shape we already know is broken; the backfill happens incrementally as each role is touched.

What we have signed up to operate:

• dark-factory/host-roles/<role>.yaml — one file per role in the ADR-101 enum. The role-manifest is the source of truth for what the role contains.
• The closed resource-type set (package | file | directory | systemd_unit | env | command). Adding a new type is an ADR amendment.
• The drift-item JSON schema at https://gyrum.ai/schemas/drift-item.v1.json. Versioned; v2 is a separate schema URL, not an in-place mutation.
• The observed-state query contract per type — read-only, deterministic, documented alongside the resource-type definition. New queries ride through the same ADR amendment that adds their type.
• The bootstrap path (gyrum-bootstrap.service cloud-init unit, host role-manifest as the universal prerequisite, ADR-147 stage markers).
• Referential integrity in validate-host-manifest — a host's roles[] entry without a corresponding host-roles/<role>.yaml fails validation.

What we revisit:

• When a third manifest layer appears (today: project, host, role. Plausibly: tenant, region) the pattern this ADR continues — declared YAML + closed-enum types + idempotent applier + drift-item-as-currency — is the template. Re-examine at the third use.
• When a role-manifest grows past ~30 resources the flat-list shape starts costing more than it saves; revisit whether structured composition (sub-roles, includes) is needed. Today's prober is 4 resources; a realistic edge is ~15.
• When the secret-fetch shape stabilises (post-broker, ADR-111) the env resource's secret_ref: semantics may move to a richer reference (e.g. versioned secrets, rotation hooks). The drift-item compares secret-resolved values by hash today; that contract survives the upgrade.

Alternatives considered

• Use Ansible role variables as the schema. What ADR-101 implicitly assumed and what dark-factory/ansible/role-*.yml does today. Loses on every reason this ADR exists: Ansible variables are not a typed schema, the observed-state query is whatever the role's tasks/main.yml decides, drift is "ansible says changed", and there is no shared check/apply representation. Rejected — the playbook stays as one possible reconciler implementation, but the schema lives above it.
• Use Puppet/Chef-style DSL. Mature, battle-tested, well-typed. Loses on the cost-to-onboard and the lock-in: a DSL the fleet does not own becomes a dependency the reconciler at H1.4 has to embed or shell out to. The closed resource-type set above is small enough that a bespoke YAML schema costs less to maintain than a Puppet manifest parser. Rejected — we are not Puppet's scale.
• Open string set for resource types. Less ceremony, easier to add types. Loses on the validation story (typos pass silently, observed-state queries cannot be defined for unknown types) and on the drift-item contract (the JSON schema's enum is the gate). Same reasoning as ADR-101 chose for role names. Rejected.
• Drift-item is unstructured prose / log lines. What ansible-playbook --check emits today. Loses on the operator UI (no rendering target), the Warp ticket reference (no stable item index), the reconciler convergence check (no machine-comparable post-apply), and the operator question this ADR exists to answer (check ≡ apply --dry-run only if they share a representation). Rejected.
• Two separate code paths for check and apply, each emitting their own format. What we'd get if we shipped a check tool first and an apply tool later. Loses on the structural answer to the operator's 2026-05-06 question, and loses on the convergence check (the post-apply re-query relies on the apply path emitting the same shape). Rejected.
• Per-role JSON schemas instead of a unified drift-item.v1.json. Stronger type safety at the leaves, weaker uniformity at the trunk. Loses on the operator UI, which would have to switch on role before knowing how to render the item; loses on the Warp ticket shape, which would need per-role tags. The unified schema with a resource: {type, id} discriminator gives 95% of the safety at 10% of the cost. Rejected.
• Skip the bootstrap path; assume the operator runs gyrum-reconcile manually first time. Cheaper to ship. Loses on the four-missing-probe shape this ADR exists to prevent: a host the operator forgot to bootstrap is exactly a host with no record of what it should carry. The cloud-init self-install is the gate that closes that door. Rejected.

Supersedes: none. Extends ADR-101 by giving each roles[] enum value a declarative manifest layer. Composes with ADR-148 by sharing the idempotent-apply contract and ADR-147 stage marker pattern. Sister to phase-0 ADRs of warp#1582: H0.1 (hosts.yaml schema, extended here) and H0.2 (critical-playbook watchdog, different surface). Superseded by: leave blank until a later ADR reverses this one