ADR-072: Infrastructure as playbooks

Status: Accepted Date: 2026-04-23 Related: ADR-067 (playbooks unified primitive), ADR-068 (playbook runtime — §14/§15 extend the Phase 4 executor roster with type: ansible), ADR-069 (playbook security), ADR-070 (playbook UI contract), ADR-071 (product name — Weft), ADR-064 (gyrum-catalog + hex arch), ADR-060 (catalog-driven infrastructure), ADR-053 (frontend observability), ADR-020 (shared infra vs per-product isolation)

Decision (one paragraph)

Every infrastructure mutation Gyrum performs — DNS records, Cloudflare Tunnel routes, Cloudflare Access applications and policies, Caddy site configuration, systemd unit state, and every future sibling surface — is expressed as an idempotent, dry-runnable Weft playbook. Deploys compose these sub-playbooks via type: subplaybook steps (ADR-068 §4). Rollback is "re-run the previous-version playbook". The infrastructure source of truth is the committed YAML in dark-factory/playbooks/infra/ plus the playbook_run_events audit stream (ADR-068 §5); runs reconcile actual state to desired. The catalogue ships in phases: a monolithic deploy-service playbook today, an eight-playbook sub-catalogue once the subplaybook and http executors land (ADR-068 Phase 4), and per-playbook scoped API tokens plus destructive-step approvals once security hardens (ADR-069 Phase 5). A fleet-wide drift reconciler is named as a future capability, not a current commitment.

Context

Weft is a dark factory. The whole point of the loom metaphor (ADR-071 §2) is that a pattern encoded in punch cards weaves products without a human on the floor — and today, when we deploy a new product or move an existing one, a human leaves the loom and goes into the Cloudflare dashboard to click through a DNS record, a Tunnel route, an Access application, and a Caddy reload. That is not a dark factory. That is an operator with a browser tab open and a shell history full of curl one-liners.

Concretely, deploying a new service today means touching at least five surfaces, each with its own interface, its own idempotency story (or lack of one), and its own rollback story (or lack of one):

1. DNS. A new A or CNAME record on the Cloudflare-hosted zone. Clicked in the UI. Sometimes curl-ed against the Cloudflare API from a laptop. No record of which record was created when, by whom, or at whose request.
2. Cloudflare Tunnel. A route added to the cloudflared tunnel that connects the public internet to the origin host behind NAT. Clicked in the Zero Trust UI. The mapping from hostname to origin service is in CF's database, not ours.
3. Cloudflare Access. An application created to gate the hostname, policies attached to allow specific email addresses or identity groups. Clicked in the Zero Trust UI. The membership of an Access app drifts silently when a user's email changes.
4. Caddy. The origin reverse proxy. A new site block added to the Caddyfile, caddy reload ssh'd onto the host. The Caddyfile is in git (in dark-factory/infra/caddy/) but the reload step is tribal and the state on disk can diverge from the state in git without anyone noticing.
5. systemd. For services not in Docker, a unit file installed and enabled, systemctl daemon-reload && systemctl restart. Again, the unit file can be in git; the "is it running?" state can only be observed by ssh'ing and asking.

Each of those surfaces has at least one of four failure modes:

• Not auditable. A Cloudflare UI click does not appear in any log the engineering team reads. Six months later, when a DNS record has the wrong origin, there is no answer to "who changed this and why?".
• Not reproducible. The procedure lives in somebody's head, or in a Slack thread, or in a PR description that has since been squash-merged and lost its context. A new operator cannot re-run the steps without tribal knowledge.
• Not rollback-able. Clicking "Create record" in Cloudflare does not create a reverse "Delete record" script. Rollback is "clicking backward", which is error-prone and uneven across surfaces.
• Not composable. A deploy needs DNS + Tunnel + Access + Caddy, but there is no primitive for "run all four in order, halt on the first failure, and give me a single audit trail of the combined run". We get four audit trails (at best) and coordinate them by eye.

As Weft grows toward its product ambition (ADR-071 §2.1 — "finished fabric" being the SaaS products the platform produces), the number of services on the fleet goes up and to the right. Each service wants its own DNS record, its own Tunnel route, its own Access policy, its own Caddy site. Today that is two products. In twelve months it will be twenty. The manual click-path does not scale linearly, let alone to twenty. It fails worse than linearly because the probability of a human forgetting a step or misconfiguring a field per deploy is roughly constant, so the expected number of broken deploys per week scales with the deploy rate.

Meanwhile, ADR-067 landed the playbook as the unified primitive for operational procedures, ADR-068 built the runtime that executes them, ADR-069 locked down the security model, and ADR-070 fixed the UI contract. Weft already has a way to express a repeatable, audited, interactive procedure. Every infra operation fits the playbook shape: trigger, steps, outcome, retro/gaps (ADR-067 §3). The primitive exists; we are currently refusing to use it for the most valuable class of procedure.

This ADR closes that gap by committing to a pattern: infra is playbooks, and infra is playbooks all the way down. A deploy is a playbook that calls sub-playbooks. A DNS change is a sub-playbook. A Tunnel route is a sub-playbook. Each sub-playbook is idempotent, dry-runnable, and scoped to the minimum privilege it needs. The YAML in dark-factory/playbooks/infra/ is the source of truth; the Cloudflare UI, when it disagrees, is wrong.

Why not Terraform / Pulumi?

This is the obvious question and the right one to answer early. See §10 for the full argument; the short version is that Terraform and Pulumi solve a strict superset of what we need and introduce a second tool, a second state file, a second DSL, and a second audit trail. We already ship a runtime (ADR-068), a security model (ADR-069), a UI (ADR-070), and a catalogue of playbooks. Duplicating those in HCL-shaped containers is a paying-for-what-we-already-have move. If at Phase 6+ we discover that managing hundreds of infra resources in YAML feels worse than HCL, we revisit — the escape hatch exists. It does not exist now.

Why now?

Three converging pressures:

1. deploy-service is landing this cycle (see the sample playbooks referenced in ADR-068 §13 — hello-world.yaml, goodbye.yaml, and deploy-service.yaml as a parallel follow-up). If deploy-service ships as a bespoke shell script today, the decomposition cost later is a rewrite. If it ships as a playbook that inlines the four-to-five infra steps today, the decomposition later is a refactor into subplaybook calls. Framing matters; we name the pattern now so the first deploy playbook is shaped for decomposition.
2. ADR-068 Phase 4 is on the runway. The subplaybook and http executors are the two primitives that make the sub-playbook catalogue in §5 executable. Phase 4 is "ongoing" per ADR-068 §14; the sub-playbook PRs are the natural work items that fill Phase 4.
3. ADR-069's scoped-token pattern is implementable. ADR-069 §3 already gated target: local behind an allow-list and named the secret vault. Per-playbook scoped API tokens are the natural extension of that pattern to external APIs (Cloudflare, specifically). Committing the pattern now gives ADR-069 Phase 5 a concrete work item.

Decision

1. Summary of gating choices (committed, not re-litigated)

Six decisions are settled. The table is the quick index.

All six enforcements are additive to ADR-068's runtime and ADR-069's security model. No new StepExecutor kind is introduced; infra playbooks compose type: http, type: ssh, type: shell, type: approval, and type: subplaybook.

2. The pattern — four non-negotiables

The pattern is four rules. A playbook under playbooks/infra/ that violates any of the four is rejected at review time; a future CI check (§9) rejects them at parse time. These rules are non-negotiable in the sense that loosening them would let the first bad playbook corrupt the audit trail or the state reconciler that depends on them.

2.1 Idempotency — read current state, compute delta, apply only if changed

Every infra sub-playbook has three phases, every time:

1. Read. Call the underlying API (GET /zones/{id}/dns_records, GET /accounts/{id}/tunnels/{tunnel_id}/configurations, ssh host cat /etc/caddy/Caddyfile) to establish what is.
2. Compute delta. Compare the desired state (derived from the playbook's inputs:) with the read result. Produce a structured delta object: {action: none|create|update|delete, current: …, desired: …, diff: …}.
3. Apply — conditionally. If delta.action == none, emit a step_completed event with outputs: {changed: false, delta: …} and return. Otherwise apply the change via the underlying API and emit step_completed with outputs: {changed: true, delta: …, response: …}.

The invariant: re-running the same playbook with the same inputs against a converged system produces changed: false. A playbook that returns changed: true on the second run is buggy and is a CI failure (Phase 4+ when the reconciler can run playbook pairs).

This is the classical configuration-management idempotency property (Ansible, Chef, Puppet all enforce it). We enforce it structurally by making the three phases the shape of every infra sub-playbook's body, not by trusting authors to remember.

2.2 Dry-run mode — show the delta, apply nothing

Every infra sub-playbook accepts a dry_run: bool input (default false). When dry_run: true:

1. Read phase runs as normal.
2. Compute-delta phase runs as normal.
3. Apply phase is skipped entirely. The step emits step_completed with outputs: {changed: null, dry_run: true, delta: …}. null (not false) signals "we don't know whether the apply would have succeeded, only that we would have attempted it".

Why the tri-state (changed: true | false | null):

• false means the playbook ran and the system was already converged. Safe to re-run.
• true means the playbook ran and mutated the system. Audit trail matters.
• null means the playbook ran in dry-run mode. The delta is real but no mutation happened.

Dry-run is required before the first real run of any destructive playbook. See §6 for the destructive-step pairing.

deploy-service threads dry-run through: passing dry_run: true to the top-level deploy runs every sub-playbook in dry-run mode and produces a combined "what would change" report. The operator reads it, confirms, and re-runs with dry_run: false. ADR-070 §1's <PlaybookRunner> surfaces the dry-run toggle as a checkbox on the run-launch form.

2.3 Secrets via ${env.X} + vault references — never inline

No infra playbook ever contains a literal secret value. The authoring rule is absolute:

• ${env.CF_API_TOKEN} — the runtime resolves env.X against the process environment of the worker, not the playbook author's environment. The author declares the variable name; the operator/SRE provisions the value out-of-band.
• ${vault.cf_api_token_rw} — the runtime resolves vault.X against the ADR-069 secret vault by id. The vault id is referenceable; the value is not streamable into the playbook_run_events log (ADR-069 §4.3).
• Never inline. A literal CF_API_TOKEN: "abc123" in YAML fails parse-time validation. The validator rejects any field whose resolved value is a non-redacted high-entropy string matching a known secret pattern (CF API tokens are 40 chars base64; GitHub tokens begin ghp_ / gho_ / etc).

The vault-vs-env choice per playbook:

• ${env.X} is appropriate for bootstrap secrets — the CF token the runtime itself uses to operate, provisioned by the operator at runtime startup.
• ${vault.X} is appropriate for customer secrets — secrets a deployed service consumes, which the runtime never needs to read but an http step needs to forward as a header.

Both resolutions happen at step-execution time, inside the executor, after authorisation (ADR-069 §2). Neither secret form appears in playbook_run_events.body — the event body stores the string ${env.CF_API_TOKEN} verbatim, not the resolved value.

The terminal-pane scrub (ADR-068 §4 prompt_secret / ADR-069 §4.3) extends here: any stdout line that contains the resolved value gets ••••••••-substituted before the stdout event is emitted. Best-effort, reverse-lookup, per ADR-069 §4.3.

2.4 Scoped blast radius — minimum-privilege token per playbook

Every infra sub-playbook declares its required API-token scope in its front-matter. The declaration is audited at parse time and at run time.

# cloudflare-dns-upsert.yaml — front-matter excerpt
---
kind: service_runbook
id: cloudflare-dns-upsert
title: "Cloudflare: upsert a single DNS record"
persona: sre
owner: "@jon"
allow_local_execution: false          # no local bash
required_scopes:                      # ADR-072 addition
  - vendor: cloudflare
    scope: "Zone:DNS:Edit"
    zone_filter: "${inputs.zone_id}"
secret_refs:
  - env: CF_API_TOKEN_ZONE_DNS_EDIT   # the token matching the required scope
---

Rules:

• required_scopes: lists the minimum vendor-level API-token scopes the playbook needs. Cloudflare's scope taxonomy is used verbatim (Zone:DNS:Edit, Account:Cloudflare Tunnel:Edit, Account:Access: Apps and Policies:Edit, etc.).
• secret_refs: names the environment variable or vault id that holds a token matching those scopes. The operator, at token provisioning time, mints a token with exactly those scopes and installs it under that name. A token with more scopes fails a Phase 5 scope-verification check; a token with fewer scopes fails at first API call.
• One token per playbook, not one token per runtime. The runtime does not carry a single global CF_API_TOKEN that every playbook uses. Each sub-playbook references its own scope-specific token. Compromise of a Caddy playbook's token does not compromise the DNS playbook's token.
• A top-level orchestrator (e.g. deploy-service) declares the union of its sub-playbooks' required_scopes:. This is computed at parse time, not author-maintained, by walking the subplaybook: references.

The scope declaration is enforceable because the vendor (Cloudflare) publishes the scope taxonomy as a machine-readable list; Phase 5's scope-verification step calls GET /user/tokens/verify and checks the declared scope is a subset of the token's actual scope. See §7 for the phase gate.

3. The sub-playbook contract — shape in YAML

Every infra sub-playbook under playbooks/infra/ has the same structural shape. Authors do not invent a new shape per playbook; the shape is the contract. Deviations are code-review and CI failures.

---
kind: service_runbook
id: cloudflare-dns-upsert
title: "Cloudflare: upsert a single DNS record"
persona: sre
owner: "@jon"
status: active
trigger: "Called by deploy-service or invoked manually to add a hostname"
success_metric: "Record exists in CF with the desired content; changed-flag accurate"
tags: [infra, cloudflare, dns]
related:
  - runbooks:cloudflare-dns-delete
  - runbooks:cloudflare-tunnel-route-upsert
  - adrs:072
created: 2026-04-23
last_reviewed: 2026-04-23

# ADR-069 authorisation
allowed_personas: [sre, owner]
allow_local_execution: false

# ADR-072 additions
required_scopes:
  - vendor: cloudflare
    scope: "Zone:DNS:Edit"
    zone_filter: "${inputs.zone_id}"
secret_refs:
  - env: CF_API_TOKEN_ZONE_DNS_EDIT

# ADR-068 runtime section
runtime:
  inputs:
    - name: zone_id
      type: string
      required: true
      description: "Cloudflare zone id (the apex domain's id)"
    - name: name
      type: string
      required: true
      description: "Record name, e.g. 'app.example.com' or '@' for apex"
    - name: type
      type: enum
      values: [A, AAAA, CNAME, TXT, MX]
      required: true
    - name: content
      type: string
      required: true
    - name: proxied
      type: bool
      required: false
      default: true
    - name: dry_run
      type: bool
      required: false
      default: false
  steps:
    - id: read
      type: http
      target: host:api.cloudflare.com
      method: GET
      url: "https://api.cloudflare.com/client/v4/zones/${inputs.zone_id}/dns_records?name=${inputs.name}&type=${inputs.type}"
      headers:
        Authorization: "Bearer ${env.CF_API_TOKEN_ZONE_DNS_EDIT}"
      expect_status: [200]
      save_as: current
    - id: compute_delta
      type: shell            # a pure-data step; "shell" here is the ADR-068 §4 shell executor in its "local-deterministic-no-side-effects" mode
      target: local
      run: |
        # parse ${current}, compare with desired (${inputs.*}), emit delta JSON on stdout
      save_as: delta
    - id: apply
      type: http
      target: host:api.cloudflare.com
      when: "${delta.action != 'none' && !inputs.dry_run}"
      method: POST           # or PUT/PATCH depending on delta.action; expressions resolve this
      url: "…"
      headers:
        Authorization: "Bearer ${env.CF_API_TOKEN_ZONE_DNS_EDIT}"
      body: "${delta.desired}"
      expect_status: [200, 201]
      save_as: response
  outputs:
    changed: "${delta.action != 'none' && !inputs.dry_run}"
    dry_run: "${inputs.dry_run}"
    delta: "${delta}"
    record_id: "${response.result.id || current.result[0].id}"
---

# cloudflare-dns-upsert

Idempotent upsert of a single Cloudflare DNS record. Reads the record,
computes a delta, applies only if changed. Passes through in dry-run
mode.

## Symptoms

…

## Diagnostics

…

## Mitigations

…

Points the shape commits to:

• Kind is service_runbook. Infra sub-playbooks are runbooks for one specific piece of infrastructure. The body follows the service_runbook H2 convention (Symptoms / Diagnostics / Mitigations per ADR-067 §3).
• Three-phase step sequence — read, compute_delta, apply — is the canonical idempotency shape from §2.1. Authors may add phases (e.g. a post_verify step that re-reads and asserts the delta is now none), but may not collapse the three.
• apply has when: "${delta.action != 'none' && !inputs.dry_run}" — this is the idempotency + dry-run enforcement. A missing when: is a CI failure.
• outputs.changed is always set. Callers (deploy-service in particular) read this to decide whether to continue, report drift, or short-circuit.

The body sections (Symptoms / Diagnostics / Mitigations) are author-written and describe what the playbook does and when to run it — they are for humans; the runtime does not read them.

4. Catalogue layout — where the playbooks live

dark-factory/
└── playbooks/
    └── infra/
        ├── cloudflare-dns-upsert.yaml
        ├── cloudflare-dns-delete.yaml
        ├── cloudflare-tunnel-route-upsert.yaml
        ├── cloudflare-access-app-upsert.yaml
        ├── cloudflare-access-policy-upsert.yaml
        ├── caddy-site-upsert.yaml
        ├── caddy-site-delete.yaml
        ├── systemd-service-reload.yaml
        ├── deploy-service.yaml           # orchestrator; calls the others
        └── _index.yaml                   # auto-generated (ADR-067 §6)

Rules:

• One infra sub-playbook per file. Bundling (a single cloudflare.yaml with multiple kinds or multiple deploys inside) is a CI failure. One-file-one-playbook matches ADR-067 §4.
• File name = id field. cloudflare-dns-upsert.yaml must have id: cloudflare-dns-upsert. CI enforces.
• dark-factory/playbooks/infra/ is THE canonical location. Not per-service directories, not per-vendor subdirectories (no playbooks/infra/cloudflare/), not a new gyrum-infra repo. A flat directory surfaces the whole catalogue to ls and keeps the ADR-067 typed-reference resolver simple.
• Cross-cutting goes through dark-factory. An infra playbook that mutates a Cloudflare resource does not live in ai-frontend/playbooks/infra/ even if ai-frontend is the service that most often invokes it. Infra is a cross-cutting concern (ADR-067 §4); dark-factory is the cross-cutting repo.

The _index.yaml is auto-generated per ADR-067 §6. Infra playbooks appear alongside journey, experiment, and incident-response playbooks in the combined fleet catalogue.

5. The initial sub-playbook catalogue

Eight sub-playbooks form the Phase 4 target. This is the committed catalogue — every one of the eight is a tracked follow-up PR that cites this ADR. Each row gives the id, the key inputs, the declared scope, the vendor API the playbook calls, and whether the playbook is destructive. Bodies are sketched in §5.x below; full YAML is explicitly out of scope for this ADR per the scope constraint at the top of this PR.

Each sub-playbook follows the §3 shape. The rest of this section sketches the front-matter + step outline per playbook. Full YAML bodies land in dedicated follow-up PRs (one per playbook, ≤ 800 lines each, citing ADR-072 in the PR description).

5.1 cloudflare-dns-upsert

---
id: cloudflare-dns-upsert
kind: service_runbook
required_scopes: [{vendor: cloudflare, scope: "Zone:DNS:Edit", zone_filter: "${inputs.zone_id}"}]
secret_refs: [{env: CF_API_TOKEN_ZONE_DNS_EDIT}]
---
runtime:
  inputs: [zone_id, name, type, content, proxied?, dry_run?]
  steps:
    - id: read         # GET /zones/{id}/dns_records?name=…&type=…
    - id: compute_delta
    - id: apply        # POST (create) | PUT (update) | noop (none); when: delta.action != 'none' && !dry_run
  outputs: {changed, dry_run, delta, record_id}

Upsert semantics: CNAME to A requires a delete-then-create, not a PUT. The delta-compute step handles the type-change case by producing an action: replace delta; the apply step then issues a DELETE followed by a POST. All within the same step execution — partial failure leaves the record in whatever state CF is in at that moment, and the run fails; operator re-runs.

5.2 cloudflare-dns-delete

---
id: cloudflare-dns-delete
kind: service_runbook
required_scopes: [{vendor: cloudflare, scope: "Zone:DNS:Edit", zone_filter: "${inputs.zone_id}"}]
secret_refs: [{env: CF_API_TOKEN_ZONE_DNS_EDIT}]
# ADR-069 destructive annotation
destructive: true
---
runtime:
  inputs: [zone_id, name, dry_run?]
  steps:
    - id: read         # GET — find the record
    - id: compute_delta
    # implicit approval step injected by the runtime — ADR-069 §6
    - id: apply        # DELETE; when: delta.action == 'delete' && !dry_run
  outputs: {changed, dry_run, delta, deleted_record_id}

Destructive flag triggers ADR-069 §6 implicit-approval behaviour: the runtime prepends an approval step with the question "Delete DNS record ${inputs.name} (type ${current.type}, content ${current.content}) in zone ${inputs.zone_id}? This is irreversible." The operator must approve before apply runs. Dry-run mode skips the approval (there is nothing to approve — no mutation).

5.3 cloudflare-tunnel-route-upsert

---
id: cloudflare-tunnel-route-upsert
kind: service_runbook
required_scopes: [{vendor: cloudflare, scope: "Account:Cloudflare Tunnel:Edit"}]
secret_refs: [{env: CF_API_TOKEN_ACCOUNT_TUNNEL_EDIT}]
---
runtime:
  inputs: [tunnel_id, hostname, service_url, dry_run?]
  steps:
    - id: read         # GET /accounts/{id}/cfd_tunnel/{tunnel_id}/configurations
    - id: compute_delta  # tunnel configs are a single document with ingress list — delta is a list-patch
    - id: apply        # PUT the full config with the patched ingress list
  outputs: {changed, dry_run, delta}

Tunnel configuration is a single document with an ingress list, not a collection of routes. The read-modify-write is whole-document; concurrent runs race. The apply step uses an If-Match: ${current.etag} header (CF supports it) to fail-fast on lost updates. A failed apply due to etag mismatch triggers an on_error: fail with a specific error class (ErrTunnelConfigConflict) so the operator can re-run.

5.4 cloudflare-access-app-upsert

---
id: cloudflare-access-app-upsert
kind: service_runbook
required_scopes: [{vendor: cloudflare, scope: "Account:Access: Apps and Policies:Edit"}]
secret_refs: [{env: CF_API_TOKEN_ACCOUNT_ACCESS_EDIT}]
---
runtime:
  inputs: [domain, name, allowed_emails[], policy_name, dry_run?]
  steps:
    - id: read         # GET /accounts/{id}/access/apps?domain=…
    - id: compute_delta
    - id: apply_app    # POST | PUT app
    - id: apply_policy # POST | PUT default policy; only if app changed or policy differs
  outputs: {changed, dry_run, delta, app_id, policy_id}

Access apps and their policies are coupled — an app with no policy is unreachable. This playbook atomically ensures both. allowed_emails is the simplest policy shape (include by email); more complex rules (group membership, service token, warp-identity) are cloudflare-access-policy-upsert territory (§5.5).

5.5 cloudflare-access-policy-upsert

---
id: cloudflare-access-policy-upsert
kind: service_runbook
required_scopes: [{vendor: cloudflare, scope: "Account:Access: Apps and Policies:Edit"}]
secret_refs: [{env: CF_API_TOKEN_ACCOUNT_ACCESS_EDIT}]
---
runtime:
  inputs: [app_id, action, include_rules, exclude_rules?, require_rules?, dry_run?]
  steps:
    - id: read         # GET /accounts/{id}/access/apps/{app_id}/policies
    - id: compute_delta
    - id: apply        # POST | PUT | DELETE a specific policy
  outputs: {changed, dry_run, delta, policy_id}

Applies to one policy on one app. The composition in §5.4's apply_policy step is a subplaybook: call into this one with the minimum policy shape; bespoke policies bypass §5.4 and call §5.5 directly.

5.6 caddy-site-upsert

---
id: caddy-site-upsert
kind: service_runbook
required_scopes: [{vendor: ssh, scope: "host:ops-vps:sudo:caddy"}]
---
runtime:
  inputs: [host, hostname, backend_url, tls?, dry_run?]
  steps:
    - id: read         # ssh host 'cat /etc/caddy/Caddyfile'
    - id: compute_delta  # parse Caddyfile, compute block-upsert
    - id: apply        # ssh host 'caddy-upsert.sh'; passes the patched Caddyfile atomically
    - id: reload       # ssh host 'sudo systemctl reload caddy'; skipped if no change
    - id: verify       # ssh host 'caddy validate --config /etc/caddy/Caddyfile'
  outputs: {changed, dry_run, delta, caddyfile_hash}

Caddy is the first non-CF infra playbook. target: on the ssh steps is host:ops-vps (resolves via gyrum-catalog per ADR-068 §4 ssh executor). The scope ssh:host:ops-vps:sudo:caddy is our internal scope taxonomy, not a vendor one — it names the sudo command pattern this playbook requires (a sudoers entry scoped to /usr/bin/systemctl reload caddy and /usr/local/bin/caddy-upsert.sh precisely, nothing else).

The caddy-upsert.sh script on the host is itself source-controlled in dark-factory/infra/caddy/ and deployed by a separate bootstrap-the-host playbook (out of scope for this ADR). The playbook's contract with the script is: accept a Caddyfile on stdin, validate it, write it atomically, print the new Caddyfile hash. The playbook verifies that hash matches its own computation.

5.7 caddy-site-delete

---
id: caddy-site-delete
kind: service_runbook
destructive: true
required_scopes: [{vendor: ssh, scope: "host:ops-vps:sudo:caddy"}]
---
runtime:
  inputs: [host, hostname, dry_run?]
  steps:
    - id: read
    - id: compute_delta
    # implicit approval step — ADR-069 §6
    - id: apply        # ssh — remove block, reload
    - id: verify
  outputs: {changed, dry_run, delta}

Destructive twin of §5.6. The approval prompt shows the block being removed, in full, before proceeding.

5.8 systemd-service-reload

---
id: systemd-service-reload
kind: service_runbook
required_scopes: [{vendor: ssh, scope: "host:${inputs.host}:sudo:systemctl:${inputs.unit_name}"}]
---
runtime:
  inputs:
    - host              # catalog host alias
    - unit_name         # systemd unit name
    - mode              # one of: reload, restart, stop, start
    - dry_run?
  steps:
    - id: read         # ssh host 'systemctl is-active ${unit_name}' && 'systemctl show …'
    - id: compute_delta  # "running" → "running after restart"; mode-dependent
    # implicit approval if mode in [restart, stop]
    - id: apply        # ssh host 'sudo systemctl ${mode} ${unit_name}'
    - id: verify       # ssh host 'systemctl is-active ${unit_name}'
  outputs: {changed, dry_run, mode, was_active, is_active}

Generic enough to cover the 80% case (reload grafana-server, restart caddy, restart a product's own unit). Modes that imply downtime (restart, stop) carry the destructive approval; modes that do not (reload, start-when-inactive) do not. Enforcement lives in the YAML via conditional destructive: rather than a hard-coded flag — Phase 4's runtime knows how to interpret a template expression in destructive:.

6. Composition — the deploy-service orchestrator

deploy-service is the first orchestrator playbook. It takes a service definition and produces a deployed, publicly reachable, Access-gated, Caddy-fronted service. It is the single point where the eight sub-playbooks compose.

---
id: deploy-service
kind: release_flow
title: "Deploy a service — DNS + Tunnel + Access + Caddy + app"
persona: sre
owner: "@jon"
status: active
trigger: "Manual invocation by an SRE with operator role; also invokable by CI on tag push (future)"
success_metric: "Service reachable at https://${inputs.hostname}, returns expected /api/version payload, /metrics gated by Access"
tags: [infra, deploy, orchestrator]

allowed_personas: [sre, owner]

# Orchestrator declares the UNION of sub-playbook required_scopes.
# Author does not maintain this list — Phase 4 CI computes it from
# the subplaybook graph. Listed here for illustration.
required_scopes:
  - {vendor: cloudflare, scope: "Zone:DNS:Edit"}
  - {vendor: cloudflare, scope: "Account:Cloudflare Tunnel:Edit"}
  - {vendor: cloudflare, scope: "Account:Access: Apps and Policies:Edit"}
  - {vendor: ssh, scope: "host:ops-vps:sudo:caddy"}
  - {vendor: ssh, scope: "host:${inputs.host}:sudo:systemctl:${inputs.unit_name}"}

runtime:
  inputs:
    - name: service_name        # logical name, e.g. "distill"
      type: string
      required: true
    - name: hostname            # public FQDN, e.g. "distill.gyrum.ai"
      type: string
      required: true
    - name: host                # catalog host alias for the origin
      type: string
      required: true
    - name: unit_name           # systemd unit on the host
      type: string
      required: true
    - name: backend_url         # what Caddy proxies to, e.g. "http://localhost:8080"
      type: string
      required: true
    - name: tunnel_id           # which cloudflared tunnel routes this service
      type: string
      required: true
    - name: zone_id             # CF zone for the hostname
      type: string
      required: true
    - name: allowed_emails      # Access allow-list
      type: list[string]
      required: true
    - name: dry_run
      type: bool
      default: false

  steps:

    - id: pre_verify
      type: http
      target: host:${inputs.hostname}
      method: GET
      url: "https://${inputs.hostname}/api/version"
      expect_status: [200, 404, 502, 503]    # "it's fine if this fails — we're about to make it work"
      on_error: continue
      save_as: pre

    - id: ensure_dns
      type: subplaybook
      playbook: service_runbook:cloudflare-dns-upsert
      with:
        zone_id: "${inputs.zone_id}"
        name: "${inputs.hostname}"
        type: "CNAME"
        content: "${inputs.tunnel_id}.cfargotunnel.com"
        proxied: true
        dry_run: "${inputs.dry_run}"
      save_as: dns

    - id: ensure_tunnel_route
      type: subplaybook
      playbook: service_runbook:cloudflare-tunnel-route-upsert
      with:
        tunnel_id: "${inputs.tunnel_id}"
        hostname: "${inputs.hostname}"
        service_url: "http://${inputs.host}:${port_from_backend_url}"
        dry_run: "${inputs.dry_run}"
      save_as: tunnel

    - id: ensure_access
      type: subplaybook
      playbook: service_runbook:cloudflare-access-app-upsert
      with:
        domain: "${inputs.hostname}"
        name: "${inputs.service_name}"
        allowed_emails: "${inputs.allowed_emails}"
        policy_name: "default-email-allowlist"
        dry_run: "${inputs.dry_run}"
      save_as: access

    - id: deploy_app
      type: ssh                # or "ansible" / "k8s" in future variants; ssh for today
      target: host:${inputs.host}
      run: |
        sudo systemctl is-active ${inputs.unit_name} || sudo systemctl enable --now ${inputs.unit_name}
      when: "!inputs.dry_run"
      save_as: app

    - id: ensure_caddy
      type: subplaybook
      playbook: service_runbook:caddy-site-upsert
      with:
        host: "ops-vps"
        hostname: "${inputs.hostname}"
        backend_url: "${inputs.backend_url}"
        dry_run: "${inputs.dry_run}"
      save_as: caddy

    - id: wait_ready
      type: shell
      target: local
      when: "!inputs.dry_run"
      run: |
        for i in $(seq 1 30); do
          curl -sf https://${inputs.hostname}/api/version && exit 0
          sleep 2
        done
        exit 1
      timeout: 90s

    - id: verify_sha
      type: http
      target: host:${inputs.hostname}
      method: GET
      url: "https://${inputs.hostname}/api/version"
      when: "!inputs.dry_run"
      expect_status: [200]
      save_as: post

  outputs:
    dns_changed: "${dns.outputs.changed}"
    tunnel_changed: "${tunnel.outputs.changed}"
    access_changed: "${access.outputs.changed}"
    caddy_changed: "${caddy.outputs.changed}"
    version_before: "${pre.body.version || 'unknown'}"
    version_after: "${post.body.version || 'unknown'}"
    dry_run: "${inputs.dry_run}"
---

The composition points:

• pre_verify is allowed to fail. expect_status: [200, 404, 502, 503] + on_error: continue captures "the service might not exist yet, or might be down — that is fine, we're about to make it work". Its output feeds the before/after comparison.
• Every infra step is a subplaybook: call. deploy-service does not inline the Cloudflare API or the Caddy edit. It composes the catalogue.
• dry_run threads through. A dry-run deploy invokes each sub-playbook in dry-run mode. The combined report is readable in the <PlaybookRunner> terminal pane.
• Each subplaybook step can be skipped if inputs not provided. The minimal deploy (an already-networked service that just needs the systemd unit restarted) is deploy_app + wait_ready + verify_sha — omit the zone_id/tunnel_id/allowed_emails inputs and the infra steps when-out. (Threading the when: expressions is an author responsibility; Phase 4 ADR-068 supports when: on subplaybook steps per §4.)
• Max subplaybook depth of 5 is plenty. This orchestrator is depth 2 (itself → sub-playbook → http / ssh). ADR-068 §4's depth cap of 5 gives us room for future composition (e.g. a deploy-product playbook that composes multiple deploy-service calls).

7. Phase gates — what lands when

Infra-as-playbooks is not a big-bang. It ships incrementally, each phase aligned with the ADR-068 phase plan, each phase independently useful.

Phases are additive. Phase 1 ships value (auditable deploy) with no Phase 4 prerequisite; Phase 4 is cleaner (one sub-playbook per concern) with no Phase 5 prerequisite; Phase 5 is secure; Phase 6 is self-healing. A deploy that runs at Phase 1 still runs at Phase 6, unchanged — the wrapping got tighter, the contract did not change.

8. GitOps angle — YAML as source of truth

The committed YAML in dark-factory/playbooks/infra/ is the canonical infra spec. Concretely:

• A change to a service's DNS record is a PR that edits the call-site in deploy-service or the inputs the caller passes, reviewed through the normal 3-persona formal review (ADR-067 §7 — Priya / Marcus / Lin + Dana advisory), and applied by running the playbook on merge. Not a click in Cloudflare. Not a curl from a laptop. A PR.
• Drift detection is a first-class capability. If someone (including the operator) edits DNS in the Cloudflare UI directly, the next Phase 6 reconciler run on cloudflare-dns-upsert with the current PR's inputs computes a non-empty delta and either reverts (aggressive mode) or alerts (conservative mode, which is what we ship first).
• "What does our infra look like right now?" is a report query. Run every infra playbook in dry-run mode, collect the current outputs of each read step, render the aggregate. This is a Phase 6 capability; Phase 5 gives us the building blocks.
• Rollback is a re-run of the previous version. The playbook file is versioned in git; playbook_runs.playbook_version is the content hash per ADR-068 §5. To roll back: git revert the PR that edited the playbook, re-run the now-reverted playbook. The re-run produces changed: true, delta: {action: update, …} and converges the system. No bespoke rollback script per surface.

Mentioned as a future capability, not a current commitment. Phase 6's reconciler is the capability that makes the GitOps loop complete. Phase 1-5 give us: auditable changes, scoped secrets, and a convergent contract — enough. Phase 6 closes the loop (detects and remediates out-of-band changes). We commit to Phase 1-5 now, and to naming Phase 6 as the next horizon; we do not commit to shipping Phase 6 in this ADR.

9. Enforcement — where each rule is checked

The four non-negotiables (§2) are not a style guide. They are enforced at specific points in the pipeline:

The playbook-lint CI job is new work; scoped to Phase 4. Until then, code review enforces. The point of the table is to name the boundary — rules that look like author discipline today become machine-checkable by Phase 4.

10. Alternatives considered

• Terraform (or OpenTofu). Rejected, for now. Terraform solves a strict superset of what we need — it already has read/compute/apply semantics, plan mode (our dry-run), state files, lockable backends, scoped providers. Using Terraform would look like: one Terraform module per Cloudflare resource kind, a state file in S3 (or equivalent), terraform plan in CI, terraform apply on merge. It works. It is battle-tested. Reasons we reject it:

  1. Second tool, second DSL, second audit trail. We already have Weft, YAML, and the playbook_audit table. Terraform introduces HCL, a state backend, and a separate audit log (whatever the state-file history shows). Operators have to context-switch: "is this a Weft playbook concern or a Terraform concern?" The boundary drifts over time.
  2. State-file operational burden. Terraform state files get corrupted, locked, diverged. Managing them is a third job on top of the two we already do (running Weft, running Postgres). We would add an S3 bucket with versioning, a DynamoDB lock table (or equivalent), and a runbook for terraform state rm.
  3. Worse UX for the interactive cases. Terraform's input UX is batch (-var-file, env vars). Our playbook UX is interactive (ADR-068 prompt / prompt_secret / approval). A terraform apply that needs to pause for a typed API key at step 3 of 7 is not a thing Terraform does. We would bolt on a wrapper — at which point we have re-implemented the playbook runtime around Terraform.
  4. Playbook composition beats Terraform composition for our case. We have exactly one deploy-orchestrator shape (deploy-service) calling ~8 sub-resources, not hundreds of modules. Terraform shines when the graph is large and state-sharing is intricate; our graph is small and mostly star-shaped.

  Reasons we might revisit: if our infra resource count hits the low hundreds (Phase 6+), Terraform's mature plan/apply and state-diff tooling might out-earn the integration cost. The escape hatch is: write a type: terraform executor that invokes terraform apply with a pinned module. We do not build it now; naming it is enough.

• Pulumi. Rejected for the same reasons as Terraform, plus: Pulumi's value proposition is "use a real programming language for your IaC". We already have a real programming language (Go) and the reason we want YAML is that authors are not writing Go for each deploy. Pulumi would be a regression on the author ergonomics we optimise for.

• Ansible for everything. Rejected. Ansible is imperative at its core — it is a very clever ssh-loop with templating. It has no persistent audit log, no run record, no idempotency contract beyond what each module ships with, and no interactive prompt semantics. We already use Ansible for service bootstrap on new hosts (setting up caddy, systemd, a non-root user, etc.); we continue to use it for that (bootstrapping a host once is imperative by nature). Day-to-day infra operations are not bootstrap and belong in the playbook runtime. See §14 for the WITHIN-HOST / ACROSS-FLEET boundary and §15 for the type: ansible step that bridges the two without inverting the orchestration direction.

• Manual Cloudflare UI forever. Rejected. Does not scale past one or two services. See §Context for the concrete failure modes. The argument for keeping it — "it's what we know" — is the argument we reject every time we adopt a new primitive.

• Direct Kubernetes controllers (Crossplane, ACK, etc.). Rejected. Crossplane/ACK solve this problem Kubernetes-natively: Cloudflare resources as custom resources, a controller that reconciles, a git-ops flow via Flux/ArgoCD. Excellent, if we were Kubernetes-first. Gyrum is not Kubernetes-first (ADR-020 commits to VMs + systemd + Caddy). Adopting Kubernetes for this specific problem means either running a control-plane cluster (all of the k8s operational overhead for one use case) or paying for a managed control plane (external dependency for an internal concern).

• A purpose-built gyrum-infra repo with hand-rolled scripts per surface. Rejected — this is the status quo slightly formalised. It gives us code-in-git (already partly true for Caddy) but neither the audit log, the idempotency contract, the dry-run, the interactive UX, nor the composition primitive. It is what we would build if ADR-068 did not exist. ADR-068 exists.

• A single "infra" monolithic playbook that does everything inline. Rejected as a Phase 4+ endpoint — but accepted as the Phase 1 shape (see §7). A flat deploy-service.yaml with inlined http + ssh steps is the right Phase 1 artifact because the subplaybook executor does not ship until ADR-068 Phase 4. Phase 1 monolithic → Phase 4 composed is the intended migration, not a sign that Phase 1 was wrong.

• Manage infra via a web UI in the Weft frontend (click-ops with audit). Rejected. A UI that mutates infra is a UI that has to own authorisation, scopes, audit, and idempotency — all the concerns that live in the playbook runtime. Building it in the UI is re-implementing the runtime in JavaScript. The UI's job (ADR-070) is to run playbooks; it is not a config tool. An operator who wants to click-deploy a service opens the Weft <PlaybookRunner> for deploy-service and fills the input form. That is the UI.

11. Consequences

Harder.

• Every infra change now needs a playbook. A new vendor surface (e.g. adding Fastly, adding a second cloud) is a new playbook (or set of them) before it is usable. This is discipline, not friction — the alternative is drift. But it is real work: a first-time-ever deployment on a new surface now takes longer than it would with curl. The payoff is every subsequent deploy.
• Playbook YAML becomes THE canonical infra spec. An operator who edits infra via the CF UI without also updating the playbook causes drift. At Phase 5 this is a CI concern (the playbook has drifted from state but nobody noticed until the next real run); at Phase 6 the reconciler either alerts or reverts. Until Phase 6, discipline is the enforcement. The operator's rule: "if you clicked it in the UI, you owe a PR to the playbook". We state this explicitly because otherwise the invariant breaks the first week.
• Author cost per sub-playbook. Eight sub-playbooks at ~300-400 lines of YAML each plus test coverage plus CI integration plus docs is roughly 8 follow-up PRs at ~600-800 lines each (within the 800-line ADR-067 PR cap). This is Phase 4 work and lands alongside ADR-068 Phase 4, not before.
• Scope-taxonomy maintenance. Cloudflare's scope names occasionally change (new scopes, renamed scopes). Our required_scopes: declarations are strings today; a CF rename breaks them. Mitigation: Phase 5's scope-verification step turns a rename into a clear first-run failure, not a silent production bug. Author updates the playbook; CI accepts.
• Dry-run diverges from apply occasionally. A playbook that passes dry-run can still fail at apply (race with another mutator, rate limit, CF API bug). Dry-run is not a guarantee; it is a higher-confidence preview. Operators have to know the difference.

Easier.

• Audit log is free. ADR-068's playbook_run_events table records every infra run with its actor, its inputs (hashed per ADR-069 §5), its duration, its outcome, and the committed playbook content hash. Six-months-later question "who changed the DNS?" is a SQL query, not an archaeology dig.
• Rollback cost drops dramatically. git revert the PR; re-run the playbook. Works for every surface in the catalogue, uniformly.
• One interactive UX across every infra surface. Operator opens <PlaybookRunner> for deploy-service, fills the form, watches the terminal pane stream, approves the destructive step when prompted, sees the before/after report. Same UX for DNS-only changes, for Caddy-only changes, for full deploys. New SRE training drops from "six vendor UIs and a shell history" to "open the runner and fill the form".
• Cross-vendor composition becomes trivial. Adding a Fastly step (hypothetical Phase 7 second CDN) is a new sub-playbook plus a new subplaybook: call in deploy-service. No re-architecture of the orchestrator.
• Infra tests become possible. Phase 6's reconciler runs every infra playbook in dry-run mode on a schedule. The aggregate changed: null|false report is a fleet-health KPI. A change that drifts infra makes the next scheduled run yellow.

What we sign up to maintain.

• The idempotency contract (§2.1). Every new infra playbook adheres. A sub-playbook that cannot be made idempotent (because the underlying API isn't — e.g. a create-only webhook) still has to declare the idempotency limit in its front-matter (idempotent: false) and pair with an ADR-069 idempotency-key guard. We write the amendment ADR if a recurring case emerges.
• The sub-playbook catalogue. The eight in §5 are the minimum. We expect to land a handful more over the next twelve months — docker-compose-up (for the stacks we still run via compose), cloudflare-waf-rule-upsert (once WAF becomes load-bearing), a second-CDN playbook if we adopt one. Each carries the four non-negotiables; each is a follow-up PR citing this ADR.
• The scope taxonomy. Vendor-scope strings (Zone:DNS:Edit, etc.) are tracked in a single dark-factory/playbooks/infra/ scope-taxonomy.yaml catalogue (follow-up; out of scope here). This is where Cloudflare scope renames land as PRs.
• The Phase 1 → Phase 4 migration. The monolithic deploy-service at Phase 1 is a debt we repay at Phase 4. Tracking issue: "Decompose deploy-service into subplaybook calls" on the dark-factory board, blocking Phase 4 exit.
• The GitOps discipline (§8). If we click the UI without owing a PR to the playbook, we betray the invariant and the reconciler (Phase 6) will churn. Operator muscle memory.

12. Migration path

Steps 1 is this PR. Steps 2-3 are in the same shipping cohort (Phase 1 value). Steps 4-6 fill ADR-068 Phase 4. Steps 7-10 are tracked follow-ups.

13. Review cadence

Re-review every 90 days (next: 2026-07-22) against:

• Sub-playbook coverage — are the eight in §5 shipped? Has a ninth or tenth been added? Have any been superseded?
• Idempotency violations — how many times has the reconciler (Phase 6) reported changed: true on a re-run? Each is a bug to file; the count is a trend KPI.
• Dry-run fidelity — how often has an apply failed after a successful dry-run? Target: rare. Rising trend is a signal to tighten the delta-compute step.
• Scope correctness — has the scope-verification step rejected any token as over-scoped or under-scoped? Each rejection is a good signal (the discipline is catching drift).
• Terraform pressure — has the escape hatch (§10) started to feel necessary? If we pass ~50 infra resources and the YAML authoring cost feels higher than HCL, we revisit.
• GitOps discipline — how many times has an operator clicked a CF UI without a matching PR? Each is a retro item for the next review.

Stale flag on the Owner dashboard once last_reviewed on this ADR passes 90 days.

References

• ADR-067 — playbooks unified primitive (the kind / front-matter / related conventions)
• ADR-068 — playbook runtime architecture (http, ssh, subplaybook executors, playbook_run_events audit stream); §14-§15 here extend its Phase-4 executor roster with type: ansible (the eleventh step kind) and name the bridge between ACROSS-FLEET Gyrum playbooks and WITHIN-HOST Ansible roles
• ADR-069 — playbook security model (authorisation, secret vault, destructive-step approval, audit chain)
• ADR-070 — playbook UI contract (<PlaybookRunner>, terminal pane, form-from-schema)
• ADR-071 — product name Weft; the "loom" metaphor (pattern → shuttle → pass → fabric) grounds why infra-as-playbooks reads as Weft's own primitive rather than a bolt-on
• ADR-064 — gyrum-catalog + hex arch; SSH Executor port backs every ssh step; host aliases (ops-vps, etc.) resolve through the catalog
• ADR-060 — catalog-driven infrastructure; service catalog entries reference the deploy-service playbook id that last deployed them
• ADR-053 — frontend observability; dry-run preview payloads stream through the same SSE channel as normal runs
• ADR-020 — shared infra vs per-product isolation; this ADR operates within the "shared infra" boundary (one CF account, one apex zone, one tunnel)
• ai-research/playbooks/hello-world.yaml — sample playbook referenced by ADR-068 §13 as the walking-skeleton template the infra sub-playbooks follow
• ai-research/playbooks/goodbye.yaml — second sample playbook; the ${inputs.x} and ${steps.y.z} reference syntax demonstrated there is the reference syntax the infra playbooks use
• Warp / warp.dev SECURITY.md — the per-command scoped-token pattern we borrow for the required_scopes: + secret_refs: front-matter shape (adapted, not copied)

14. Ansible boundary — WITHIN-HOST vs ACROSS-FLEET

Ansible came up twice already in this ADR — §10 ("Ansible for everything") as a rejected alternative, and §11 ("Consequences") as the tool we already use for host bootstrap. This section names the boundary explicitly, because the previous treatment conflates two questions: "should Ansible be the orchestrator?" (no, §10) and "should Ansible exist at all in Weft's world?" (yes, and here is where). Getting this wrong in either direction is expensive: replicate Ansible modules inside Weft playbooks and we ship a worse apt wrapper forever; ignore the boundary and a Weft author reaches for an inline ssh … apt install … step when a hardened role already exists.

14.1 The division

Two tools, two scopes. Gyrum playbooks and Ansible roles address different axes of the same problem. Draw the line through the host boundary: what crosses it and what does not.

                       ACROSS-FLEET                                 WITHIN-HOST
                     (Gyrum playbooks)                              (Ansible)
 ┌──────────────────────────────────────────┐           ┌──────────────────────────────────┐
 │  deploy-service                          │           │   roles/docker/                  │
 │    • Cloudflare DNS upsert ──────────────┼──(CF API) │   roles/postgres/                │
 │    • Cloudflare Tunnel route             │           │   roles/caddy/                   │
 │    • Cloudflare Access app/policy        │           │   roles/node_exporter/           │
 │    • Caddy site upsert ──────────────────┼───(ssh)───▶   roles/firewall/                │
 │    • systemd unit reload                 │           │   roles/cloudflared/             │
 │    • claude step (rendered commit msg)   │           │   roles/<per-service>/           │
 │    • approval step                       │           │                                  │
 │    • prompt / prompt_secret              │           │   tasks:                         │
 │    • http / ssh / subplaybook            │           │     - apt                        │
 │                                          │           │     - systemd                    │
 │  cross-service orchestration, API-level  │           │     - copy / template            │
 │  state convergence, human-in-the-loop,   │           │     - user / group / file        │
 │  audit, dry-run, approvals, scopes       │           │     - sysctl, ufw, seccomp       │
 └──────────────────────────────────────────┘           └──────────────────────────────────┘
              │                                                         ▲
              │                                                         │
              └──────── type: ansible step (§15) ───────────────────────┘
                       ansible-playbook --limit host --tags …

Read the diagram as two planes: ACROSS-FLEET sits above the host boundary and moves API state between services (Cloudflare, Caddy-on-another-host, a systemd unit on some host); WITHIN-HOST sits below the boundary and moves package / file / service state inside a single machine. Every call between the planes goes through the type: ansible step (§15) — there is no other bridge.

14.2 Which tool handles which class of task

The heuristic for an author deciding where a new task goes: if the task operates on state inside a single host and can be expressed as "install / configure / run this service, once, with these parameters", it is an Ansible role. If the task operates on state across multiple hosts or outside any host (an API, a DNS zone, a tunnel config), or it needs interactive UX (approval, prompt, claude), it is a Gyrum playbook.

14.3 The rule — Gyrum calls Ansible, never replicates it

Gyrum playbooks never replicate what Ansible modules do. They CALL ansible-playbook as a single type: ansible step (§15).

This rule is non-negotiable in the same sense as the four in §2. Violations are code-review and (post Phase-4) CI failures. The rule has two operational implications:

1. No inline apt in a type: shell step. A Weft playbook that writes run: sudo apt-get install -y postgresql is wrong even when it works. The idempotency story is weaker than ansible.builtin.apt; the error taxonomy is shell exit codes, not module-level structured results; the audit trail says ran a shell rather than applied role postgres.
2. No inline systemctl to enable a unit that a role would manage. ssh host 'sudo systemctl enable --now postgres' is fine for a reload after a config change the role just made (that is systemd-service-reload territory, §5.8). It is wrong for the initial enable of a unit whose installation is a role concern.

What a Gyrum playbook may still do at the host level, without going through Ansible: the narrow set of primitives ADR-072 committed to in §5 — systemd-service-reload, caddy-site-upsert (which edits the caller's Caddyfile fragment and reloads), ssh one-shots that observe rather than mutate (e.g. systemctl is-active). These are observe-and-nudge, not configure-from- scratch.

14.4 Worked example — wrong vs right

Wrong — inlining the package install in a Weft shell step:

# playbooks/infra/deploy-observ-stack.yaml  (fragment — DO NOT SHIP)
steps:
  - id: install_prom
    type: shell
    target: host:ops-vps
    run: |
      sudo apt-get update
      sudo apt-get install -y prometheus prometheus-node-exporter
      sudo systemctl enable --now prometheus
      sudo systemctl enable --now prometheus-node-exporter
      sudo tee /etc/prometheus/prometheus.yml <<EOF
      # ... 40 lines of yaml ...
      EOF
      sudo systemctl restart prometheus

What is wrong here:

• apt-get install is not idempotent in a useful sense. It returns 0 on both "installed now" and "already present", so changed: true/false is not recoverable. Ansible's apt module reports changed truthfully.
• tee is not atomic. A failed restart mid-tee leaves a half-written config. ansible.builtin.template writes to a temp file and renames.
• Package version drift. No pinning, no hold, no dpkg-lock retry. Ansible's apt has all three.
• Re-running is dangerous. A second run re-runs apt update + tee even when nothing changed — noisy in the audit log, and any transient apt mirror failure fails the whole playbook.
• The audit log reads ran a 12-line shell. Not "applied role node_exporter on ops-vps, changed=0".

Right — delegate the host-level work to an Ansible role:

# playbooks/infra/deploy-observ-stack.yaml  (fragment — correct)
steps:
  - id: install_observ_stack
    type: ansible
    target: local
    playbook: install-observ-stack.yml      # under $ANSIBLE_DIR
    limit: "ops-vps"
    inventory: prod.ini
    extra_vars:
      prometheus_version: "${inputs.prometheus_version}"
      node_exporter_version: "${inputs.node_exporter_version}"
    tags: [install, configure]
    diff: true
    become: true
    save_as: observ_install

  - id: register_metrics_route
    type: subplaybook
    playbook: service_runbook:cloudflare-access-app-upsert
    with:
      domain: "metrics.gyrum.ai"
      name: "prometheus"
      allowed_emails: "${inputs.sre_emails}"
      policy_name: "metrics-access"
      dry_run: "${inputs.dry_run}"
    save_as: access

The right shape leaves the host-level concerns (package install, config template, systemd enable + restart) inside the Ansible role and keeps the Weft playbook as the cross-fleet orchestrator. ansible/playbooks/install-observ-stack.yml references roles/prometheus and roles/node_exporter, which are apt-based, idempotent, check-mode clean. The audit log records applied install-observ-stack.yml on ops-vps, changed=2 (the two newly installed packages), and the structured step output (§15.3) gives the reconciler something meaningful to compare on the next run.

14.5 The Ansible role library — seven roles to grow

The rule in §14.3 is only fair if the roles exist. Committing to the rule commits us to growing the role library in step with the sub-playbook catalogue (§5). The initial seven:

These are the bootstrap-plus-baseline roles every ops-class host needs. Growth beyond seven is author-proposed via the same three-persona review the ADRs use (§8 review ladder); a new role is a PR to dark-factory/ansible/roles/<name>/ with tasks/ handlers/ defaults/ README.md.

14.6 Structured inventory discipline

Ansible's inventory is a first-class artefact, not a scratchpad.

dark-factory/
└── ansible/
    ├── ansible.cfg                         # inventory = inventory/prod.ini
    ├── inventory/
    │   ├── prod.ini                        # committed, canonical
    │   ├── staging.ini                     # committed, mirrors prod
    │   ├── group_vars/
    │   │   ├── all.yml                     # fleet-wide defaults
    │   │   ├── web.yml                     # web-facing VMs
    │   │   ├── db.yml                      # database VMs
    │   │   └── ops.yml                     # observability / tooling VMs
    │   └── host_vars/
    │       ├── ops-vps.yml                 # per-host overrides
    │       ├── db-01.yml
    │       └── web-01.yml
    ├── playbooks/
    │   ├── bootstrap-host.yml              # `roles: [firewall, cloudflared, node_exporter]`
    │   ├── install-observ-stack.yml        # `roles: [prometheus, node_exporter]` (on ops-vps)
    │   ├── install-caddy.yml
    │   ├── install-postgres.yml
    │   └── install-X.yml                   # one per service
    └── roles/                              # per §14.5

Rules:

• Three groups, not more. web, db, ops. Anything else is a tag or a host-var, not a group. Three groups keep the group-var explosion bounded; a fourth group is an ADR amendment.
• One host per line in inventory/*.ini. ansible_host
  • ansible_user explicit; no SSH-config magic. Matches the gyrum-catalog host aliases (ADR-064) — a host alias ops-vps in gyrum-catalog corresponds to an inventory entry ops-vps ansible_host=… ansible_user=ops one-for-one.
• No secrets in host_vars/ or group_vars/. Secrets go through the Weft vault (ADR-069 §4). host_vars/ops-vps.yml may reference cf_api_token: "{{ vault_cf_api_token_zone_dns_edit }}" where the right-hand side resolves at runtime from the Weft vault (bridged via extra_vars in the type: ansible step — §15.2). Ansible Vault is not used; we have one vault, the Weft vault, per ADR-069.
• Inventory is reviewed. A new host is a PR. The 3-persona review applies — Priya checks SSH exposure, Marcus checks group/host-var sanity, Lin checks role invocation.

14.7 Idempotency test — every role runs twice

The non-negotiable on the WITHIN-HOST side is the same as the §2.1 non-negotiable on the ACROSS-FLEET side: idempotency. We enforce it identically:

• First-run acceptance. A role applied to a clean host produces changed=N for some N >= 0 and exits 0. Ansible's default behaviour.
• Second-run assertion. Immediately re-applied, the same role on the same host must produce changed=0. Any non-zero changed count is a bug in the role.
• CI check. For each of the seven roles in §14.5, a CI job stands up a disposable VM (or container), applies the role, re-applies the role, asserts changed=0. The assertion reads the structured PLAY RECAP output (same shape §15.3 names) and fails the CI job on changed>0 or failed>0.
• Phase-4 integration. The CI assertion is the WITHIN-HOST analogue of the Phase-4 reconciler's delta.action == 'none' check (§8). Together they make the fleet self-describing: across-fleet drift = Weft playbook reports a non-empty delta; within-host drift = an Ansible role re-run reports changed>0.

The idempotency test catches the class of mistake that breaks config-management most often: a role that works once (installs a package, writes a config) but mutates on every subsequent run (adds another line to a config, rewrites a file with a new timestamp). A role that fails the second-run check is not merged.

14.8 What this section does not say

• Ansible is not the orchestrator. §10's rejection stands. Weft does not adopt Ansible as a top-level driver. Ansible is a callee of Gyrum playbooks, invoked via §15.
• We do not ship Ansible Tower / AWX. The Ansible we run is plain ansible-playbook from the CLI (or from the type: ansible executor, §15). No control-plane service, no separate UI. The UI for running Ansible is Weft's <PlaybookRunner> — the same as every other step type.
• We do not write custom Ansible modules. Upstream ansible.builtin + community.general + community.docker cover every WITHIN-HOST primitive the seven roles need. A hypothetical custom module is an ADR amendment.
• We do not use Ansible Vault. One vault (Weft's), one secret-resolution path (${vault.X} / ${env.X}). Vault is bridged into Ansible via extra_vars on the type: ansible step — §15.2.

15. type: ansible step executor — Phase 4 runtime addition

§14 named the boundary. This section names the bridge. ansible becomes a first-class step type in the ADR-068 runtime, alongside shell, ssh, claude, http, prompt, prompt_secret, approval, pipeline, experiment, and subplaybook. That is eleven executor kinds in the Phase 4 runtime, and ansible is the eleventh.

15.1 Why first-class, not shell-wrapped

An obvious cheaper alternative is to let authors write a type: shell step that runs ansible-playbook …. We reject that alternative for the same reasons §14 rejects inline apt: the contract the executor gives us is load-bearing. Specifically:

1. Parse-time validation of playbook + limit. A shell step's run: is an opaque string; parse-time can only syntax-check it. The type: ansible step has typed fields, so the validator can check: playbook resolves to a file under $ANSIBLE_DIR; limit resolves to a host or group in the declared inventory; tags match tags declared in the playbook; extra_vars keys are expected inputs. A typo in limit: web-o1 (oh-one vs zero-one) fails at CI, not at the fourth step of a deploy.
2. Structured output for idempotency assertions. §14.7's CI check and §8's drift reconciler both want the same structured summary: changed, failed, unreachable. A type: shell step emits stdout bytes; the caller would have to parse PLAY RECAP with a regex. The type: ansible executor parses it once, in Go, and hands callers a typed struct via ${steps.X.changed} / ${steps.X.failed}.
3. Per-task run-log events. The Phase-2 SSE stream (ADR-068 §6) wants fine-grained progress. ansible-playbook --stdout-callback=json emits one JSON event per task; the executor re-emits those as step_progress events with {task: "…", host: "…", status: "ok|changed|failed", duration_ms: N}. A shell-wrap would collapse them into a single stdout blob.
4. Typed downstream references. A downstream step can write when: "${steps.install_X.changed > 0}" or when: "${steps.install_X.unreachable == 0}" and have the expression engine evaluate integers, not strings. The claude step composes with this — ${steps.install_X.per_host} passed as prompt context gives Claude a structured input to render a release note from.

The cheap alternative gets us some of (1) — a shell step can at least check the command is syntactically valid — but none of (2-4). First-class wins.

15.2 Full executor specification

steps:
  - id: deploy_X
    type: ansible
    target: local                     # runs `ansible-playbook` on the worker; the playbook SSHes fan-out
    playbook: install-X.yml           # path relative to $ANSIBLE_DIR (dark-factory/ansible/playbooks/)
    limit: web-01                     # --limit; host, group, or comma-separated list
    inventory: prod.ini               # optional; default is ansible.cfg's inventory
    extra_vars:                       # --extra-vars; passed as JSON (preserves types)
      version: "${inputs.version_tag}"
      emails: "${inputs.allowed_emails}"
      cf_api_token: "${vault.cf_api_token_zone_dns_edit}"   # vault bridge (§14.6)
    tags: [deploy, verify]            # --tags
    skip_tags: [debug]                # --skip-tags (optional)
    check: false                      # --check; Ansible's dry-run (default: false)
    diff: true                        # --diff; show template/config deltas (default: true)
    become: true                      # --become; sudo on target (default: false)
    become_user: root                 # --become-user (default: root when become: true)
    forks: 5                          # --forks; parallelism across hosts (default: 5)
    timeout: 600s                     # executor timeout; kills ansible-playbook on overrun
    save_as: ansible_result           # ${steps.deploy_X.*} downstream

Field semantics:

• target: local — the ansible executor always runs on the Weft worker. The worker has the SSH key, the inventory file, and the role library on disk; the fan-out happens inside ansible-playbook. target: host:ops-vps is rejected at parse time (nonsensical — Ansible would then need its own Ansible to reach each host).
• playbook: — required. Resolves relative to $ANSIBLE_DIR (defaulted to dark-factory/ansible/playbooks/ at runtime startup; see ADR-068 §3 runtime config). Parse-time validator confirms the file exists; CI validator confirms it parses via ansible-playbook --syntax-check.
• limit: — optional but strongly recommended. If absent, the playbook's own hosts: declaration governs — which is usually all or a group, and usually too broad for safety. Authors default to a narrow limit:; broad limits require a comment explaining why.
• inventory: — optional. Defaults to the inventory in ansible.cfg. Explicit override supports staging/prod switches (inventory: staging.ini in a dry-run variant).
• extra_vars: — a map, JSON-serialised when passed to --extra-vars. This is the vault-bridge: a ${vault.X} reference is resolved by the Weft vault resolver (ADR-069 §4) before the JSON is built, so the token is in Ansible's process environment only, never on the command line. stdout scrubbing (§2.3, ADR-069 §4.3) extends to Ansible's stdout.
• check: / diff: — Ansible's dry-run surface. check: true is the natural bridge for the playbook-level dry_run: true input at §2.2: the top-level playbook sets check: "${inputs.dry_run}" on every type: ansible step, so a dry-run deploy runs Ansible in check mode on every host. diff: true is almost always desired — it surfaces template + copy deltas in the per-task event stream.
• become: / become_user: — Ansible's sudo control. Defaults to become: false, deliberately: the author opts in. A role that needs root says so loudly.
• forks: — parallelism. Small by default (5) because fan-out to 5+ hosts is the rare case in our fleet size. Bumps require a comment.
• timeout: — executor-level; distinct from Ansible's own --timeout (which is per-SSH-connection). On expiry the executor SIGTERMs ansible-playbook, captures partial state, emits step_failed with {reason: "executor_timeout", per_host: {…partial…}}.

Forbidden fields (parse-time rejection):

• ansible_vault_password_file: — Ansible Vault is not used; see §14.8.
• target: host:* — see above.
• run: / command: — this is not a shell step; the playbook is the unit of execution.
• Literal passwords or tokens in extra_vars — same entropy/pattern check as §2.3.

15.3 Structured output — step_completed body

On success, the executor emits a step_completed event whose outputs field is:

{
  "host_count": 1,
  "changed": 3,
  "failed": 0,
  "unreachable": 0,
  "ok": 14,
  "skipped": 2,
  "per_host": {
    "web-01": {
      "ok": 14,
      "changed": 3,
      "failed": 0,
      "unreachable": 0,
      "skipped": 2,
      "rescued": 0,
      "ignored": 0,
      "tasks": [
        {"name": "install postgres", "status": "changed", "duration_ms": 1821},
        {"name": "ensure postgres service running", "status": "ok", "duration_ms": 112}
      ]
    }
  },
  "stderr_tail": "...",
  "playbook": "install-X.yml",
  "limit": "web-01",
  "tags": ["deploy", "verify"],
  "check_mode": false,
  "duration_ms": 18243
}

Field contract:

• host_count — number of hosts in the resolved --limit. Driven from Ansible's JSON callback, not re-derived client-side.
• changed / ok / failed / unreachable / skipped — fleet-wide sums across all hosts. These are the primary signals for downstream when: expressions.
• per_host — map of host → per-host stats. Matches Ansible's PLAY RECAP per-host counts exactly, plus a tasks array for the per-task trace when --stdout-callback=json was used. tasks is capped at 500 entries; overflow is elided with a tail note in stderr_tail.
• stderr_tail — last 4KB of stderr, newline-trimmed. For failure-mode debugging without streaming the full stderr into playbook_run_events.body (ADR-068 §5 event-size cap).
• playbook / limit / tags / check_mode — echo-backs of the step's inputs, for audit-time reconstruction without reading the step definition.
• duration_ms — executor wall-clock.

Failure variants:

• Ansible exits non-zero but produced a recap. Emit step_failed with the same shape; failed and unreachable are non-zero and name the offending hosts.
• Ansible did not start (missing inventory, syntax error). Emit step_failed with {reason: "ansible_startup", stderr_tail: "…"}; no per_host.
• Executor timeout. Emit step_failed with {reason: "executor_timeout", per_host: {…partial…}} as above.

Downstream when: composability:

steps:
  - id: install_X
    type: ansible
    playbook: install-X.yml
    limit: "${inputs.target_host}"
    save_as: install_X

  - id: cloudflare_dns
    type: subplaybook
    when: "${steps.install_X.changed > 0}"          # only register DNS if the unit actually rolled
    playbook: service_runbook:cloudflare-dns-upsert
    with:
      name: "${inputs.target_host}"
      content: "${inputs.tunnel_id}.cfargotunnel.com"
      dry_run: "${inputs.dry_run}"

  - id: announce_release
    type: claude
    when: "${steps.install_X.failed == 0 && steps.install_X.changed > 0}"
    prompt: |
      Render a release note for version ${inputs.version_tag}.
      Per-host changes: ${steps.install_X.per_host}
      Task summary: changed=${steps.install_X.changed},
      ok=${steps.install_X.ok}, skipped=${steps.install_X.skipped}.

This is the payoff of first-class structure: the downstream steps read typed values, not parsed stdout; the claude step composes the structured trace directly into its prompt.

15.4 Idempotency test — runtime-asserted

The executor supports an assertion mode for the CI idempotency check (§14.7). Setting assert_idempotent: true makes the executor run the playbook twice and fail the step unless the second run reports changed == 0 && failed == 0 && unreachable == 0:

- id: verify_role_idempotency
  type: ansible
  playbook: install-observ-stack.yml
  limit: ci-disposable
  inventory: ci.ini
  become: true
  assert_idempotent: true                     # ADR-072 §15.4
  save_as: idem

On pass, outputs.second_run is attached with the same shape as §15.3; outputs.idempotent: true. On fail, step_failed with {reason: "not_idempotent", first_run: {…}, second_run: {…}}. This is the machine check §14.7 describes; the flag is off by default because a normal deploy only wants one run.

15.5 Phase-4 placement — alongside ssh / http / claude

The type: ansible executor lands in ADR-068 Phase 4, the phase that also carries subplaybook, http, and (per §14 here, newly enumerated) the refinement of ssh, claude, prompt, prompt_secret, and approval to their Phase-4 contracts. The executor roster at the end of Phase 4 is:

1. shell — already in Phase 1.
2. ssh — already in Phase 1; hardened in Phase 4.
3. http — Phase 4.
4. claude — Phase 4.
5. prompt — Phase 4.
6. prompt_secret — Phase 4.
7. approval — Phase 4.
8. subplaybook — Phase 4 (also gates the §7 Phase-4 exit criteria for this ADR).
9. pipeline — Phase 4.
10. experiment — Phase 4.
11. ansible — Phase 4 (this ADR, §14-§15).

Phase 4 is the "complete the runtime" phase. ansible does not introduce new infrastructure (no new queue, no new SSE topic, no new event shape beyond what step_completed already supports); it is another executor registered against the same runtime. The implementation is an executor that shells out to ansible-playbook, parses the JSON callback, and emits the structured events. The role library (§14.5) grows in parallel but on its own schedule — a Phase-4 exit requires only the executor and at least one first-party role under roles/bootstrap/ so that the executor has something to run against in integration tests.

15.6 Generalisation — the reconciliation-tool executor shape

type: ansible is the first of a class. Two obvious future siblings, not committed to here, are:

• type: terraform — target: local, module:, workspace:, variables:, auto_approve: false, structured plan output via -json, structured apply output via -json. Same shape: shell out to a reconciliation tool, parse its structured output, emit typed step results. §10's escape hatch (Terraform for very large resource counts) lands as a type: terraform executor, not as a Terraform-calls-Weft inversion.
• type: pulumi — target: local, stack:, program:, config:, preview: bool, structured output via --json. Same shape again.

The pattern the three share — Gyrum invokes a reconciliation tool; the tool operates on its own scoped state; Gyrum parses the structured summary; downstream references read typed fields — is what makes type: ansible first-class rather than shell-wrapped. The executor class name in the runtime taxonomy is reconciliation_tool; ADR-068's Phase-4 executor registry can accept a plugin contract so terraform / pulumi can be added without an ADR-068 amendment, only an ADR-072 amendment per new tool.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one