← Decisions operate mode →

Decisions

ADR-087: Cloud-init for host bootstrap instead of post-provision SSH harden

Every cloud VM provisioned by a gyrum pipeline carries a cloud-init `user-data` payload that handles package install, ops-user creation, firewall rules, and systemd service enablement at first boot. The post-provision…

#087

ADR-087: Cloud-init for host bootstrap instead of post-provision SSH harden

Status: Accepted Date: 2026-04-24 Related: ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-084 (outbound-call chokepoint), ADR-085 (typed I/O ports)

Decision (one paragraph)

Every cloud VM provisioned by a gyrum pipeline carries a cloud-init user-data payload that handles package install, ops-user creation, firewall rules, and systemd service enablement at first boot. The post-provision SSH-based harden step is removed; cloud-init status --wait becomes the readiness gate. The pipeline renders a cloud-init YAML from declared inputs (service role, ssh keys, firewall profile), writes it to a tmpfile, and passes it to hcloud server create --user-data-from-file …. Cloud-init runs the harden inline on the host before the first SSH-as-ops connection; what used to be three stringly-coupled steps (provision → wait_ssh → harden) collapses into one atomic provision_and_bootstrap step whose only failure modes are "the API call failed" or "cloud-init reports failed status".

Context

Tonight's incident — UFW heredoc bug in the harden step

Tonight's real run of provision-and-deploy-warp.yaml (see today's session log) failed inside the harden step. The step invokes ssh root@host 'bash -s' << REMOTE … REMOTE and inside the remote bash invokes ufw --force allow 22/tcp with shell-escaped substitutions. Three layers of shell escaping (local heredoc, ssh's argv concatenation, the remote bash's parsing) interact unpredictably; one expansion that looked correct in the source mis-rendered on the wire and ufw rejected the rule. The provisioning succeeded, the SSH wait succeeded, the deploy succeeded — but the firewall was wide open because the harden step's failure was non-fatal in the way it was being invoked.

This is a class-of-bug, not an individual mistake. Heredoc-over-SSH is a stack of three stringly-typed parsers (the local shell, the ssh transport, the remote bash) and every one of them has its own escaping rules. There is no way to validate the composed command at design-time; the only test is "does it run on a real host without a typo", and a typo only surfaces on the run that triggers it.

The three-step → one-step collapse

Today's provision-and-deploy-warp.yaml shape, steps 2–4:

flowchart TB
    subgraph before["BEFORE (3 steps, 2 sync barriers)"]
        prov["step 2: provision<br/>hcloud server create<br/>→ IP"] --> wait["step 3: wait_ssh<br/>ssh root@IP 'true'<br/>poll 5s × 30 (≤150s)"]
        wait --> harden["step 4: harden<br/>ssh root@IP bash -s<br/>&lt;&lt;HEREDOC ufw + apt + systemctl HEREDOC<br/>brittle: 3-layer shell escape"]
    end
    subgraph after["AFTER (1 step, 1 sync barrier)"]
        render["step 1: render_cloudinit<br/>template + inputs → /tmp/*.yaml"] --> bootstrap["step 2: provision_and_bootstrap<br/>hcloud server create --user-data-from-file …<br/>ssh root@IP 'cloud-init status --wait'<br/>declarative: parsed by cloud-init on host"]
    end

Three steps with two synchronisation barriers (was-the-VM-up?, did-the-harden-succeed?) collapse into one step with one barrier (did-cloud-init-finish-cleanly?). The harden moves from "fragile heredoc executed by us over an SSH transport" to "declarative YAML executed by the host's own cloud-init subsystem". The new render_cloudinit step is bookkeeping (read template, substitute, write tmpfile) and runs locally — it cannot fail on network or remote-shell concerns.

Hetzner's native cloud-init support

Hetzner Cloud's server create API accepts a user-data field on creation; the hcloud CLI exposes it as --user-data-from-file. The VM's cloud-init runs the payload before the user's first SSH lands. This is not gyrum-specific: AWS EC2, Azure, GCP, DigitalOcean, and OpenStack all accept the same cloud-init format on instance creation. Choosing cloud-init is choosing the cloud-portable bootstrap mechanism over a hcloud-bespoke SSH chain.

Deterministic-vs-polling tradeoff

Today's wait_ssh is a polling loop (try-true-every-5s) with a hard 150-second cap. It is a heuristic: SSH-up is a proxy for "the box is ready", but it lies in both directions — sshd can be up before package installs finish, and a slow-imaging VM can take longer than 150s to surface. Cloud-init's status command is a true ready signal: cloud-init status --wait blocks until cloud-init reports either done or error, with no false positive and no arbitrary timeout. We trade a polling bound for a deterministic exit signal.

Template-as-input pattern

The pipeline does not compose cloud-init YAML by string concatenation. It reads a checked-in template, substitutes a small set of declared inputs, and writes the rendered file to /tmp for one-shot consumption. The template lives in dark-factory/ansible/cloud-init/<service>.yaml; warp's lives at dark-factory/ansible/cloud-init/warp.yaml. Substitutions are limited to: service role (string, picks the per-service overlay), SSH authorised keys (read from ~/.ssh/id_ed25519.pub at render time), and firewall profile (one of web, internal, db, encoded as a port allowlist).

The render step is itself a playbook step; its inputs are typed (ADR-085). The output is a path to a tmpfile that the next step consumes via --user-data-from-file. Files are render-then-read; they are never edited in place.

Readiness gate

After hcloud server create returns an IP, the pipeline runs ssh -o StrictHostKeyChecking=accept-new root@IP 'cloud-init status --wait; cloud-init status --long'. The --wait flag blocks until cloud-init transitions out of running. Exit code 0 means done; non-zero means error or degraded. The --long follow-up captures the human-readable status into the step's stdout for when a failure needs forensics. There is no separate wait_ssh polling — the cloud-init status command IS the wait, and SSH-up is implicit because cloud-init runs after sshd is healthy enough for the call to land.

What cloud-init can't do well — and where ansible still takes over

Cloud-init is first-boot single-host configuration. It is the wrong tool for: multi-host coordination (cloud-init does not see the rest of the fleet); vault-encrypted secrets (user-data is plaintext to anyone with hcloud API read access); compose stack lifecycle (cloud-init runs once, never again, so a stack restart is not a cloud-init concern); anything requiring the host catalog (cloud-init does not query the gyrum catalog, it has no ambient knowledge of which other hosts exist); and templating that requires cross-host data (the postgres host's IP, the load-balancer's name).

Ansible owns ongoing cross-host state. The split is: cloud-init = "make this fresh box look like a baseline gyrum host", ansible = "make this gyrum host run service X at version Y, wired to the rest of the fleet". The handover point is /etc/gyrum/onboarded — cloud-init writes the marker, ansible's install-warp.yml precondition checks for it.

Security notes

User-data is visible to anyone with hcloud API read access (hcloud server describe --output yaml includes the rendered user-data verbatim). Treat the rendered template as plaintext: never embed secrets, vault passphrases, API tokens, postgres passwords, or anything an HCLOUD_TOKEN scope leak would expose. Public SSH keys are fine — they are public. Vault-encrypted secrets remain ansible-side, fetched at deploy time from dark-factory/ansible/group_vars/<group>/vault.yml against an ansible-vault passphrase the operator supplies. The boundary mirrors the cloud-init/ansible split above: cloud-init carries only what is safe-to-leak; ansible carries everything that is not.

A second concern is template tampering. The cloud-init template is checked-in at dark-factory/ansible/cloud-init/<service>.yaml — any drift between the working-copy and the upstream main is a code-review concern. Phase 3's typed provider fingerprints (sha256s) the rendered template per run so a runtime-injected modification is detectable in the run record; until Phase 3, the operator's git diff before push is the only gate.

Consequences

Positive

  • The class-of-bug that hit tonight (heredoc/ssh shell-escaping) goes away. Cloud-init YAML is parsed by cloud-init's own parser on the host; there is no shell composition layer.
  • One fewer step in every host-provisioning playbook — provision-and-deploy-warp.yaml collapses 9 → 6 steps.
  • Cloud-portable: the same template shape works on AWS, Azure, GCP, DO. Tonight's hcloud-specific heredoc would have to be re-written if we ever moved a workload to AWS.
  • Deterministic readiness: cloud-init status --wait replaces a 150-second polling cap, removing a class of race conditions where a slow-imaging VM was declared dead.
  • The harden definition becomes declarative YAML diffable in code review, instead of a heredoc string the reviewer has to mentally execute through three shell layers.

Negative

  • Cloud-init runs once. To re-harden an existing host (e.g. add a new firewall rule across the fleet), the operator runs ansible against inventory.ini — there is no "re-render the template and re-apply" path. This is fine for the harden which is set-once-at-boot, but it is a real constraint that means cloud-init cannot replace ansible for ongoing fleet management.
  • Template drift across services becomes a maintenance concern. If warp.yaml and a future buzzy.yaml diverge on the ops-user definition, two services with subtly different baselines is a debugging hazard. Mitigation: Phase 2 introduces a shared dark-factory/ansible/cloud-init/base-ubuntu-24.yaml with service-specific overlays.

Risks

  • Cloud-init failures are debuggable but unfamiliar. The first real failure will require the operator to know where cloud-init logs (/var/log/cloud-init.log, /var/log/cloud-init-output.log) and how to read its module reports. Mitigation: the playbook's provision_and_bootstrap step always runs cloud-init status --long at the end; any failure surfaces the cloud-init module name + error directly in the playbook stdout.
  • Hetzner's user-data has a hard size cap (we hit 16 KB per the Hetzner docs). A bloated template is a deploy-time failure. Mitigation: keep templates lean — the warp template lands at ~30 lines; if a template ever exceeds 8 KB we factor the bulky bits to ansible.

Phases

Phase 1 — Cloud-init template for warp (this ADR)

A single template at dark-factory/ansible/cloud-init/warp.yaml, consumed by provision-and-deploy-warp.yaml via a render_cloudinit step that substitutes server_name and the operator's SSH public key. The harden step is removed; the firewall + ops-user + systemd enablement live in the template's runcmd: and users: sections. Validates the pattern on one service before generalising. ~1 day of work.

Phase 2 — Shared template library

Extract the common bits (ufw default rules, ops-user shape, fail2ban + unattended-upgrades) to dark-factory/ansible/cloud-init/base-ubuntu-24.yaml. Per-service templates (warp.yaml, buzzy.yaml, postgres.yaml) become thin overlays that merge: the base. A linter runs on PRs touching the templates to catch drift between the base and per-service overlays. Lands when the second service (likely buzzy) onboards via cloud-init. ~3 days.

Phase 3 — Cloud-init provider in the outbound chokepoint

A typed cloud-init provider in either gyrum-go/pkg/outbound/cloudinit/ or gyrum-go/pkg/outbound/ansible/ (TBD by ADR-084 maintainers). Validates the YAML against the cloud-init schema before write, fingerprints the rendered template (sha256 of base + overlay + substitutions) for drift detection across runs, and emits typed outputs (template_version, render_path) per ADR-085. The pipeline step then becomes a typed block instead of a shell wrapper. ~1 week, scheduled after Phase 2 settles the template shape.

Related

  • ADR-072 — infrastructure as playbooks. This ADR is a refinement of how a "provision a host" playbook is shaped, not a departure from the playbook-as-primitive design.
  • ADR-076 — project-to-host binding. The /etc/gyrum/onboarded marker cloud-init writes is what register_inventory checks before binding the project to the host.
  • ADR-084 — outbound chokepoint. Phase 3's typed cloud-init provider lives in the chokepoint, alongside the hcloud, cloudflare, and ansible providers.
  • ADR-085 — typed I/O ports. The render_cloudinit and provision_and_bootstrap steps are early candidates for typed-block migration once the provider in Phase 3 lands.

Motivating session — see dark-factory/docs/sessions/2026-04-24.md for the heredoc-bug shipping context that surfaced this ADR.