ADR-148: Bootstrap stages are independently re-runnable, not transactional

Status: Accepted Date: 2026-04-21

Context

Server bootstrap scripts historically came in two shapes:

One-shot imperative scripts (see infrastructure/init-vps.sh — 300-ish lines, runs top to bottom, can't be partially rerun). They work exactly once per host. When stage 6 fails, you either start over (destroying stages 1–5's state) or hand-fix the host outside the script.
Configuration-management runs (Ansible, Puppet). Idempotent by construction; expensive to debug when something drifts because the declarative layer hides what's actually changing.

The gap we keep hitting: a stage fails mid-bootstrap, we ssh in, fix it by hand, and the script never gets the chance to verify the fix. Next operator finds a host that looks bootstrapped but has hand-edits the bootstrap kit doesn't know about.

The infrastructure/server-bootstrap/ kit is the replacement. Constraints:

We don't want a full CM dependency (no Ansible on the operator's laptop, no pull-mode agent on the VPS).
We need to resume after a partial failure without manual state tracking.
Operators need to be able to run any single stage on demand — e.g. re-apply the UFW rules after someone fat-fingered ufw disable.
A new operator reading any stage script should understand what it does without loading the whole kit into their head.

Decision

Every stage under infrastructure/server-bootstrap/stages/ is a standalone bash script that:

Checks its own precondition before acting. docker network inspect $NET >/dev/null before docker network create, dpkg -s PKG before apt-get install, cmp -s src dst before install.
Emits === STAGE: <Name> === on stdout (ADR-029) so progress shows up in the Platform UI whether it's run standalone or by bootstrap.sh.
Respects DRY_RUN=1 as an env var. The top-level bootstrap.sh --dry-run exports it; standalone runs set it explicitly.
Exits non-zero on any hard failure, leaving no marker in /deploy/.bootstrap-state. bootstrap.sh --resume re-runs from the first stage missing its marker.
Makes one thing true. A stage is scoped to one concern (hardening, docker install, secrets). No "install docker AND configure UFW AND seed Grafana" megastages — they're the scripts that become impossible to rerun safely.

State is a plain append-only file. Each successful stage records <NN-name> <iso8601>. Failure leaves the file untouched. The file is on /deploy/.bootstrap-state so it survives OS reinstalls of /root but is lost if the whole disk is wiped — which is the right failure mode (a blank disk should re-run every stage).

Consequences

Hand-fixes become rare. If a stage keeps failing, the fix goes into the stage script, not the host. The host stays a cattle, not a pet.
Rerunnable => testable. We can run bash stages/04-networking.sh under a throwaway LXC container and assert the expected post-state. Each stage is the unit of test.
Precondition checks are code we own. Every stage grew a handful of if already-done; then skip branches. That is code — it has bugs, it gets stale. We accept this as the cost of rerunnability.
Some things can't be idempotent-cheap. Examples:
- ufw --force reset in stage 04: cheaper to rebuild the rule set than to diff existing rules against desired. Documented side effect is "external UFW rules are wiped."
- docker run in stage 09 (monitoring agents): stop+rm+run emulates idempotence; any in-flight query to cadvisor briefly fails on rerun.
Both are called out in the kit's README.
State file drift is possible. Someone edits /deploy/.bootstrap-state to "skip" a stage — nothing in the kit prevents that. We trade drift risk for simplicity; a real configuration-state database is the next-order-of-magnitude problem.
No rollback. Stages are forward-only. Once 02-os-hardening.sh has disabled password SSH, re-enabling it means editing the drop-in file or writing an inverse stage. We are explicit: the bootstrap kit sets up; it does not tear down.

Alternatives considered

Ansible playbook. Idempotent by design, widely understood. Rejected on cost of entry: every operator needs Ansible on their laptop and a working inventory file, and debugging a playbook means reading through modules instead of the actual shell commands the host will execute. Bash-with-discipline matches the "script-first culture" the rest of dark-factory/infrastructure/ already uses.
Makefile targets. Natural idempotence via "target newer than deps." Rejected because most of what we do has no filesystem artifact (docker networks, ufw rules, systemd reloads) — shoe-horning them into Make's file-timestamp model loses clarity.
Single monolithic init-vps.sh (the status quo). Rejected: the failure-resume story is the problem we're solving.
Declarative host manifest (NixOS-style). Solves all of the above at an order-of-magnitude cost we can't afford: we'd rebuild our entire infra pattern on a single tool's assumptions.

Supersedes: none Superseded by: