ADR-060: Catalog-driven infrastructure (capture + compose)

Status: Accepted Date: 2026-04-22 Related: ADR-058 (5-mode Platform UI), ADR-059 (project-first IA) Followed by: ADR-061 (extract pipelines + projects to standalone libraries)

Context

Two engines already live in gyrum-labs/ai-research:

• Experiment runner (pipeline_experiment.go:91 runExperiment) — invokes numbered experiments at experiments/exp-N-slug/ as subprocesses, sets EXP_DIR and any declared NeedsInput env var, captures stdout, watches for progress strings, writes run-report.json. ~314 experiments exist today across categories: business, blueprint, extraction, generation, analysis, verification, infrastructure, proposals.
• Pipeline engine (cmd/server/pipeline/) — composes reusable Blocks registered in blocks_builtin.go. Each Block has a stable ID (group:name), Phase, Model tier, Version, prompt template (or custom function), Inputs/Output declarations. Pipelines are JSON in pipelines/ that reference blocks by ID. The runner supports RepeatFrom for layered loops (Plan → Dev → PR → Dev → PR → …). Retro feedback targets a Block by ID and auto-archives when its Version bumps.

What is missing from this otherwise capable stack:

• Discovery: when an operator points at a new server (or new GitHub repo), there is no automated way to capture what is actually running on it. The Platform UI shows what we think runs there based on docker-compose / Prometheus targets / project YAML; reality may have drifted.
• Provisioning replay: no way to recreate a known-good server elsewhere from captured state. The architecture lives only in the operator's head and a partial collection of compose files.
• Drift detection: server state today vs yesterday is invisible. Drift is discovered when something breaks, not before.

The operator's stated vision (2026-04-22 design conversation): catalog-driven infrastructure. Stage 1 captures state from any target into JSON. Stage 2 consumes that JSON to provision matching infrastructure elsewhere. The catalog is the contract.

Decision

Adopt a two-layer architecture for infrastructure orchestration. The two layers correspond exactly to the two existing engines.

Layer 1 — Capture (experiments)

Single-purpose probes that produce structured JSON catalog data. Each is a numbered experiment in experiments/exp-N-slug/, invoked by the existing runner. Plug-in scripts live alongside in plugins/; the entry script orchestrates them with non-fatal failure semantics — a single failing probe does not abort the whole audit.

Initial capture experiments (numbers reserved):

Each is self-contained, scriptable, runnable from CLI without the platform. Plug-in shape:

experiments/exp-315-hetzner-server-audit/
├── README.md                    # Documents inputs, outputs, plug-in catalog
├── audit-server.sh              # Entry — orchestrates plug-ins, aggregates
└── plugins/
    ├── 010-docker-containers.sh # docker ps → JSON
    ├── 020-listening-ports.sh   # ss -tlnp → JSON
    ├── 030-systemd-services.sh  # systemctl → JSON
    ├── 040-disk-usage.sh        # df + du → JSON
    ├── 050-installed-packages.sh
    └── 060-reverse-proxy-config.sh

Numeric prefixes lock execution order without making the entry script hardcode plug-in names. Adding a new plug-in is dropping a file in plugins/; the entry script picks it up by glob.

Layer 2 — Compose (pipelines)

Multi-block flows that consume capture-layer outputs. Defined as JSON in pipelines/, executed by the existing pipeline engine. Reusable blocks register in cmd/server/pipeline/blocks_builtin.go.

Initial blocks (registered in audit: and provision: namespaces):

Initial pipelines:

• pipelines/server-provision-from-catalog.json
  • Sequence: audit:capture(source) → provision:emit-bootstrap → provision:dry-run → provision:apply → audit:capture(target) → audit:verify-match
  • The terminal verify-match is the contract: provisioning success means the new target's audit matches the source's audit.
• pipelines/server-drift-alert.json
  • Sequence: audit:capture(server) → audit:diff(against baseline) → if-non-empty → record-finding
  • Scheduled (cron from outside the engine) to run nightly per server.
• pipelines/fleet-snapshot.json
  • Loops audit:capture per known server (sourced from dark-factory/projects/*.yaml's server list when ADR-059's runtime fields land), aggregates into fleet.json, archives.

Catalog contract

Catalog files match a documented schema (see docs/specs/catalog-spec.md). The shape is stable across capture experiments — every catalog has the same envelope, only the per-plug-in result shapes vary.

{
  "schemaVersion": "1.0",
  "target": { "type": "hetzner-server", "id": "cax11.gyrum.io" },
  "capturedAt": "2026-04-22T22:30:00Z",
  "capturedBy": "exp-315-hetzner-server-audit",
  "plugins": [
    { "name": "docker-containers", "ok": true,  "durationMs": 142, "result": [/* ... */] },
    { "name": "listening-ports",   "ok": false, "error": "ss not installed" }
  ]
}

Plug-in result shapes are documented per-plug-in in their experiment's README.md. The envelope is universal; the content is plug-in-specific.

Boundary rules

These four rules prevent the architecture from drifting:

1. Catalogs never contain secrets. Capture-layer plug-ins capture structure (image names, port numbers, file paths, package names) — never values (env values, certificates, license keys, tokens). Provisioning emits a secrets.env.template listing required values; the operator fills it in out of band. CI lint should grep committed catalogs for common secret patterns and fail on hits.

2. Plug-in independence. Each capture-layer plug-in runs in isolation. A plug-in failure produces {ok: false, error: "..."} in its slot; the aggregate run continues. The catalog is "best effort by plug-in", not "all-or-nothing". This is the operator's core plugins are all independent rule from the design conversation.

3. Experiments produce, pipelines consume. Capture-layer experiments do not orchestrate other experiments. Composition lives in pipelines. This keeps capture experiments unit-testable in isolation, and pipelines testable as composition without touching real targets.

4. Schema versioning. Adding a new plug-in is non-breaking (new slot in the plugins array). Removing a plug-in or changing its result shape is a schemaVersion bump — and consumers (Layer 2 blocks) declare the schema version they accept.

Catalog lifecycle

A catalog has a small lifecycle once captured:

Storage (initial): JSON files in dark-factory/catalogs/<target-type>/<target-id>/<timestamp>.json, committed to git for audit trail. Promote to a SQLite/Postgres store when capture frequency exceeds daily-per-target.

Where this fits in the Platform UI

ADR-058 gives us five modes; ADR-059 gives us project-first IA. Catalog surfaces:

• Infra mode (ADR-058's infra palette) gets the catalog UI: an "Audit" tab on each server detail page, a fleet-wide "drift" surface, and a per-target catalog history.
• Project pages (ADR-059) get a "Catalog" tab when a project owns servers — shows the latest audit per server, links to drift if any.
• Experiment mode (experiment palette) shows individual capture-layer experiment runs as they execute, the same as any other experiment.

Consequences

Easier:

• Onboarding. Point at a new box, get a catalog in minutes; know what is there without SSHing in to grep.
• Disaster recovery rehearsal. Run provision-from-catalog into a test box, verify match — recovery validated without a real outage.
• Migration. Capture source, provision-into target, verify equivalence, cut traffic. The verification step removes the "did I miss something" anxiety.
• Drift detection. Compare today's catalog to baseline; the diff IS the alert. No bespoke monitoring infrastructure needed.
• Documentation generation. A catalog is the architecture description. Render to markdown for human consumption when needed.
• Audit trail. Versioned catalogs in git become an answer to "what changed when".

Harder:

• Two-layer model means new contributors learn the experiment vs pipeline distinction before they can extend the system. The boundary rules above are the explanation.
• Block registry needs governance. The current implicit rule ("everything goes in blocks_builtin.go") will not scale. ADR-061's library extraction is part of the answer.
• Catalog storage decisions (filesystem vs DB) become harder as scale grows. Filesystem is the right MVP; promote when measured pain shows up.
• Schema versioning discipline. Once schemaVersion: 1.0 is in the wild, breaking changes need migration paths. Worth the cost; the alternative (silent schema drift) is worse.
• SSH-key exposure on the orchestrator. The backend now needs SSH access to every audited server. This is the trust boundary; the audit log of every capture invocation is the mitigation.

What we sign up to maintain:

• Catalog schema as a stable contract — docs/specs/catalog-spec.md is the source of truth, not the implementation.
• Plug-in author guide — how to write a new capture-layer plug-in (input env, output JSON shape, error handling, exit codes).
• Block registry hygiene — bump Version on behaviour change so retro feedback auto-archives.
• The secrets boundary — catalog never has secrets, period; CI should grep for likely-secret patterns in committed catalogs and fail on hits.
• Audit log of capture invocations — who triggered, what target, when, with what credentials. Non-trivial for compliance.

Alternatives considered

• Single layer (just experiments). Rejected: the catalog → provision flow is genuinely multi-step and benefits from composition. Forcing it into one experiment script creates an unmaintainable mega-script.

• Single layer (just pipelines). Rejected: capture-layer probes are single-purpose and do not need block-registry overhead. Experiments are right-sized for them.

• Server-side daemon (pull model). Rejected for v1: requires installation on every target, version drift problem, more security surface (daemon needs auth). On-demand SSH execution is enough at current scale; revisit if fleet exceeds ~50 nodes.

• Treat catalog as Terraform-style declared state. Rejected: catalog is captured from reality, not declared to reality. Mixing the two reverses the direction of authority. Provisioning is the consumer of truth, not its source.

• Adopt an existing CM tool (Ansible, Salt, Puppet). Rejected as scope: those are full configuration-management systems with their own learning curves. The pipeline engine + experiment runner already exist; building inside them is faster than adopting and integrating an external system. ADR can be revisited if the gyrum platform itself outgrows its current fleet.

• Catalog stored in DB from day one. Rejected as YAGNI: filesystem + git gives us the audit trail and diff for free until capture frequency exceeds daily-per-target. Promotion path is documented; do not pre-build for it.

Phasing

Phase 1a is in flight at the time of this ADR (PRs against ai-research).

Reference experiments: experiments/exp-315-hetzner-server-audit/, experiments/exp-316-github-repo-audit/ (in flight).

Reference pipeline: pipelines/server-provision-from-catalog.json (follow-up).

Schema reference: docs/specs/catalog-spec.md.