ADR-085: Pipeline blocks as typed I/O ports

Status: Accepted Date: 2026-04-25 Related: ADR-067 (playbooks unified primitive), ADR-068 (playbook runtime architecture), ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-084 (outbound-call chokepoint)

Decision (one paragraph)

Every playbook step declares its inputs and its outputs as named, typed ports. Wiring between steps is explicit — a downstream input names an upstream output by path: step.wait_ssh.inputs.host: ${steps.provision.outputs.ip}. The validator resolves every reference at design-time and checks type compatibility before the runtime ever loads the playbook. The runtime marshals each step's outputs by declared type, so an ipv4 flowing into an int64 input fails before the call dispatches. The playbook UI (ADR-070) renders the DAG as blocks with connectors drawn between typed ports. The existing shell step survives as the stringly-typed escape hatch — its outputs are still {stdout, stderr, exit_code} — for operations the typed-block library does not yet cover.

Context

Today's playbook step is a shell wrapper with an implicit contract

The walking-skeleton runtime (ADR-068) treats steps as shell wrappers. Whatever stdout the command emits, whatever ${inputs.X} or ${steps.Y.Z} references the command string contains — the runtime discovers references by regex-scanning the command, not by reading a declared schema.

Outputs are untyped (map[string]any). There is no schema saying "this step produces an IP address and a server ID"; there is only "this step ran and printed some lines". The internal/playbook/resolve.go resolver already substitutes ${inputs.X} and ${steps.X.Y} placeholders, so the port shape is latent in the engine — we just have not made the ports declared.

This was fine for Phase 1. It does not scale to the eight-playbook infrastructure catalogue ADR-072 committed to.

Tonight's incident — provision-and-deploy-warp.yaml

Tonight's session shipped provision-and-deploy-warp.yaml. Step 2 calls hcloud server create and produces an IP. Steps 3, 4, 5, 6, and 8 each need that IP. The playbook works around the missing typed-output passing by having each later step re-query hcloud for the server's IP.

That is five wasted API calls per run. It is brittle on hcloud rate-limits — a 429 in step 6 fails a deploy that step 2 already succeeded at. It is invisible to the validator: nothing warns that step 6's hcloud server describe is conceptually a re-fetch of data step 2 already produced.

A typed block with outputs: [{name: ip, type: ipv4}] lets step 3 wire host: ${steps.provision.outputs.ip} cleanly, costing one API call total across the playbook. The same pattern recurs across every non-trivial playbook in the queue (deploy-service, deploy-warp, host-onboarding, rotate-credentials, etc.).

The adjacent pieces all point this direction

ADR-084 (merged tonight as v0.1.56) ships an outbound toolkit whose Go Client methods return typed values — ServerCreate returns *Server, not map[string]any. A typed executor wrapping a typed Client method is a natural fit: the executor's declared outputs map 1:1 to the struct fields its Client method already returns. The chokepoint and the typed-block layer are designed to compose.

The frontend contract (ADR-070) already needs to render runs. A block-graph view with typed connectors is a UI that pays for itself the moment outputs are declared — without declarations the UI would have to infer the graph from regex-scraped references, which is the same brittleness the runtime suffers from today.

Decision body

What a typed block looks like

A step grows an outputs: list paralleling its inputs: map. Each output names a port and a type. The runtime guarantees that after the step succeeds, every declared output is populated — failure to populate is a runtime error, the same way a missing required input is.

- id: provision
  type: hetzner-server-create
  inputs:
    name: ${inputs.server_name}
    server_type: ${inputs.server_type}
    location: ${inputs.location}
    image: ubuntu-24.04
    ssh_keys: [jon-dev]
  outputs:
    - name: ip
      type: ipv4
    - name: server_id
      type: int64
    - name: status
      type: string

- id: wait_ssh
  type: ssh-wait
  inputs:
    host: ${steps.provision.outputs.ip}
    user: root
    timeout: 150s
  outputs:
    - name: connected_at
      type: timestamp

Compare to today's shell-step shape — provision-and-deploy-warp.yaml shipped tonight has type: shell, a command: string with embedded substitutions, and an output map populated only by stdout parsing. Every downstream step that needs the IP either re-queries hcloud or scrapes stdout with a regex.

The validator's new job

Today's validator checks YAML well-formedness and a small set of schema constraints (every step has an id, every reference at least parses). Phase 1 of this ADR adds three checks:

• Every ${steps.X.outputs.Y} reference resolves to a declared output of step X.
• Every ${inputs.Z} reference resolves to a declared playbook-level input.
• Output names within a step are unique.

Phase 4 turns the soft warnings of Phase 1 into hard failures and adds type-compatibility checking — ipv4 cannot flow into int64; string can flow into anything; timestamp requires explicit cast to string.

How shell steps coexist

Shell remains a first-class step type. Its outputs are the existing {stdout, stderr, exit_code} triple, all stringly-typed. Wiring a shell step's stdout into a typed block's int64 input requires an explicit cast block (type: cast-to-int64) — the cast block is where parse-failure errors get a proper home.

Typed executors declare their own output shapes. The shell escape hatch is preserved for the long tail of operations the typed-block library does not yet cover.

Migration path — additive

Phase 1 is non-breaking. Existing playbooks keep working with no outputs: declarations — the validator emits a warning, not a failure. New playbooks grow outputs: declarations as authors write them. Phase 2 adds typed executors next to the shell step, not in place of it. Phase 4 — full type-enforcement — only flips on once every existing playbook has been migrated. No big-bang.

Worked example — block graph

flowchart TB
    inputs[playbook inputs<br/>server_name · server_type<br/>location · domain]
    provision[provision<br/>hetzner-server-create]
    wait_ssh[wait_ssh<br/>ssh-wait]
    dns[dns_upsert<br/>cloudflare-dns-upsert]
    deploy[deploy_warp<br/>ansible-playbook]
    verify[verify<br/>http-probe]

    inputs -->|server_name| provision
    inputs -->|server_type| provision
    inputs -->|location| provision
    inputs -->|domain| dns
    provision -->|outputs.ip ipv4| wait_ssh
    provision -->|outputs.ip ipv4| dns
    provision -->|outputs.ip ipv4| deploy
    wait_ssh -->|outputs.connected_at timestamp| deploy
    dns -->|outputs.fqdn string| verify
    deploy -->|outputs.warp_url string| verify

The graph is what the ADR-070 /playbooks/[id] view renders in Phase 3 — nodes are blocks, edges are typed connectors, edge labels carry the port name and type.

Consequences

Positive

• Design-time validation catches typos before a run starts — ${steps.provison.outputs.ip} (note the typo) fails the validator instead of dying mid-run with a confusing nil-deref.
• The block-graph UI becomes feasible — the graph is read directly from declarations, not reverse-engineered from regex scans.
• Typed substitution prevents silent coercion — an ipv4 cannot flow into an int64 input, which today is a class of bug that surfaces only at the API call.
• Blocks become self-documenting — the I/O declarations are the docs for the step.
• Outbound-toolkit callers (ADR-084) map naturally to typed blocks — the executor is a thin shim around a typed Client method.

Negative / risks

• YAML verbosity rises ~50% per step. Mitigation: most playbooks have ≤10 steps, the verbosity is one-time per author, and the alternative is regex-scraping stdout which is worse.
• The shell escape hatch becomes tempting if typed blocks are made strict too early. Mitigation: Phase 4 (type-enforcement) does not land until Phases 1-3 are done, and shell stays a first-class step type forever.
• Migration cost on existing playbooks (deploy-service, deploy-warp, host-onboarding, provision-and-deploy-warp, etc.) is real. Mitigation: migration is mechanical, incremental, and one playbook per PR — the additive design means nothing breaks while in-flight.
• The block-graph UI is a future-phase deliverable, not an MVP. Mitigation: this is fine — Phase 1 ships value (validator catches typos) without needing the UI.

Phases

Phase 1 — Schema + validator (non-breaking)

Add outputs: declaration to the step schema. The validator checks every ${steps.X.outputs.Y} reference resolves to a declared output. Runtime behaviour is unchanged — no type enforcement yet. Existing playbooks keep working with no outputs: declared; the validator emits a warning (not a failure) on undeclared outputs and on shell-only chains. ~1 week of runtime work.

Phase 2 — Typed executors

Each ADR-084 outbound-toolkit provider gets a matching executor type in internal/playbook/ — hetzner-server-create, cloudflare-dns-upsert, github-pr-create, etc. The executor calls into gyrum-go/pkg/outbound/<provider>/, marshals results into typed outputs. Shell executor stays. Migrate provision-and-deploy-warp.yaml to use typed blocks where providers exist, shell for the rest. ~2 weeks.

Phase 3 — Graph UI

/playbooks/[id] grows a block-graph tab showing nodes and connectors. Live-run view highlights data flow on hover. Requires solid Phase 1/2 output plumbing. Builds on ADR-070. ~1 week.

Phase 4 — Type-enforcement

The validator fails (not warns) on type mismatches. Requires every existing playbook to be migrated first. End-state. Timing depends on migration velocity — likely 4-8 weeks after Phase 2 lands.

Open questions

1. Type system extent. Basic types (string, int64, bool, timestamp, ipv4) or structured types (a Server struct matching hcloud.Server)? Recommendation: start narrow with the basic types, extend to structured types when a real downstream consumer needs more than one field of an upstream output.

2. Optional vs required outputs. Should outputs: declare which are always-produced vs which are conditional (a step that creates-or-updates may produce different outputs depending on which branch ran)? Likely yes — add optional: true to the output declaration, validator warns when a downstream consumer reads an optional output without a guard.

3. Failure-case outputs. Do outputs still emit on step failure, or is the step's outputs map empty on error? The existing shell executor returns stdout+stderr even on error; typed executors need a consistent answer. Recommendation: failed steps emit no typed outputs; downstream on_failure: handlers see only the error, not partial data.

4. Retry and idempotency. If a step with typed outputs is retried, do we cache outputs for idempotency or re-run? Cache wins on the cost axis but cache invalidation is ADR-material of its own — likely a follow-up ADR (086 or later) once Phase 2 has real data.

5. Backward compatibility timeline. Existing playbooks without outputs: declarations — warn forever, or deprecate after Phase 2? Recommendation: warn through Phase 2, hard-fail at Phase 4. Gives migration a clear deadline tied to a structural gate.

Related work

• ADR-067 and ADR-068 — the primitive this ADR extends. Typed ports are a property of the step schema; the runtime contract ADR-068 defined is the substrate.
• ADR-084 — the outbound toolkit that Phase 2 typed executors consume. This ADR is the playbook-side counterpart to the toolkit-side typing already shipped.
• ADR-070 — the frontend contract this ADR's Phase 3 builds on. The block-graph view is a new tab on /playbooks/[id], not a new page.

Prior art — n8n, Dagster, Airflow, Node-RED. All four are DAG-of-blocks systems with typed connectors. All four chose this shape for the same reasons we are choosing it: design-time validation, graph rendering, self-documenting I/O. The convergence is evidence the shape is correct, not a coincidence.

Motivating session — see dark-factory/docs/sessions/2026-04-24.md for the provision-and-deploy-warp.yaml shipping context that surfaced the re-query workaround driving this ADR.

Amendment — 2026-04-29 (warp#698 / ADR-124)

ADR-124 ratifies the recursive form of this ADR's typed-port contract: block names MUST be generic primitives; domain shape lives in the pipelines composing them. The single concrete consequence for authors of new blocks is that a block name describing a domain (generate-api-contract) is wrong-shaped — it should be a generic primitive (json-transform) composed into a domain-named pipeline. Reviewers cite ADR-124 when blocking a new domain-named block.