← Decisions operate mode →

Decisions

ADR-124: PRINCIPLE — pipeline tooling itself must be block-composition, not monolithic Go

The fleet has shipped two compatible-but-distinct shapes for "what a pipeline step looks like". The library shape — `gyrum-pipelines` — is block-composition: each step is a typed `block.Def`, the pipeline is a JSON…

#124

ADR-124: PRINCIPLE — pipeline tooling itself must be block-composition, not monolithic Go

Status: Proposed Date: 2026-04-29 Related: ADR-067 (playbooks-as-unified-primitive), ADR-068 (playbook-runtime), ADR-085 (pipeline-blocks-typed-io-ports), ADR-090 (ai-as-scaffold-pipelines-as-runtime), ADR-113 (principles tier above initiative), ADR-115 (principle-aware PR reviewers), ADR-117 (module guidelines), ADR-123 (templates-as-generator)

Context

The fleet has shipped two compatible-but-distinct shapes for "what a pipeline step looks like". The library shape — gyrum-pipelines — is block-composition: each step is a typed block.Def, the pipeline is a JSON ordering of BlockRefs, and a thin runner.Run(ctx, Config) walks them. The application shape — ai-research/cmd/server/pipeline/ — is imperative: each step is a Go function reaching into a shared RunContext, the pipeline is a hand-coded []Step slice in AllBuildSteps(), and the orchestration is hand-wired control flow.

Today's empirical evidence makes the gap visible: ai-research/cmd/server/pipeline/ contains 367 .go files; exactly one (library_bridge.go) imports gyrum-pipelines/pkg/block. The bridge file's own header comment names the shape as Phase 1 of a multi-PR migration that has not graduated. ADR-123 named the same failure mode one layer up — Build phase was a single opus-driven monolith generating 80% of every project from scratch — and the resolution was the same shape: deterministic templates plus declarative composition plus scoped AI gap-fill.

The pipeline tooling layer never received that resolution. The 600s sub-agent watchdog incident on warp#622 (memory: feedback_subagent_watchdog_600s.md) was the same monolith shape: a single agent shipping a single large change because the substrate offered no smaller atomic unit. The library_bridge.go comment ("ai-research's local BlockDef carries a CustomFn closure that the library deliberately excludes") is honest about why the migration stalled — but the principle that justifies completing it has never been written down. Reviewers seeing a new monolithic RunContext-shaped step had no canonical rule to cite; the shape kept reproducing.

The principle below is the recursive application of the warp#724 generic-primitives principle — block names are abstract; domain shape lives in the pipelines composing them — to pipeline tooling itself. ADR-085 typed I/O ports gave us composable units; ADR-090 declared pipelines are the runtime; ADR-123 declared templates beat AI-as-generator. This ADR closes the loop: the pipeline tooling that runs those pipelines must itself be a block-composition pipeline.

Decision

Pipeline tooling itself MUST be block-composition pipelines with declarative logic-routing. Imperative Go orchestration is not the pipeline's runtime — it is at most the executor's implementation detail behind a runner.Executor port (per ADR-085 + the gyrum-pipelines hex-arch boundary).

The shape of a pipeline-step contribution becomes:

  • Block names are abstract primitives. read-json, validate-schema, transform, write-json, http-probe, assert-equal. They name a generic capability and are reusable across pipelines.
  • Pipeline names are domain-specific. discovery-brief-enhancement, api-contract-extract, provision-and-deploy-warp. They name the business concern and compose blocks into it.
  • Logic routing lives in the pipeline file, not in the block. if, foreach, cache, retry, timeout are routing primitives expressed declaratively in YAML/JSON between block invocations. A block with branching switch statements over its inputs is a smell — that's pipeline-shape leaking into the block.
  • Single concern per block. A block does one thing. If it needs to know about two upstream concerns to decide what to do, it is two blocks plus routing.
  • Typed I/O ports per ADR-085. Block declarations carry inputs: and outputs: with named, typed ports. The validator catches reference typos and type mismatches at design time, before the runtime ever loads the pipeline.

Imperative Go orchestration (func Run(ctx *RunContext) reading from a shared bag, hand-wiring []Step{NewStep(...)...} slices) is reclassified as the legacy graduation backlog. Existing consumers keep working — Phase 1 of ADR-085 was deliberately non-breaking — but new pipeline tooling MUST be authored in the block shape, not the imperative shape.

This is a kind: principle per ADR-113 and is being filed as a principle- warp item alongside the other founding principles. The applies_to patterns are cmd/server/pipeline/**, pkg/pipeline/**, pkg/block/**, and any new directory whose go.mod imports gyrum-pipelines/pkg/block. Reviewers (ADR-115) cite this principle when blocking PRs that introduce a new monolithic step.

Consequences

What becomes easier.

  • New pipeline tooling has one shape, not two. Authors compose existing blocks; the validator catches mistakes at design time; the diff a reviewer reads is a small YAML change against a known catalogue, not a 200-line imperative function.
  • Atomic units shrink, killing the 600s watchdog failure mode. A sub-agent ships one block (or one pipeline edit) per PR, well inside the time-box, without having to bundle several concerns into a single monolithic step.
  • Substrate convergence with the rest of the fleet. ADR-067 unified playbooks; ADR-085 typed pipeline blocks; ADR-123 inverted the build pipeline; this ADR brings the pipeline tooling layer into the same substrate. One mental model across all four.
  • Reviewability. A block with typed I/O ports is self-documenting; the pipeline file is a literal DAG. The cost of "what does this step do" drops to reading two declarations.

What becomes harder.

  • The 366 imperative-shaped step files in ai-research/cmd/server/pipeline/ are now the legacy graduation backlog. The migration is real work — tracked in warp#707 as a separate epic — and lands incrementally, not in a big-bang rewrite. ADR-085's Phase 1/2/4 staging applies: existing pipelines keep running while new work is authored in the block shape.
  • Authoring overhead per simple step rises. A two-line "transform JSON" used to be a Go function; it now declares typed ports plus a registered block. Mitigation: the verbosity is one-time per author and the alternative is the failure modes named in library_bridge.go (CustomFn closures, untyped bags, regex-scraped references).
  • The block catalogue itself is a fleet-wide artefact that needs care. Block names must stay abstract (warp#724); domain leakage into a block name is a bug. Mitigation: ADR-117 module guidelines treat pipeline-step as a module type with block-name = generic primitive as a forbidden-pattern check.

What we sign up to maintain.

  • The new pipeline-step module guideline (ADR-117 sibling), authored in r3 of this principle's epic.
  • The 4-layer enforcement gate stack (warp#705) that rejects new monolithic steps at author / PR / persona / drift time.
  • The graduation campaign (warp#707) for the existing 366 imperative files, prioritised by churn — high-traffic steps migrate first.
  • The principle-as-warp-item record per ADR-113, with applies_to, revision_history, and constrained_by (this principle is constrained by warp#724 generic-primitives).

Alternatives considered

  • Allow both shapes indefinitely. Rejected. The status quo is one bridge file in 367, and the bridge has not graduated since it landed. Without a principle citing the canonical shape, the ratio stays where it is and reviewers have no anchor to push new monolithic steps back from.
  • Imperative is the canonical shape; deprecate the library. Rejected. The library is the substrate the entire fleet (warp, distill, ai-frontend, plus seven other consumers) already depends on for typed pipeline primitives. Deprecating it would require rebuilding ADR-085's typed I/O contract from inside cmd/server/pipeline/ — the same work, with fewer consumers, and no path to the typed-block UI ADR-085's Phase 3 is built on.
  • Block-composition only for new pipelines; never migrate the legacy. Rejected. The legacy ships findings into the same retro store as new pipelines; a divergent shape there means two debugging models, two test harnesses, two reviewer mental contexts forever. The migration is gated on warp#707's prioritisation (churn-weighted), not on a flag-day cutover.
  • Add block-composition as a new option without naming it the canonical shape. Rejected. That is what the library_bridge.go PR did, and the result is the present ratio (1 in 367). A principle is a citable rule; an option is a preference. The cost of NOT writing this down is the recurrence of the library_bridge.go shape — a sound migration that stalls because the principle was never named.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

References

  • ADR-067 — playbooks-as-unified-primitive (the substrate this principle extends)
  • ADR-068 — playbook-runtime (the runtime this principle constrains)
  • ADR-085 — pipeline-blocks-typed-io-ports (the typed-port contract block-composition rests on)
  • ADR-090 — ai-as-scaffold-pipelines-as-runtime (the principle this one refines for pipeline tooling)
  • ADR-113 — principles tier above initiative (the substrate this principle is filed under)
  • ADR-115 — principle-aware PR reviewers (the gate that cites this principle on blocking)
  • ADR-117 — module guidelines (the per-module guideline pipeline-step@v1 rests on this principle)
  • ADR-123 — templates-as-generator (the same shape one layer up — deterministic substrate plus scoped AI gap-fill)
  • warp #698 — the principle being ratified by this ADR
  • warp #724 — generic-primitives principle, constrains the block-naming rule above
  • warp #705 — sibling enforcement: 4-layer gate stack against monolithic pipeline tooling
  • warp #706 — this doc cascade
  • warp #707 — graduation epic: convert 366 imperative cmd/server/pipeline/ files to block-composition
  • Memory feedback_modular_pipelines_default.md — operator preference, 2026-04-29 (motivates this ADR)
  • Memory feedback_subagent_watchdog_600s.md — the failure mode this principle structurally prevents