ADR-061: Extract pipelines + projects to standalone libraries

Status: Accepted Date: 2026-04-22 Related: ADR-058 (5-mode UI), ADR-059 (project-first IA), ADR-060 (catalog-driven infrastructure)

Context

Two pieces of generic infrastructure currently live inside application repositories:

• Pipeline engine — block registry + JSON pipeline parser + runner + retro-feedback machinery, currently at gyrum-labs/ai-research/cmd/server/pipeline/. It is not specific to AI research; the GUIDE.md explicitly calls it "a generic layered development workflow" that "works for Go backends, SvelteKit frontends, HTML pages, or any other tech stack". The model is JSON-data + a registry of reusable Blocks.
• Project schema + validator — YAML parser + dependency-graph validator + mirror-edge enforcement, currently at gyrum-labs/ai-research/internal/projects/ (Go, ADR-059's PR #576) and gyrum-labs/dark-factory/scripts/validate-projects.sh (bash, PR #294). Neither is specific to ai-research or dark-factory; they are generic fleet-membership data models.

Both have outgrown the application repos that birthed them. When the catalog work in ADR-060 starts using the pipeline engine, it will reach across into ai-research for code that has nothing to do with AI research. The pattern repeats: every consumer of either engine has to either depend on ai-research (wrong abstraction layer) or copy the code (maintenance hell).

The operator's framing on 2026-04-22: extract both to standalone repos with full documentation. This ADR captures that decision and the shape of the extraction.

Decision

Extract two new libraries to dedicated GitHub repositories:

Each repo is GitHub-standard, documented end-to-end, and TDD'd (rules below).

Why two separate libraries (not one)

Pipelines and projects are conceptually orthogonal:

• Pipelines = "execute a graph of work"
• Projects = "describe a fleet of related repositories"

A consumer can want one without the other. Releasing them together means every consumer pays the dependency surface of both. Two libraries, two release cadences, two scopes.

Why Go AND TypeScript

Both engines are consumed from both stacks today:

• Pipelines: ai-research backend (Go) executes them; ai-frontend (SvelteKit/TypeScript) needs to validate them client-side and may eventually compose them visually. Backend roundtrips to validate JSON schema are wasteful.
• Projects: ai-research backend (Go) reads YAML for the API; ai-frontend (TypeScript) parses the same YAML for project pages and import wizard validation. Without a shared library, schema drift is inevitable.

Go is the primary implementation (where the runner lives, where validation logic is canonical). TypeScript is the mirror (read-only parser + validator + types). Tests in both languages run against the same JSON/YAML fixtures so drift fails CI.

Repository layout (proper GitHub standard)

Both repos follow the same skeleton (substituting <lib> for the library name):

gyrum-labs/<lib>/
├── README.md                          # Quick-start + the elevator pitch
├── LICENSE                            # MIT (matches gyrum convention)
├── CONTRIBUTING.md                    # How to add a new block / plug-in / class
├── CODE_OF_CONDUCT.md                 # Standard contributor covenant
├── SECURITY.md                        # Vuln reporting + supported versions
├── CHANGELOG.md                       # Keep-a-Changelog format
├── VERSION                            # Anchor for version-tag.sh derivation
├── .github/
│   ├── ISSUE_TEMPLATE/
│   │   ├── bug_report.md
│   │   └── feature_request.md
│   ├── PULL_REQUEST_TEMPLATE.md
│   ├── workflows/
│   │   ├── ci-go.yml                  # Build + test + lint Go side
│   │   ├── ci-ts.yml                  # Build + test + lint TS side
│   │   ├── parity.yml                 # Cross-language fixture parity
│   │   └── release.yml                # Tag + publish on main
│   └── dependabot.yml
├── docs/
│   ├── concepts.md                    # Mental model — read first
│   ├── userguide.md                   # Full reference for consumers
│   ├── tutorials/
│   │   ├── 01-first-pipeline.md       # Walk through writing one
│   │   ├── 02-custom-block.md         # Register a new reusable block
│   │   ├── 03-retro-feedback.md       # How findings + version flow works
│   │   └── 04-rag-pipeline.md         # End-to-end RAG example (pipelines repo)
│   ├── examples/
│   │   ├── discovery-only.json        # Minimal pipeline
│   │   ├── layered-build.json         # Plan→Dev→PR loop
│   │   └── catalog-to-bootstrap.json  # ADR-060's provision pipeline
│   ├── reference/
│   │   ├── block-schema.md            # Full BlockDef field reference
│   │   ├── pipeline-schema.md         # JSON Schema for pipelines
│   │   └── runner-api.md              # Runner Go + TS API
│   └── faq.md
├── pkg/                               # Go (canonical impl)
│   ├── block/
│   ├── pipeline/
│   ├── runner/
│   └── retro/
├── ts/                                # TypeScript mirror
│   ├── src/
│   │   ├── block.ts
│   │   ├── pipeline.ts
│   │   └── retro.ts
│   ├── test/
│   ├── package.json
│   └── tsconfig.json
├── fixtures/                          # Shared JSON/YAML used by both langs
│   ├── valid/
│   │   ├── minimal.json
│   │   └── full.json
│   └── invalid/
│       ├── missing-blocks.json
│       └── circular-ref.json
├── examples/                          # Runnable demos (Go binaries + TS scripts)
│   ├── go/
│   └── ts/
├── go.mod
├── go.sum
└── .gitignore

Documentation contract

Each repo ships:

• README.md: 1-page; what it is, what it isn't, install + 30-second example, links to deeper docs. No marketing fluff.
• docs/concepts.md: the mental model. Block, Pipeline, Runner. One diagram. No code yet.
• docs/userguide.md: full reference. Every public API documented. Every parameter explained. Worked examples for every common task.
• docs/tutorials/: step-by-step walkthroughs for the most common consumer journeys. Each tutorial has a runnable code sample at the end.
• docs/examples/: reference pipeline JSONs and project YAMLs that are CI-verified to parse and validate.
• docs/reference/: complete API docs. Auto-generated from doc-comments where the language supports it (Go's godoc, TypeScript's TypeDoc).
• docs/faq.md: real questions from real users. Grows over time.

Two readers, two doors

Both libraries have two distinct audiences and the docs explicitly address both.

For an AI researcher

The pipeline engine is a prompt-graph orchestrator:

• A Block is a reusable AI prompt with a model tier (haiku / sonnet / opus / qwen / local). Versioned. Cost-aware.
• A Pipeline is an experiment chain — declarative graph of blocks with named inputs flowing between them.
• Retro feedback is a research notebook bound to specific blocks; when a block's behaviour changes (Version bump), the old notes auto-archive so conclusions are not silently invalidated.
• The runner handles loops (RepeatFrom) for layer-by-layer building, the same primitive a researcher needs for refining a hypothesis.

For a project YAML schema, an AI researcher sees:

• A way to bound the scope of an experiment ("apply this analysis to Project X's repos only").
• A dependency graph that can be walked when an experiment outputs affect downstream consumers ("if I change gyrum-go, which projects' results need re-validation?").

For a software developer

The pipeline engine is a workflow runner:

• A Block is a unit of work — code or template.
• A Pipeline is a JSON-defined orchestration. No code to deploy a new workflow; just a JSON file + a registry entry.
• Retro feedback is code-review-bound-to-step.
• The runner handles loops, retries (when added), and the 3-persona PR review at the end of any workflow that ships code.

For a project YAML schema, a developer sees:

• A canonical place to declare what repos belong to what product.
• CI-enforced membership rules (no orphans, no cycles, no sunset-with- consumers).
• A read-only API the IDE / dashboards can consume to show "is my repo in any project?"

How RAG fits in (per the operator's question)

Retrieval-Augmented Generation is one of the most natural use cases for both libraries:

• gyrum-pipelines for building a RAG system: chunking, embedding, vector search, retrieval ranking, prompt synthesis are all blocks. A RAG pipeline is a JSON file. Tweaking retrieval strategy is a JSON edit.
• gyrum-pipelines USING RAG: when the registry exceeds ~100 blocks, block discovery itself benefits from RAG ("find me a block that does retry with exponential backoff"). The library can surface block-search as a built-in retrieval layer over its own registry.
• gyrum-projects as RAG context: the project graph is structured context. A coding assistant retrieving "what services depend on X" gets a precise answer from the dependency graph instead of a fuzzy keyword search.

The RAG tutorial (docs/tutorials/04-rag-pipeline.md in gyrum-pipelines) walks through a working example end-to-end.

TDD requirement

Both libraries enforce TDD as a contract, not a suggestion:

• Tests written before the implementation, not after
• Coverage gate: ≥85% for new code, ≥80% overall
• Cross-language parity: every fixture in fixtures/valid/ parses identically in Go and TS; every fixture in fixtures/invalid/ produces the same error category
• parity.yml workflow runs both stacks against the same fixtures on every PR
• Tutorial code samples are CI-tested — broken tutorial code fails CI

The pre-push hook (gyrum-devtools) gates on local test runs; CI re-runs + checks coverage delta.

CI workflows

Each repo has four GitHub Actions workflows:

Release + versioning

• Semantic versioning, anchored by VERSION (matches gyrum-devtools' version-tag.sh)
• Pre-1.0 (0.x.y) until the schema and API have at least three external consumers each (other gyrum products) — early API churn is OK
• Once 1.0: breaking changes only on a minor or major bump with a documented migration path
• Each release ships: tagged commit, GitHub release notes, Go module proxy entry, npm package version

GitHub-standard checklist (per repo)

• LICENSE (MIT)
• CONTRIBUTING.md (how to add a block / submit a fix)
• CODE_OF_CONDUCT.md (Contributor Covenant 2.1)
• SECURITY.md (vuln reporting, supported versions)
• CHANGELOG.md (Keep-a-Changelog format)
• Issue templates (bug / feature)
• PR template (checklist + 3-persona review reference)
• Branch protection on main: 3 required reviews, required CI checks, no force-push, no admin bypass
• Dependabot enabled
• CodeQL enabled (defensive)
• Topic tags (gyrum, workflow, ai, …)
• About section: 1-line description + homepage URL + license
• Pinned README in repo home

Migration path

This is a multi-PR sequence, not a big-bang:

Each step is a separate PR, each independently reviewable, each with its own gyrum-review-pr + gyrum-complete-pr cycle.

Consequences

Easier:

• New consumers (catalog work in ADR-060, future projects, external adopters) import a versioned dependency instead of forking application code.
• Cross-language parity is enforced at the library level, not duplicated per-app.
• Schema-on-paper (pipeline-schema.md) and schema-in-code stay in sync because the library OWNS both.
• Open-source potential. Both libraries are generic enough to be useful outside gyrum. MIT-licensed; the path to going public is "flip the visibility toggle".

Harder:

• Two more repos to maintain (release cadence, branch protection, CI quotas, Dependabot noise).
• Cross-repo refactors. Touching the library API now triggers PR storms in every consumer.
• Versioning discipline. Breaking changes need migration docs; we cannot silently rename a Block field anymore.
• TypeScript mirror is a permanent commitment to dual-language work.

What we sign up to maintain:

• The repo skeleton above as a template (probably via gyrum-create-repo with a --stack lib-go-ts option) so any future library extraction is consistent.
• The cross-language parity discipline (fixture-driven; no language gets to implement an extra feature without the other catching up).
• The documentation contract (every release with a public API change updates README + userguide + reference).

Alternatives considered

• One library containing both engines. Rejected: pipelines and projects are orthogonal. A consumer of one paying for the other is noise. Two repos give two release cadences, two scopes.
• Keep code in application repos, abstract via internal Go modules. Rejected: doesn't fix the TS/Go schema drift problem, doesn't fix the "every consumer forks" problem, doesn't unlock open-source.
• Go-only library, TS frontend calls Go via WASM. Rejected: WASM build artifact is heavier than a hand-mirrored TS implementation, and the parity test catches drift cheaper than WASM debugging.
• Use an existing workflow library (Temporal, Airflow, n8n). Rejected as scope: those are full systems with their own runtime requirements. The gyrum pipeline engine is small, opinionated, and cost-aware (model tier per block); third-party libraries don't carry those properties.

Related ADRs

• ADR-058: 5-mode UI — both libraries surface in the Platform UI's Infra mode (pipelines for execution, projects for the IA).
• ADR-059: project-first IA — gyrum-projects is the canonical implementation of that ADR's schema and rules.
• ADR-060: catalog-driven infrastructure — gyrum-pipelines is the composition layer for catalog-to-bootstrap pipelines.

Open questions for follow-up

• Should gyrum-pipelines ship a CLI (gp run pipelines/foo.json) or stay library-only? CLI is convenient but adds maintenance.
• Block discovery via RAG (mentioned in the dual-perspective section above) — interesting; not v1.
• npm package naming: @gyrum-labs/pipelines vs gyrum-pipelines. Match the GitHub org or use the unscoped pattern? Pick when registering.
• Visual pipeline composer (drag blocks, save JSON) — out of library scope; lives in ai-frontend or a future tool.