← Decisions operate mode →

Decisions

ADR-172: Submitter ↔ Worker Pipeline Contract

The companion JSON Schema lives at [`docs/specs/pipeline-contract-v0.md`](../specs/pipeline-contract-v0.md).

#172

ADR-172: Submitter ↔ Worker Pipeline Contract

Status: Proposed Date: 2026-05-07 Tracker: warp#1881 (Phase 0 of EPIC warp#1838) Cross-refs: ADR-170, ADR-171, ADR-118, ADR-166 (§4 verification — sibling protocol)

Status note — Proposed not Accepted

Same gating as ADR-171: this contract locks the wire format now while the substrate paper is fresh, but the contract's fitness is gated on Stage B and Stage C shipping evidence. Reclassify to Accepted when warp#1763 Phase 0 prototype's "what breaks" doc lands without surfacing structural issues.

The companion JSON Schema lives at docs/specs/pipeline-contract-v0.md.

Context

ADR-171 names player + pipelines + blocks as the unattended-execution model. ADR-170 names useless-on-compromise as the security commitment. This ADR commits to the wire contract that lets the model work across machines: a typed schema at every seam between submitter, queue, worker, and downstream consumer.

Without a typed contract:

  • The submitter and worker can disagree about the pipeline's shape and the disagreement only surfaces mid-run.
  • Inputs that are large (test artefacts, fleet configs) have no canonical transport.
  • Outputs that need persistence (test reports, log bundles) have no retention story.
  • Scope-declarations get re-derived per consumer; ADR-170's useless-on-compromise commitment can't be enforced declaratively.

The contract is what makes the substrate implementable across boundaries, not just runnable in one process.

Decision

The substrate's submitter ↔ worker contract has four parts: (1) the pipeline declaration schema, (2) three transport modes, (3) the submission and execution protocols, (4) the failure-mode taxonomy.

1. Pipeline declaration schema

Pipelines declare in YAML, version by integer, content-address by SHA-256. Every declaration carries:

  • id, version, hash — identity. Hash mismatch = different artefact.
  • description — operator-readable rationale.
  • inputs — named, typed (inline / git-ref / folder), with JSON Schema or constraint.
  • outputs — named, typed (inline / folder), with retention.
  • scope — credentials + network + filesystem + privacy. Per ADR-170; the vault, host firewall, and runtime read this declaratively.
  • contract — timeout, heartbeat cadence, cancellation semantics, retry policy.
  • steps — ordered block invocations with input/output mappings.
  • emit — the mapping from internal step state to declared outputs.

The full JSON Schema is at docs/specs/pipeline-contract-v0.md. Validators on both the submitter and worker consume the same schema.

2. Three transport modes

Inputs and outputs are typed; each type has exactly one transport.

Type Transport Use for Size cap (v0)
inline JSON embedded in the job body Flags, structured config, small verdicts 64 KiB
git-ref {repo, sha} pair; worker clones with read-only deploy key Source code n/a (depth-1 clone)
folder tar.gz in the object store; URI + SHA-256 in the job body Test artefacts, fleet configs, log bundles 100 MiB

Three modes is deliberate. inline handles small structured data without object-store roundtrips. git-ref avoids re-tarballing repos already content-addressed by Git. folder handles arbitrary file trees as one atomic, hashable, transferable unit.

3. Submission and execution protocols

Submitter (operator's laptop, scheduled job, another pipeline):

  1. Resolve pipeline-ref. Verify hash.
  2. Prepare inputs per transport mode.
  3. POST /api/v1/jobs with pipeline_ref + pipeline_hash + inputs manifest + expected outputs + scope_consent.
  4. Poll job status or subscribe to events.

Worker (player on a worker host):

  1. Atomic claim from queue.
  2. Pre-flight contract validation (fail-closed):
    • Pipeline ref + hash known to player.
    • Player version meets contract.pipeline_player_min_version.
    • All required inputs present, types match, sizes within caps.
    • Vault can issue declared credentials.
    • Egress allowlist reachable.
  3. Materialise inputs locally.
  4. Execute pipeline (each block in a sandboxed subprocess; per-block scope enforcement).
  5. Heartbeat every contract.heartbeat_seconds.
  6. Post-flight output validation.
  7. Materialise outputs to object store (folders) or completion payload (inline).
  8. POST /jobs/{id}/complete with output manifest.

4. Failure-mode taxonomy

The contract distinguishes who is at fault:

Failure Cause State Recovery
Pre-flight contract violation Submitter sent invalid inputs, version skew, or scope unobtainable blocked Submitter fixes, re-submits
Worker block failure (retry-exhausted) A block returned non-zero or invalid output failed Block author investigates
Output contract violation Worker produced outputs that don't match the schema failed_complete Block author investigates; trace preserved
Lease expiry Worker died mid-run ready (re-queued) Idempotency is the pipeline's responsibility
Cooperative cancellation cancel_requested: true set on the job cancelled Outputs to that point preserved
Timeout Run exceeded contract.timeout_seconds failed_timeout Partial outputs preserved if any

Each failure state surfaces in /operate and produces a structured trace bundle (per ADR-170 Layer 4).

Consequences

What becomes load-bearing

  • Submitter and worker can disagree only at known seams. Pipeline-hash mismatch, version skew, scope-unobtainable: these are caught pre-flight, not mid-run.
  • Idempotency is structurally encouraged. Lease expiry returns the job to ready; the next worker re-runs. Pipelines either are idempotent (most are) or document why not.
  • Audit trail is first-class. Every job produces a trace bundle (substrate paper §3.4). Forensic investigation has a complete, structured record.

What stays out of scope

  • How the queue is implemented. ADR-173 covers the queue's relationship to Warp. This ADR covers the contract, not the storage.
  • The block-author contract. ADR-118 already covers per-block validation. This ADR covers the pipeline-author contract.
  • Schema for type:ci_run and type:deploy_request Warp tickets. Those are siblings (ADR-166 + warp#1763's future ADR). They share this contract's structural shape but evolve at their own pace.

What changes for tooling

  • factory-cli queue submit becomes the canonical submission CLI.
  • factory-cli player run becomes the canonical local-execution CLI (mocked credentials available for development).
  • factory-cli pipeline validate and factory-cli block validate enforce the schemas at authoring time.
  • gyrum-review-pr calls the validators as a pre-merge gate (per ADR-118).

Compliance / how reviewers check

A pipeline-shaped PR satisfies this ADR iff:

  • All required declaration fields are present (id, version, hash, inputs, outputs, scope, contract, steps, emit).
  • Every input/output declares its transport mode and JSON Schema.
  • The scope declaration is justified; the validator rejects wildcards in egress_allowlist.
  • The contract declares cooperative cancellation (preemptive is target-state for specific use cases only).
  • The pipeline's regression suite (per ADR-118 sister contract on blocks; this is the pipeline-level analogue) covers happy-path + at-least-one failure mode.

Cross-link

The substrate paper Part III walks through the contract with the JSON Schema in Appendix A; the spec markdown at docs/specs/pipeline-contract-v0.md is the standalone implementation target. ADR-166's §4 verification protocol is the deploy-shaped instance of this ADR's general form.