ADR-068: Playbook runtime architecture

Status: Accepted Date: 2026-04-23 Related: ADR-067 (playbooks unified primitive), ADR-064 (gyrum-catalog + hex architecture), ADR-062 (design system + component strategy), ADR-061 (extract pipelines + projects libraries), ADR-060 (catalog-driven infrastructure), ADR-053 (frontend observability), ADR-050 (bearer auth for product metrics)

Decision (one paragraph)

We build an embedded, Postgres-backed playbook runtime inside ai-research that executes playbooks (ADR-067) as first-class server-side work. The runtime streams events and terminal output to the frontend over SSE, accepts interactive replies over POST, and dispatches each step to a pluggable StepExecutor — shell, ssh, claude, http, pipeline, experiment, prompt, prompt_secret, approval, subplaybook. The queue is a Postgres-backed job queue (the river library, SELECT … FOR UPDATE SKIP LOCKED) so durability and back-pressure live in the database we already run, not a second piece of infrastructure. Every step declares target: local | host:X | container:Y at parse time — there is no bare-localhost default. Spec is YAML with a JSON-Schema validator generated from a Go struct. Security hardening (who can run what, secret vault, destructive-step annotation, audit chain) is explicitly deferred to ADR-069; the frontend-side rendering contract (Svelte component, form shapes, terminal pane) is deferred to ADR-070; extraction of the runtime to its own service (gyrum-playbook-runtime) is deferred to ADR-071 and triggered by operational need, not speculation.

Context

ADR-067 declared the playbook a unified primitive: one schema, one catalogue, one review cadence, six kinds today (journey, experiment, incident_response, service_runbook, release_flow, onboarding). The primitive settled the shape of an operational procedure.

Nothing executes one.

Today, when a human or an agent wants to "run" a playbook, they do one of:

• Open the Markdown file and follow the steps by hand (journey, runbook, onboarding).
• Run bash experiments/exp-315-hetzner-audit/run-*.sh from a laptop terminal (experiment).
• Invoke gyrum-complete-pr from a laptop terminal (release flow).
• Fall into an Alertmanager page and open a Slack thread (incident response).

Each path has a different substrate (human eyes, a shell on a laptop, an SDK CLI, a chat room). None of them streams progress back to the platform UI, pauses for a typed question, resumes after an answer, records a structured run envelope, or emits metrics into the distill observability stack (ADR-050, ADR-053). Ad-hoc work is done ad-hoc-ly and the trail is lost.

Concretely, four things are missing:

1. A runtime. No server-side engine takes a playbook id, resolves its front-matter and body, and executes the steps in order. Every screen that wants to "run the onboarding flow" or "run exp-316 again" has to reimplement orchestration.

2. Async + durability. A Hetzner audit run (exp-315) takes ~90 seconds on a cold cache and ~20 seconds on a warm one. A release flow can take ~10 minutes. Neither fits in an HTTP request window. Today nothing survives an ai-research restart mid-run, because nothing is persisted mid-run.

3. Interactivity. The operator's framing is explicit:

   if we had a install grafana and it needed some expra answers the frontend user, it would be helpful if they could send back answers … i like when you deploy you show the termianl output

   Playbooks that pause for a typed answer, a choice, or a yes/no approval are the majority of the useful ones (install-this-service, rotate-this-key, approve-this-destructive-change). Fire-and-forget is a subset, not the default.

4. A security boundary around arbitrary shell. Several kinds of playbook step ultimately run bash (shell step), or shell out to ssh, or invoke claude with agent tooling. Each of these is a large security surface. No primitive to gate them exists today. We must not ship a shell-running web service without one — but we also must not block the walking skeleton on the full security model. This ADR names that gap and fences it.

The sibling constraint, from the same operator framing:

1 yaml 2 probably best on the server, i dont want the frontend doing any dirty work, this may even mean a message bus to handle them and they return a job no and provide status whilst they work

The frontend is thin. It displays progress; it does not orchestrate. Authoritative state lives on the server, in a persistent store, and the server is the only thing that executes.

This ADR layers the runtime on top of the ADR-067 data model. It is deliberately an architecture ADR — Go interfaces, SQL tables, HTTP endpoints, SSE payload shapes — because the contracts here outlive the implementation and every follow-on ADR (069, 070, 071) pivots on them.

Decision

1. Summary of gating choices (committed, not re-litigated)

Five decisions are settled in this ADR. Each is motivated in the sections below; the table is the quick index.

2. The state machine

A playbook_run is a row whose life is a short, enumerated state machine. States are values in a Postgres enum; transitions are explicit; every transition is a committed SQL transaction before the runtime moves to the next step.

    pending ──▶ running ──▶ completed
       │          │
       │          ├──▶ awaiting_input ──▶ running
       │          │
       │          ├──▶ failed
       │          │
       │          └──▶ cancelled
       │
       └──▶ cancelled

States:

• pending — run row is written, queue job enqueued, no worker has claimed it yet.
• running — worker has claimed the run; current step is in flight.
• awaiting_input — current step is a prompt, prompt_secret, or approval; the run is suspended, the SSE stream has emitted awaiting_input, and nothing will advance until a valid reply lands on POST /input.
• completed — terminal; all steps returned success (or a step with on_error: continue advanced past its own failure).
• failed — terminal; a step returned an error that was not tolerated.
• cancelled — terminal; a user hit the cancel endpoint or the run exceeded its max wall-clock.

Sub-state on the in-flight step lives on the playbook_run_steps row: queued | running | awaiting_input | completed | failed:<reason>. The run-level state is a projection of the step-level states.

On-restart behaviour

ai-research is restart-safe in the tier-1 sense (observability, DB-first writes) but it is not guaranteed up. When a replica restarts:

1. On boot, the runtime queries for playbook_runs with state = 'running' whose owning worker (worker_id) is this replica's previous boot (worker_id includes a boot nonce).
2. Each such run has its current playbook_run_steps row marked failed:interrupted in a single transaction.
3. The run transitions to awaiting_input with a synthetic prompt: {question: "Step <name> was interrupted by a restart. Resume from this step, skip it, or abort the run?", schema: {type: enum, values: [resume, skip, abort]}}.
4. An awaiting_input SSE event is emitted; the operator's next POST /input reply selects the recovery path.

This is deliberate: automatic resume is wrong for any step with side effects (ssh, shell, claude with tools, http POST). We make the operator confirm. The run is not lost; the next action is not guessed.

A run whose owning replica is confirmed dead (out of scope for this ADR — a future supervisor role) is handed to a new worker through the same awaiting_input handshake.

3. The step executor contract

The central extension point. A step executor is a single Go interface:

// StepExecutor runs exactly one playbook step.
//
// Implementations:
//   - MUST stream stdout/stderr lines through `events` as they arrive,
//     not buffered until the end.
//   - MUST honour ctx.Done() for cancellation; a cancelled run will
//     cancel the context within one worker tick (~100ms).
//   - MUST NOT write to the playbook_runs or playbook_run_steps
//     tables directly; state transitions are the runtime's job.
//   - MUST return outputs as a flat map[string]any; nested structures
//     are allowed but top-level keys are what `save_as:` binds.
//   - SHOULD return a typed error from runtime/errors.go so the
//     runtime can classify (tolerable / fatal / awaiting_input).
type StepExecutor interface {
    // Kind returns the `type:` discriminator this executor handles
    // (e.g. "shell", "ssh", "claude"). One executor per kind.
    Kind() string

    // Validate is called once, at parse time, before any run starts.
    // It inspects the step's spec (after YAML → struct decoding) and
    // returns an error if the step is malformed. Validation errors
    // are reported before the run is enqueued, never mid-run.
    Validate(step Step) error

    // Execute runs the step. `inputs` is the result of resolving the
    // step's `with:` block against the run's variable environment
    // (previous `save_as:`, initial inputs, secrets).
    //
    // The executor writes events to `events`; the channel is
    // buffered (default 256). A full channel causes the executor to
    // block, which back-pressures stdout-chatty steps to the rate
    // the consumer (SSE pump) can drain.
    Execute(
        ctx context.Context,
        step Step,
        inputs map[string]any,
        events chan<- Event,
    ) (outputs map[string]any, err error)
}

Event payloads are a small, closed set:

type Event struct {
    Kind   EventKind      // step_started, stdout, stderr, step_completed,
                          // awaiting_input, run_completed
    StepID string         // stable id, e.g. "step-3-install-grafana"
    TS     time.Time
    Body   map[string]any // kind-specific; see §8
}

The runtime itself owns the run_completed event (the executor never emits it) and the awaiting_input → running transition (the executor emits awaiting_input and returns a sentinel error errAwaitingInput; the runtime persists the pause, suspends the worker slot, and resumes the same executor instance when the reply arrives).

Back-pressure

The events channel is buffered (default 256 events, tunable per run). If the SSE pump cannot drain fast enough — a chatty shell step emitting thousands of lines per second — the executor blocks on send. This is on purpose: the alternative (drop events) loses terminal output; the alternative (unbounded buffer) OOMs the replica. Blocking the producer is the correct failure mode.

For steps that legitimately produce very high-rate output (a large rsync, a database dump), the step's spec can declare output_mode: summary | tail:N | full (default full). summary stores a one-line digest; tail:N stores only the last N lines; full keeps everything. This gives the playbook author an explicit knob without changing the contract.

4. Built-in step types

Ten executors ship in Phase 1-4. Each has a fixed shape.

shell

Local bash. The most powerful and most dangerous. Gated behind the step's target: declaration; target: local requires the playbook front-matter to carry an explicit allow_local: true flag and an entry in a future runtime allow-list (ADR-069). target: host:X delegates to the ssh executor internally. target: container:X runs inside a disposable container (Phase 4+).

- id: dump-disk
  type: shell
  target: local                # REQUIRED — no default
  run: "df -h"
  timeout: 30s
  save_as: disk_summary

ssh

Delegates to the gyrum-catalog SSH Executor port (ADR-064). The SSH executor already has the host-resolution and credential-handling shape; the runtime passes the resolved host alias, the command, and a callback that forwards streamed lines into the runtime's events channel. No secrets ever appear in the runtime's memory longer than needed to populate the SSH session.

- id: restart-grafana
  type: ssh
  target: host:ops-vps         # resolves via gyrum-catalog
  run: "sudo systemctl restart grafana-server"
  timeout: 60s

claude

Calls the Anthropic SDK from Go directly — not claude -p CLI. Rationale: fewer moving parts (no CLI binary, no subprocess), secrets stay in-process, streaming tokens arrive as they are generated, and the SDK wrapper centralises rate-limit back-off.

- id: summarise-diagnostics
  type: claude
  target: local                # "local" for claude = in-process SDK call
  prompt: |
    Given the runbook diagnostics output below, summarise the three
    most likely root causes in one paragraph each.
    ---
    {{ outputs.dump_disk.stdout }}
  model: claude-opus-4-7
  expects: text                # text | json
  on_error: fail               # fail | continue | default_to: "<value>"
  timeout: 60s
  save_as: diagnostics_summary

Every type: claude step must declare expects: and on_error:. Claude calls are slow (2-10s typical, 30s tail) and non-deterministic; the author cannot hand-wave either dimension. expects: json triggers a JSON-mode SDK call plus a parse step; a parse failure is an executor-level error and respects on_error.

http

Arbitrary REST call. Method, URL, headers, body, expected status, response capture. Used for calling out to gyrum services, GitHub API, etc.

- id: create-gh-issue
  type: http
  target: host:github.com
  method: POST
  url: "https://api.github.com/repos/{{ inputs.repo }}/issues"
  headers:
    Authorization: "Bearer {{ secrets.github_token }}"
  body:
    title: "Runbook triggered: {{ inputs.alert_id }}"
    body: "{{ outputs.diagnostics_summary }}"
  expect_status: [201]
  save_as: issue

target: here is the hostname the HTTP call is going to — it is recorded for audit and can be constrained by ADR-069.

prompt

Pauses the run, emits awaiting_input, waits for POST /input. The reply is validated against a schema and stored in the variable environment under the step's id.

- id: ask-grafana-hostname
  type: prompt
  target: local
  question: "What hostname should Grafana be served at?"
  schema:
    type: string
    pattern: "^[a-z0-9.-]+$"
  default: "grafana.example.com"
  save_as: grafana_host

prompt_secret

Same shape as prompt but the reply is written to the secret vault (ADR-069) and referenced by id, never echoed back into playbook_run_events. The terminal pane shows •••••••• for any stdout line that contains the secret value (a cheap textual scrub).

- id: ask-api-key
  type: prompt_secret
  target: local
  question: "Paste the API key to store in the vault."
  save_as: api_key_secret_id

approval

Boolean gate. Pauses until POST /input with value: approve or value: reject. Reject is a fatal step error (run → failed); the on_error: knob lets the author override to continue.

- id: confirm-destructive
  type: approval
  target: local
  question: "This will delete the old Grafana config. Proceed?"

pipeline

Invokes a named gyrum-pipelines run (ADR-061). The pipeline runs in-process (library call, not a subprocess), streams its step events through the same events channel, and returns the pipeline's final outputs as this step's outputs.

- id: run-hetzner-audit-pipeline
  type: pipeline
  target: local
  pipeline: hetzner-audit
  with:
    project_id: "{{ inputs.project_id }}"
  save_as: audit_result

experiment

Invokes a named experiment (by its ADR-067 kind: experiment id). The experiment's run-*.sh is executed via the shell executor under the hood, with its declared inputs/outputs surfaced through the playbook runtime's variable environment. This is the migration bridge — existing experiments become step executors without being rewritten.

- id: audit-hetzner
  type: experiment
  target: host:ops-vps
  experiment: hetzner-audit
  save_as: hetzner_report

subplaybook

Invokes another playbook by its kind:id reference (same syntax as ADR-067 related: fields). The sub-run is a full playbook_run row with its parent's id on parent_run_id. Events from the sub-run are forwarded into the parent's SSE stream prefixed with the sub-run id. Max nesting depth is 5; deeper nesting fails at parse time.

- id: run-release-flow
  type: subplaybook
  target: local
  playbook: release_flows:gyrum-complete-pr
  with:
    pr_number: "{{ inputs.pr_number }}"
  save_as: release_result

The ten executors above are the Phase 1-4 set. Adding an eleventh (e.g. type: slack-post, type: github-pr-create) is a new implementation of StepExecutor plus a registry entry; the contract does not change.

5. Queue + durability

A run is a row. A step is a row. An event is a row (bounded retention). The queue is a table. One Postgres, three concerns.

The river library

We use river — a pure-Go, Postgres-backed job queue using SELECT … FOR UPDATE SKIP LOCKED. Rationale:

• Pure Go, no cgo, fits the ai-research module stack.
• SKIP LOCKED is the correct primitive for work distribution across replicas without a coordinator.
• Job state, retries, dead-letter are tables we can inspect with psql. There is no extra observability story to write; our existing slog + distill metrics wrap the worker loop.
• No second piece of infrastructure to operate. The Postgres we already back up is the only backing store.

river is a dependency, not an architecture commitment. If it goes unmaintained, the port is ~400 lines of SQL — we wrote a smaller equivalent for earlier experiments. We would only move off Postgres (to NATS or Temporal) if the queue becomes a throughput bottleneck; that evaluation happens at Phase 4.

Schema sketch

CREATE TYPE playbook_run_state AS ENUM (
  'pending', 'running', 'awaiting_input',
  'completed', 'failed', 'cancelled'
);

CREATE TABLE playbook_runs (
  id                 uuid PRIMARY KEY,
  playbook_kind      text NOT NULL,           -- ADR-067 kind
  playbook_id        text NOT NULL,           -- ADR-067 id
  playbook_version   text NOT NULL,           -- content hash of the playbook file
  state              playbook_run_state NOT NULL DEFAULT 'pending',
  parent_run_id      uuid REFERENCES playbook_runs(id),
  actor              text NOT NULL,           -- github handle of the initiator
  initial_inputs     jsonb NOT NULL,
  outputs            jsonb NOT NULL DEFAULT '{}',
  worker_id          text,                    -- replica + boot nonce
  started_at         timestamptz,
  finished_at        timestamptz,
  created_at         timestamptz NOT NULL DEFAULT now(),
  updated_at         timestamptz NOT NULL DEFAULT now(),
  CHECK (parent_run_id IS NULL OR id <> parent_run_id)
);

CREATE INDEX ON playbook_runs (state) WHERE state IN ('pending', 'running', 'awaiting_input');
CREATE INDEX ON playbook_runs (playbook_kind, playbook_id, created_at DESC);
CREATE INDEX ON playbook_runs (parent_run_id);

CREATE TABLE playbook_run_steps (
  id                 uuid PRIMARY KEY,
  run_id             uuid NOT NULL REFERENCES playbook_runs(id) ON DELETE CASCADE,
  step_index         int NOT NULL,            -- 0-based, matches YAML order
  step_id            text NOT NULL,           -- author's `id:` field
  step_type          text NOT NULL,           -- shell / ssh / claude / …
  target             text NOT NULL,           -- local / host:X / container:Y
  state              text NOT NULL,           -- queued / running / awaiting_input / completed / failed:<reason>
  inputs             jsonb NOT NULL,
  outputs            jsonb NOT NULL DEFAULT '{}',
  error              text,
  started_at         timestamptz,
  finished_at        timestamptz,
  UNIQUE (run_id, step_index)
);

CREATE TABLE playbook_run_events (
  id                 bigserial PRIMARY KEY,
  run_id             uuid NOT NULL REFERENCES playbook_runs(id) ON DELETE CASCADE,
  step_id            text,
  kind               text NOT NULL,           -- step_started / stdout / stderr / …
  body               jsonb NOT NULL,
  ts                 timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ON playbook_run_events (run_id, id);

Events are append-only. A nightly job (Phase 2+) truncates events for runs older than 30 days, keeping the run + step rows for historical aggregation.

Transaction boundaries

State transitions are committed before the next step runs. The sequence for a healthy run is:

1. Client POST /runs → row inserted state: pending, river job enqueued, txn commits.
2. Worker claims job (SKIP LOCKED), updates run to state: running, worker_id set; txn commits.
3. For each step:
   • Insert playbook_run_steps row state: running; txn commits.
   • Executor runs; events are written to playbook_run_events as they arrive (single-row inserts, no txn per event — batched).
   • On success: update step to completed, merge outputs into playbook_runs.outputs; txn commits.
4. Update run to completed; emit run_completed event; txn commits.

The invariant: at every commit boundary, the run row and its step rows are internally consistent. A replica crash mid-step leaves the run in state: running with a running step whose worker_id matches the dead replica; restart recovery (§2) detects and repairs.

6. Streaming protocol (SSE)

Endpoint surface

POST   /api/v1/playbook-runs
GET    /api/v1/playbook-runs/{id}
GET    /api/v1/playbook-runs/{id}/events         # SSE
POST   /api/v1/playbook-runs/{id}/input
POST   /api/v1/playbook-runs/{id}/cancel
GET    /api/v1/playbook-runs?kind=…&status=…    # list, paginated

All endpoints sit behind the existing ai-research auth (session cookie + CSRF, per ADR-036). The SSE endpoint additionally accepts a short-lived query-string token for the browser's EventSource (which cannot send custom headers) — the pattern we already use for distill metrics (ADR-050) and will formalise in ADR-069.

POST /playbook-runs

{
  "playbook": "service_runbook:alertmanager-debug",
  "inputs": {
    "alert_id": "alert-0xABCD",
    "service": "grafana-server"
  },
  "idempotency_key": "client-chosen-uuid"
}

Returns:

{
  "run_id": "0195b3ee-…",
  "state": "pending",
  "events_url": "/api/v1/playbook-runs/0195b3ee-…/events"
}

Idempotency: an identical (actor, playbook, inputs, idempotency_key) tuple within 5 minutes returns the existing run_id instead of creating a second row.

GET /playbook-runs/{id}/events — SSE stream

event: run_started
data: {"run_id":"0195b3ee-…","playbook":"service_runbook:alertmanager-debug","actor":"@jon","ts":"2026-04-23T14:00:00Z"}

event: step_started
data: {"step_id":"dump-disk","step_index":0,"type":"shell","target":"host:ops-vps","ts":"2026-04-23T14:00:01Z"}

event: stdout
data: {"step_id":"dump-disk","line":"Filesystem      Size  Used Avail Use% Mounted on","ts":"2026-04-23T14:00:01.123Z"}

event: stdout
data: {"step_id":"dump-disk","line":"/dev/sda1       100G   84G   12G  88% /","ts":"2026-04-23T14:00:01.124Z"}

event: step_completed
data: {"step_id":"dump-disk","exit_code":0,"outputs":{"stdout":"…","exit_code":0},"ts":"2026-04-23T14:00:01.250Z"}

event: step_started
data: {"step_id":"ask-api-key","step_index":1,"type":"prompt_secret","target":"local","ts":"2026-04-23T14:00:01.251Z"}

event: awaiting_input
data: {"step_id":"ask-api-key","question":"Paste the API key to store in the vault.","schema":{"type":"string"},"ts":"2026-04-23T14:00:01.252Z"}

…client POSTs /input…

event: step_completed
data: {"step_id":"ask-api-key","outputs":{"secret_id":"vlt_01J…"},"ts":"2026-04-23T14:00:05.000Z"}

event: run_completed
data: {"run_id":"0195b3ee-…","state":"completed","outputs":{…},"ts":"2026-04-23T14:00:10.000Z"}

Event ids are monotonic per run (Last-Event-ID header supported for reconnect). A client reconnecting within the retention window replays events strictly greater than its last seen id — no missed output, no duplicate output.

POST /playbook-runs/{id}/input

{
  "step_id": "ask-api-key",
  "value": "sk-ant-…"
}

Returns 204 No Content on success. The runtime validates value against the step's schema: and, on success, resumes the worker slot (the prompt_secret executor's Execute returns from its block-on-channel). On schema failure, returns 400 with the validation error; the run remains awaiting_input so the UI can re-prompt without a new step id.

Why SSE and not WebSocket

SSE is one-way (server → client) plus a separate POST for the client → server direction. That asymmetry matches our data flow exactly — we stream a lot of lines out; we send a small number of replies in. WebSocket gives us a bidirectional channel we would half-use, plus framing, plus proxy quirks (Caddy handles SSE trivially; WS needs header forwarding and timeout tuning).

SSE also degrades to a plain curl — a developer investigating a hung run can literally curl -N /events and read the stream. WebSocket cannot do that without a dedicated client.

7. Target restriction — target: on every step

Every step declares where it runs. There is no implicit default.

target: local
target: host:<alias>         # alias resolves via gyrum-catalog
target: container:<image>    # runs inside a disposable container (Phase 4+)

This is enforced at parse time, not runtime. A playbook missing a target: on any step fails YAML → struct decoding with a specific error. The operator cannot accidentally ship a playbook that runs bash on the platform host because they forgot a field.

Why this matters. target: is the hook that ADR-069 will hang its allow-lists and audit annotations on. A step that declares target: local is asking for the most privileged execution context available; that declaration is explicit, reviewable in the playbook file, and recorded in playbook_run_steps.target for every run.

target: local in Phase 1. Allowed, because the walking skeleton needs it. Gated by a runtime-wide allow-list:

// ai-research config
runtime.local_execution_enabled: true | false
runtime.local_execution_allowlist: [release-flow:gyrum-complete-pr, …]

Before ADR-069 lands, the allow-list is operator-managed and tiny (one or two playbooks). After ADR-069, the allow-list mechanism is replaced by the full security model (signed playbooks, role-based permissions, secret-vault reads).

target: container:<image>. The Phase 4 answer to "shell playbooks need to run untrusted code". A disposable container (Docker, or firecracker via a future abstraction) is the sandbox. Spec-ed here; not built in Phase 1.

8. Event payload shapes (full reference)

All SSE data: fields are JSON. Fields marked ? are optional.

The event schema is versioned: a v: field at the top level, v: 1 for this ADR, bumped on any breaking change. The frontend-observ SDK already uses this pattern (ADR-053).

9. YAML spec + JSON-Schema validator

The spec is YAML — it matches the playbook front-matter convention (ADR-067), it is diff-reviewable, and humans write it.

The source of truth is a Go struct. go generate walks the struct and produces a JSON-Schema artifact committed under schemas/playbook-runtime.schema.json. Consumers:

• CI runs a YAML linter that validates every playbook's runtime section against the schema. Broken spec fails the PR.
• Editor LSP (a thin wrapper around redhat.vscode-yaml) autocompletes field names and flags unknown fields inline.
• The runtime itself uses the Go struct directly (no schema-at-runtime parsing; the struct is the parser).

Struct sketch (abridged):

type Playbook struct {
    // ADR-067 front-matter fields — kind, id, title, persona, owner, etc.
    FrontMatter PlaybookFrontMatter

    // ADR-068 runtime section — present when the playbook is executable.
    Runtime *PlaybookRuntime `yaml:"runtime,omitempty"`
}

type PlaybookRuntime struct {
    Inputs  []InputSpec  `yaml:"inputs"`
    Steps   []Step       `yaml:"steps"`
    Outputs []OutputSpec `yaml:"outputs,omitempty"`
    Timeout Duration     `yaml:"timeout,omitempty"` // whole-run cap
}

type Step struct {
    ID       string         `yaml:"id"`
    Type     string         `yaml:"type"`              // shell / ssh / claude / …
    Target   string         `yaml:"target"`            // REQUIRED
    With     map[string]any `yaml:"with,omitempty"`    // templated
    When     string         `yaml:"when,omitempty"`    // expression
    SaveAs   string         `yaml:"save_as,omitempty"`
    OnError  string         `yaml:"on_error,omitempty"`// fail / continue / default_to:…
    Timeout  Duration       `yaml:"timeout,omitempty"`
    // Kind-specific fields are carried in a sidecar map and validated
    // by the executor's Validate(step) at parse time.
    Raw      map[string]any `yaml:",inline"`
}

The Raw sidecar is how per-executor fields (run:, prompt:, url:, playbook:) ride without the top-level struct knowing them. The executor's Validate() method decodes Raw into its own kind-specific struct and errors if fields are missing, unknown, or mistyped.

Templating: we use a tiny, audited expression language (Go's text/template plus a sprig subset, explicitly whitelisted functions). No arbitrary code execution inside templates. Expression evaluation is pure, no I/O.

10. Frontend integration

In scope for this ADR: the contract. The rendering implementation details are ADR-070.

A Svelte component consumes the SSE stream:

<PlaybookRunner
  playbookId="service_runbook:alertmanager-debug"
  initialInputs={{alert_id, service}}
  on:completed={(e) => navigateToOutputs(e.detail.outputs)}
/>

Internally the component:

1. POST /playbook-runs → receives run_id + events_url.
2. Opens new EventSource(events_url).
3. For each stdout / stderr event, appends a line to a terminal pane. ANSI colours are rendered by anser — a ~4KB library that produces HTML from ANSI-escaped text. Not xterm.js — it is ~300KB, brings a full VT emulator we do not need, and its DOM shape fights Svelte's reactivity.
4. For each awaiting_input event, renders a form derived from the step's JSON-Schema fragment; the operator's submit becomes POST /input.
5. For run_completed, emits a Svelte event upward so the host page can react (navigate, refresh a list, display a report).

The component ships first inside ai-frontend. Once it has two independent callers (the Platform UI runbook page and the Ops UI release page), it migrates to gyrum-ui as <gy-playbook-runner> per ADR-062's "prove then lift" pattern.

The frontend never touches the database, never has an executor, never orchestrates. Its job is: show progress, collect replies, display the final report.

11. Observability

Metrics (via distill, ADR-050 / ADR-053):

Cardinality: playbook_id is bounded by the catalogue (≤ a few hundred long-term); actor is bounded by team size. Both safe per ADR-005.

Structured logs: every worker loop logs slog.With("run_id", run_id, "step_id", step_id, "playbook", kind+":"+id) once per run claim. Every step emission also includes those fields. A grep run_id=<uuid> against the ai-research logs yields the full story of any run.

Tracing: OpenTelemetry spans — playbook.run wraps the whole run, playbook.step wraps each step, executors optionally add kind-specific spans (e.g. claude.sdk.call). Propagation from the originating HTTP request is automatic.

12. Claude-step safety

Claude steps deserve a dedicated section because they fail differently from every other step.

Slow. A single call is 2-10s typical, 30s tail. Several calls in a sequence compound. The step-level timeout: defaults to 30s; run-level timeout: defaults to 10 minutes.

Non-deterministic. Two identical prompts can produce different outputs. on_error: is therefore required on every type: claude step; the author cannot implicitly assume "the model will get it right".

Rate-limited. The SDK wrapper handles 429s with exponential back-off, up to the step's timeout. If the timeout is hit, the step fails and on_error: takes over.

JSON mode. expects: json triggers the SDK's JSON mode plus a json.Unmarshal into map[string]any. Parse failure is a typed error (ErrClaudeJSONParse) distinct from a timeout or an API error, so on_error: default_to: can target it specifically.

Prompt injection. Outputs of earlier steps flow into the prompt: template via {{ outputs.X.stdout }}. Untrusted input means a shell output can contain "ignore previous instructions…". Mitigations:

• The Anthropic SDK wrapper wraps user-templated content in an explicit <untrusted>…</untrusted> delimiter and prepends a system prompt that tells the model to treat that content as data, not instructions.
• For type: claude steps inside a playbook whose target: history includes a shell run, ADR-069 will require a trust: untrusted marker that downgrades the model's allowed tools.

No tool use in Phase 1-3. A type: claude step is a one-shot prompt → text / JSON. Tools (code execution, web fetch, file access) are a Phase 4+ feature with its own ADR; the target: field will be how we gate per-tool permissions.

13. Mapping existing artifacts

The primitive lands on a live fleet. Nothing is thrown away.

The rule: every existing piece becomes either a step executor, a playbook spec, or both. No parallel system.

14. Phased build plan

Phase 1 is deliberately the smallest useful thing — one playbook, one step type, synchronous. The point is to de-risk the contract end-to-end before we pay for the queue infrastructure. Phase 2 adds the queue; everything after is additive.

15. Consequences

Harder.

• New service surface in ai-research. The runtime is not small: ~6 HTTP endpoints, ~10 executors, a state machine, a queue integration, SSE pump, event retention. We budget this explicitly and use the phased plan to keep each PR ≤ 800 lines.
• Postgres is load-bearing for run state. The existing operational disciplines (nightly backup, restore drill per ADR-020 disaster-recovery) now cover a new kind of data. Retention policy: events 30 days, runs + steps indefinitely, PII never present in either (secrets go to the vault, not the row).
• Every new screen action is a YAML + executor pair. A developer who wanted to add a "click here to restart Grafana" button now writes a playbook and maybe an executor, not a bespoke backend handler. This is the point (uniform audit, uniform observability, uniform pause-for-input) but it is friction in the short term.
• Composition via subplaybook risks infinite recursion. Max depth is 5, enforced at parse time by walking the related: + subplaybook: graph. Cycles fail parsing loudly.
• Arbitrary shell is a huge security surface. We name it, we gate it behind target: local, we gate that behind an allow-list, and we fence the hardening in ADR-069. Production playbooks with a shell step do not ship before ADR-069 lands.
• SSE has a concurrency footprint. One open SSE connection per active run per viewing client. ai-research's HTTP server (net/http with a large ulimit) handles thousands. We revisit if we ever need tens of thousands of concurrent viewers, which is not a 2026 problem.
• Retrofit cost. The migration (§13) touches the import wizard, ~316 experiments, the release flow, and every runbook that wants a runtime section. ADR-067's 30-40 PR migration plan now has a sibling runtime-section migration plan of similar size.

Easier.

• One control plane for operational work. "Run the onboarding flow", "replay exp-315 on a fresh VM", "execute the release flow" all go through the same runtime, the same SSE stream, the same terminal pane, the same metrics. An operator learns one thing.
• Frontend becomes thin and forgettable. New operations do not require frontend work beyond pointing <PlaybookRunner> at a new id. Design reviews shrink.
• Agents and humans share a substrate. A Claude agent that needs to "restart the service" POSTs to the same endpoint a human uses. Audit logs don't need to distinguish "human ran this" vs "agent ran this" — actor is either.
• The gaps KPI gets teeth. ADR-067's "% playbooks with known gaps" becomes "% playbook runs that completed successfully". Trend data on actual operational health replaces trend data on self-reported gap counts.
• Interactivity is first-class. "What hostname should I use?" mid-run is a prompt step, not a back-and-forth in Slack. The playbook is the transcript.

What we sign up to maintain.

• The StepExecutor interface. Adding an executor is easy; changing the interface is a breaking change across every executor we have shipped. We budget interface churn to once per year, planned.
• The SSE event schema. Versioned; bumping v: is an ADR.
• The Postgres schema. Standard migration discipline (one forward migration per PR, never squash the history).
• The max depth of 5 for subplaybook. Lowering it is easy; raising it requires demonstrated need.
• The 30-day event retention. Long-term audit requires a separate append-only store (ADR-038 audit sink, ADR-039 hash chain) — ADR-069 will detail how playbook events flow into it.

16. Alternatives considered

• Temporal (full workflow engine). Rejected. It is the right tool if we had dozens of long-running, cross-service workflows with sophisticated retry, versioning, and migration needs. We have one service and a walking skeleton. Temporal shapes your code around its SDK (every workflow is a specific function signature); we would be paying that tax from day one for capabilities we will not use until Phase 5+. The evaluation trigger to revisit: if Phase 4 cross-service playbooks outnumber single-service playbooks, or if the Postgres queue hits its ceiling.

• NATS (or any message broker). Rejected, for now. NATS becomes attractive when playbook events matter to other services — a dashboard in grafana-server subscribing to playbook.*.completed, a metrics-compute service reacting to failures. Until then, Postgres NOTIFY + our existing metrics pipeline cover the same ground. We re-evaluate at Phase 4.

• Redis Streams. Rejected. We do not run Redis. Adopting it for this is a new operational concern; river on the existing Postgres is not.

• In-memory queue only (no durability). Rejected. Any run longer than ~30s is vulnerable to a deploy. We deploy ai-research frequently.

• WebSocket for the bidirectional channel. Rejected. The channel is asymmetric (many events out, few replies in); SSE + POST reflects that asymmetry cleanly and degrades to curl. WebSocket requires us to design a client-to-server protocol (message kinds, reconnect semantics) that POST already gives us.

• Long-poll. Rejected. Strictly worse than SSE on every axis (latency, reconnect, proxy behaviour). Its only advantage — working through strict proxies that strip SSE — is not a problem we have.

• Frontend-driven execution (browser runs the steps). Rejected, by the operator directly:

  i dont want the frontend doing any dirty work

  Additionally: frontend-driven execution puts secrets in the browser, cannot ssh, cannot run bash, cannot be trusted by third-party services that need a stable origin. The browser is a UI, not an orchestrator.

• claude -p CLI shellout vs direct Anthropic SDK. Rejected the CLI. Shelling out to a CLI adds a subprocess, requires a binary in the ai-research image, loses token-level streaming, and moves secrets into process argv / env. The SDK call is in-process, streams, and keeps secrets confined to a credentials.Provider in memory.

• JSON spec instead of YAML. Rejected. Authors write these. YAML's multi-line strings (for run: and prompt:) and comments are load-bearing. JSON would force us into escaped string blobs for every shell command and every prompt.

• HCL / custom DSL. Rejected. YAML is the format every other gyrum primitive uses (ADR-013 templates, ADR-018 product.yml, ADR-023 alerts, ADR-067 playbooks). One more format is one more thing for authors to learn and for tooling to support.

• Standalone gyrum-playbook-runtime service from day one. Rejected. A new service means a new deploy pipeline, a new on-call rotation, a new auth integration, a new metrics scrape — on top of the runtime implementation itself. Embedding in ai-research, where the operator UI already lives, shortens the walking-skeleton path by weeks. The operational cost of extraction later is bounded (the runtime is already a module with a clean boundary — the HTTP handlers, the executors, the queue integration); the operational cost of a premature extraction is unbounded. ADR-071 will pull the trigger when any of: (a) a non-ai-research service wants to run playbooks, (b) runtime load threatens ai-research's response-time SLOs, or (c) the security model wants to run the runtime in a different trust boundary than the main UI.

17. Follow-on ADRs

• ADR-069: Playbook security model. Who can run what (role → playbook permission). Secret vault (where prompt_secret values live, how type: http and type: shell reference them). Destructive-step annotation (destructive: true in the YAML; requires an approval step before the destructive step; the pair is enforced at parse time). Audit chain (every run writes into the ADR-038 audit sink with ADR-039 hash-chain integrity). Signed playbooks (only playbooks from a trusted repo can carry target: local). Idempotency keys and destructive-step deduplication.

• ADR-070: Playbook UI contract. The frontend binding. Exact Svelte component API, form-rendering rules from JSON-Schema, terminal pane rendering with anser, reconnect UX, error UX, cancel UX, run-history list, drill-through to a completed run's events. Ports to gyrum-ui as <gy-playbook-runner>.

• ADR-071: Playbook runtime extraction. The trigger conditions (§16 above), the extraction plan (lift the runtime/ package, the HTTP handlers, and the river tables into a new repo), the cutover (blue-green on the HTTP surface, Postgres stays shared until the security model tightens), the operational hand-off (dashboards, alerts, runbooks).

• ADR-078: Trigger-driven playbook orchestration. The front door to this runtime — how a playbook_run gets created without a human clicking. Additive triggers: YAML block; cron / webhook / alert kinds (and a deferred polled_change); migration 003 adds trigger_kind + trigger_id to playbook_runs. Complementary to ADR-077 (agent coordination) — triggers fire the start, approvals gate the middle.

Migration path

Shippable in the small-PR rhythm ADR-067 already uses.

Steps 1-4 are in-scope for this ADR's shipping cohort. Steps 5-11 are tracked follow-ups on the ai-research board. Step 12 is explicitly gated.

Review cadence (for this ADR)

Re-review every 90 days (next: 2026-07-22) against:

• Runtime throughput — are we queue-bound, CPU-bound, or executor-bound? Does the "revisit Postgres queue" trigger from §5 fire?
• Failure modes in the wild — restart recovery (§2), back-pressure (§3), SSE reconnect (§6). Have the designed behaviours held?
• type: claude reliability — actual p50/p95/p99 latencies, error rates, rate-limit incidence. Does §12 need tightening?
• Security hardening — has ADR-069 landed on schedule? If not, what production playbooks with target: local are at risk?
• Extraction pressure — does ADR-071 want to fire?

Stale flag on the Owner dashboard once last_reviewed on this ADR passes 90 days.

References

• ADR-067 — playbooks unified primitive (the data model this runtime executes)

• ADR-064 — gyrum-catalog library + hex architecture; the SSH Executor port backs type: ssh

• ADR-062 — design system; <PlaybookRunner> follows the "prove in ai-frontend, lift to gyrum-ui" pattern

• ADR-061 — gyrum-pipelines library; backs type: pipeline

• ADR-060 — catalog-driven infrastructure; host aliases in target: host:X resolve against the catalog

• ADR-053 — frontend observability; client-side SSE errors flow through /client-log

• ADR-050 — bearer auth for product metrics; the short-lived token pattern for SSE auth borrows this shape

• ADR-046 — tier-1 discipline dashboard pack; where the runtime's dashboard lives

• ADR-039 — audit hash chain; tamper-evidence for playbook-run events (ADR-069 wires this in)

• ADR-038 — audit sink; long-term home for run history

• ADR-036 — CSRF double-submit cookie; applies to POST /playbook-runs and POST /input

• ADR-023 — alerts as code; the runtime's failure alerts are written here

• ADR-020 — shared infra vs per-product isolation; Postgres backup/restore now covers playbook run state

• ADR-005 — cardinality labels vs fields; the metric labels in §11 are vetted against this

• Operator framing, 2026-04-23:

  take each playbook, giving it a guide and using the experiments and pipeline as the actual running, introducing text input and a report at the end… everything else I suppose action wise leads you to that point, even actions on screens will really be dealing with these playbook pages behind the scenes

  1 yaml 2 probably best on the server, i dont want the frontend doing any dirty work, this may even mean a message bus to handle them and they return a job no and provide status whilst they work, but id also like the to be interactive ith the user, suppose we had a install grafana and it needed some expra answers the frontend user, it would be helpful if they could send back answers, i suppose if the backend ran bash, so might have claude -p, i like when you deploy you show the termianl output

Supersedes: none Superseded by: leave blank until a later ADR reverses this one