ADR-166: ai-research splits along an executor-isolation boundary; the queue is the seam

Status: Proposed Date: 2026-05-07

Related: ADR-092 (host/project separation), ADR-109 (type: claude step + observer: field), ADR-110 (observe-and-file dominant pattern), ADR-111 (broker as model-routing layer), ADR-161 (auto-deploy fires on workflow_run completion), ADR-163 (self-hosted-only fleet CI), warp#186 (deploy-drift / auto-deploy epic), warp#1556 (workflow_run trigger swap — PR#847)

TL;DR

ai-research today is one binary holding every fleet secret (SSH, Hetzner, GitHub PATs). The operator wants to put it on the internet but the secret surface is too big. This ADR splits the binary along a queue boundary: a thin receiver binary (internet-facing, no deploy secrets) evaluates ADR-161's gate cascade and files a type: deploy_request warp ticket; an executor binary (operator hardware, holds SSH + cloud + ansible creds) polls warp, claims atomically, validates against the project manifest one more time (§4), and fires deploy-project. The queue is the seam. ADR-161's trigger semantics are preserved verbatim — only the location of evaluation changes.

The executors are pull-based agents that look exactly like self-hosted GitHub Actions runners. Multiple instances can run on operator-controlled hardware (laptop today, office machines tomorrow, a dedicated trusted server later); whichever is online claims work. The same pattern extends naturally to CI in a follow-on initiative — see dark-factory/docs/features/warp-as-fleet-ci-queue.md.

Architecture at a glance

flowchart TB
    subgraph Internet["Internet-facing perimeter (no deploy secrets)"]
        gh[GitHub workflow_run.completed]
        rcv[Receiver binary<br/>ADR-161 gate cascade<br/>HMAC + filer token]
    end

    subgraph Queue["Warp queue — the seam"]
        wq[type: deploy_request<br/>idempotent on event_id]
    end

    subgraph Trusted["Operator hardware (SSH, cloud, ansible)"]
        exec[Executor binary<br/>§4 verification<br/>polls + claims]
        play[deploy-project playbook]
    end

    gh -->|webhook| rcv
    rcv -->|POST ticket| wq
    wq -->|claim| exec
    exec -->|fire| play

    classDef perim fill:#fff3cd,stroke:#856404
    classDef seam fill:#d1ecf1,stroke:#0c5460
    classDef trust fill:#d4edda,stroke:#155724
    class rcv,gh perim
    class wq seam
    class exec,play trust

[!IMPORTANT] Key invariant: the executor never trusts the queue. §4's verification protocol (source allowlist, schema validation, manifest re-evaluation, image-existence check) makes a queue compromise survivable — an attacker who captures a filer token can file tickets, but can't make the executor fire arbitrary code.

Context

ai-research today is one Go binary running on the operator's laptop. It does five jobs at once:

1. Webhook intake — receives GitHub events (push, workflow_run, etc.); evaluates ADR-161's gate cascade.
2. API server — serves /api/projects/*, /api/version, etc. for the UIs.
3. Playbook executor — runs ansible against fleet hosts via SSH; fires deploy-project, destroy-host, provision-and-deploy.
4. Cloud control — talks to Hetzner Cloud, snapshots images, manages VMs.
5. Approval gate — operator-in-the-loop for risky operations.

The operator's stated goal is to make ai-research available on the public internet — both so the receiver path doesn't depend on a cloudflared tunnel into a single laptop, and so the API can serve more clients than today's local-only consumers. The blocker is structural: today's single binary holds every secret in the fleet at once. SSH private keys for prod hosts. Hetzner Cloud API tokens. GitHub PATs scoped to issue deploys. Database credentials for the warp Postgres. A compromise of the binary running on the public internet would mean full fleet root.

The structural fix the operator's intuition is reaching for is correct: not all five jobs need the same security perimeter. (1) and (2) are read-or-receive surfaces that don't need any of the dangerous secrets; they need the GitHub webhook signing secret and read-only state access. (3), (4), and (5) carry the blast radius — they need every credential the fleet has and they fire irreversible operations.

ADR-161 just landed (today, 2026-05-06; PR#847 against ai-research is the receiver implementation). It pins what triggers a deploy: a workflow_run.completed event for release.yml on the default branch with conclusion=success, opted-in via the project's deploy manifest, gated by an actor-based loop guard plus a sliding-window dedupe. ADR-161 is correct and stays. The receiver binary inherits PR#847's code verbatim; nothing is undone. What this ADR addresses is orthogonal to ADR-161: not what triggers a deploy, but where the evaluator and the executor live, and what the auth seam between them looks like.

The seam this ADR draws is a queue. The fleet already runs warp as the durable intent queue (HTTP service, Postgres-backed, atomic claim semantics, heartbeat protocol — see gyrum-labs/warp/docs/AGENT_GUIDE.md). Today every agent in the fleet picks work off warp's queue. Adding a deploy_request item kind extends the same primitive into the deploy path. The executor becomes one more agent claiming items off the queue.

Decision

ai-research splits into two binaries — ai-research-receiver (internet-facing) and ai-research-executor (operator hardware) — communicating only through a deploy_request ticket on the warp queue. The receiver holds the GitHub webhook signing secret and a warp filer token; nothing else. The executor holds SSH keys, cloud tokens, and ansible credentials; it is never internet-reachable. ADR-161's trigger semantics are preserved verbatim — they live in the receiver's evaluator code (PR#847), not relocated.

The shape has six parts: the queue ticket schema, the receiver's responsibilities, the executor's responsibilities, the verification protocol that closes the queue's tampering surface, the migration path from today's single binary, and the failure-mode contract.

1. Queue ticket schema

A new ticket kind on warp: type: deploy_request. Filed by any trusted filer; consumed only by the executor.

kind: ticket
type: deploy_request
title: "deploy <slug> <version_tag>"
repo: gyrum-labs/<slug>
priority: high
status: ready
spec:
  slug: <project-slug>             # e.g. "warp"
  image_tag: <semver-or-sha>       # e.g. "v0.3.47" — receiver-resolved, not from the wild
  target_sha: <full-git-sha>       # informational; for operator notification
  triggered_by:
    filer: <receiver|gha|operator-cli>
    event_id: <github-delivery-id-or-equivalent>  # idempotency key
    received_at: <iso-8601>
  manifest_version: <hash-of-deploy-manifest-at-fire-time>

Schema rules:

• The triggered_by.event_id is the idempotency key. The receiver and executor both deduplicate on (slug, event_id) so a redelivered GitHub webhook does not file two tickets and a re-claimed ticket does not run twice.
• manifest_version pins the deploy manifest snapshot the receiver evaluated against. The executor re-validates against the live manifest before firing — if the manifest moved between filing and firing, the executor refuses with a structured error and re-files the ticket as blocked for operator review.
• Required fields are exactly the structured ones above; the body field carries human-readable rendering for the warp UI but the executor never parses prose.

2. Receiver responsibilities

The receiver is the internet-facing perimeter. PR#847's webhook handler code becomes the receiver binary's code with two changes:

• After all of ADR-161's gates pass (action, workflow.path, head_branch, conclusion, manifest opt-in, master switch, loop guard), instead of calling firer.Fire(), the receiver POSTs a deploy_request ticket to warp.
• Every gate evaluation result is written to the receiver's audit log (receiver-decisions.jsonl) regardless of whether it fires — including drops. The audit trail is the operator's "why did/didn't this fire" surface.

What the receiver holds:

• GitHub webhook signing secret (HMAC verification)
• A warp filer token, scoped to "POST /items only" with a filer:receiver tag
• Read-only access to deploy manifests (via gyrum-catalog HTTP endpoint or local cache, not direct file read)

What the receiver does NOT hold:

• SSH private keys
• Hetzner Cloud or any other cloud-provider tokens
• Database credentials for the warp Postgres (it talks to warp via HTTP API)
• Ansible inventory passwords

The receiver is stateless except for ADR-161's dedupe LRU. It's a small Go binary (estimate: PR#847's code plus ~150 lines of warp-filer logic) deployable as a single container on any small box. Restartable; restart loses the LRU but GitHub's redelivery semantics already cover that gap.

3. Executor responsibilities

The executor runs on operator hardware (laptop today; potentially a dedicated trusted server with hardware-key auth later — see §6). It polls warp for type: deploy_request items in status: ready, claims atomically per the standard agent protocol (SELECT … FOR UPDATE SKIP LOCKED semantics warp already provides), and dispatches to the existing playbook firer.

Polling cadence: 10s default, env-tunable. Operator-side: the warp dashboard's "ready deploy_request count" makes drift visible if the executor stops polling.

What the executor holds:

• SSH private keys for prod hosts
• Hetzner Cloud API token
• Cloud-provider credentials (AWS, GCP later if needed)
• A warp consumer token, scoped to "claim/heartbeat/complete on type: deploy_request only"
• Read access to gyrum-catalog deploy manifests (for the §4 verification step)

The executor never receives webhooks, never holds the GitHub webhook signing secret, never has an HTTP listener facing the public internet. The only inbound connection it accepts is from the operator's terminal for ad-hoc commands — which is the same as today.

4. Verification protocol — the executor doesn't trust the queue

The queue is shared infrastructure. A compromised warp token (any filer with write access) could file an arbitrary deploy_request. The executor is the last line of defense, not the queue:

• Source check. Each deploy_request carries triggered_by.filer. The executor maintains an allowlist of trusted filers; tickets from unknown filers are refused with status: blocked and a structured note. Today's allowlist: receiver, gha, operator-cli.
• Schema validation. image_tag must match the project's allowed pattern (semver, sha-prefix, or branch name from a configured list). slug must exist in gyrum-catalog. The executor refuses unknown projects.
• Manifest re-evaluation. The executor re-fetches the live deploy manifest at claim time. The ticket's manifest_version is a hint; if the live manifest disagrees, the executor refuses (the receiver's claim about "this slug opted into auto-deploy" must hold at fire time, not just at filing time).
• Image-existence check. Before firing, the executor confirms ghcr.io/gyrum-labs/<slug>:<image_tag> exists. A deploy_request for a non-existent image is refused with a structured note — closes the "filer specified a typo'd tag" failure mode.
• Audit log. Every claim, every validation outcome, every fire (or refusal) is appended to executor-decisions.jsonl. The operator can replay any incident from this log without rebuilding state.

These checks make a queue compromise survivable: an attacker who captures a filer token can file a deploy_request, but they can't make the executor fire a deploy of arbitrary code. The image must already exist in GHCR, the slug must already be opted in, and the executor's local manifest cache must agree.

5. Migration from today's single binary

Three steps over a defined window. None of them rip out PR#847's code; the receiver binary inherits it.

1. Phase 0 — audit (warp follow-up). Classify every ai-research API endpoint and webhook handler by what secrets it touches. Tag handlers in source as // safe: read-only or // dangerous: ssh-required. Identify destroy-* and other irreversible playbooks; each gets type: pause per ADR-109/110 so even an executor-side compromise can't destroy hosts without operator confirmation. Output: a markdown table in dark-factory/docs/runbooks/ and a code annotation pass on ai-research. Ships before any binary split.
2. Phase 1 — split the binary. Extract the receiver code from ai-research/cmd/server/webhooks/ into ai-research/cmd/receiver/. Add the warp-filer step at the post-gate-cascade dispatch point. Extract the executor's playbook-firer into ai-research/cmd/executor/. Add the polling loop, claim logic, and §4 verification. The two binaries share the existing internal packages (internal/autodeploy, internal/playbookfire); only the entrypoints split. Run both new binaries in parallel with the old monolithic binary for one week of soak; tickets fire via warp queue but the legacy webhook→playbook path also runs as a redundant safety net (idempotent via assert_sha_changed, so double-fires no-op the second one). Operator confirms the queue path is winning every race; legacy path is then disabled.
3. Phase 2 — receiver moves to a host. Once the queue path is the canonical fire mechanism, the receiver binary deploys to a tiny internet-facing host (a $4/month Hetzner CAX11 is sufficient — receiver is stateless and tiny). The cloudflared tunnel comes down. The receiver is now a real internet service; the executor stays on operator hardware. The factory.gyrum.ai UI stops talking to ai-research over the tunnel and instead talks to the receiver's read API (and to warp directly for state). End-state: operator's laptop holds executor + the SSH/cloud secrets; everything else is internet-reachable on its own perimeter.

The migration order is intentional: Phase 1 is a refactor with a working safety net; Phase 2 is a deployment change with no logic change; Phase 0 is the audit that informs both. Phase 0 is independent and can ship first.

6. Failure-mode contract

What goes wrong, and what catches it:

• Receiver is offline. GitHub retries the webhook per its delivery contract. ADR-161's polling watcher (warp#191 r5) is the second line. The receiver coming back catches up via the LRU's reset; redelivered webhooks deduplicate on event_id.
• warp queue is offline. Receiver's POST fails; it retries with backoff. Eventually the receiver's filer-side dedupe LRU prevents re-firing for the same event_id once warp comes back. Polling watcher backstops within 60s.
• Executor is offline. Tickets pile up in status: ready on warp. The factory dashboard's release-status pane (Phase 1 of warp#186) shows the gap: live SHA stays behind main HEAD. Operator-visible. The executor coming back claims and fires in queue order; if multiple deploy_requests for the same slug pile up, the executor takes the most-recent one (max(triggered_by.received_at) per slug) and refuses the older ones with a status: superseded note — closes the "ten queued deploys all run when laptop comes back online" thrash.
• Receiver is compromised (worst-case). Attacker can: file deploy_request tickets that look legitimate. Cannot: fire a deploy of arbitrary code (executor §4 checks block: image must exist, slug must be opted in, manifest must agree). Cannot: read SSH keys (not present on receiver). Damage bounded to "deploys of code that already exists in GHCR" — same blast radius as a stolen GitHub PAT, which is the existing trust boundary.
• Executor is compromised. Same as today; the executor host has been the security boundary all along. The split doesn't worsen this.
• A deploy_request carries a malicious image_tag. Executor §4 rejects it (image-existence check + tag-pattern validation). The malicious filer cannot push to GHCR without org-level access, which is its own boundary.
• Manifest moved between filing and firing. Executor refuses; ticket goes to blocked; operator sees it on the dashboard. Cost: a deploy that was filed seconds before a manifest flip is missed; the next push re-files. Acceptable.
• Two filers race on the same event. Idempotency key is (slug, event_id). Warp rejects the second filer's POST. No double-deploys.

What does not change: the operator's manual-deploy escape hatch. Today the operator clicks a button to fire deploy-project. Tomorrow the same button files a deploy_request with triggered_by.filer: operator-cli. Same UI, same outcome, just the canonical path.

Consequences

What becomes easier:

• The receiver perimeter holds zero deploy secrets. An audit of "what does the public-internet-facing process have access to" returns a tiny answer: GitHub webhook secret, warp filer token, deploy-manifest read access. That's the security improvement the operator is reaching for.
• Executor logic is testable in isolation — receiver mock fires a deploy_request into a test warp instance, executor processes it, assertions on the playbook-fire side. Today's webhook-coupled code path is harder to mock.
• The queue is the audit trail. Every deploy attempt, decision, and fire is a warp ticket with a UUID and a structured event_id. "Why did warp deploy at 14:32?" answers from the queue without log spelunking.
• Adding a new filer is bounded: it needs a warp filer token and the right ticket schema. No code changes to the executor. Phase 2's GHA-as-additional-filer (alternative considered §below) becomes a one-PR add later if wanted.
• The executor's polling loop generalises. A future "deploy this manually scheduled cron" ticket, a future "rollback to previous version" ticket, and today's "deploy on release" ticket all flow through the same primitive. The executor doesn't grow new code paths per trigger source.

What becomes harder:

• Two binaries instead of one. CI builds both; release.yml ships both image artifacts; deploy-project for ai-research itself becomes deploy-receiver + deploy-executor. The infra around ai-research's own self-deploy doubles. Mitigation: gyrum-catalog deploy manifest gains a companion_images: field; the existing deploy machinery handles the multi-image case. (One-time work, not recurring.)
• The verification protocol (§4) is non-trivial. Each check is a few lines, but together they're an attack-surface review. Integration tests cover the rejection paths; a regression here would silently widen blast radius. Mitigation: pin the §4 checks in a structured test corpus that lives next to the executor; CI fails if any §4 check is removed without an ADR amendment.
• Latency increases. Today: webhook → ai-research → playbook fire is ~100ms. Tomorrow: webhook → receiver → warp POST → executor poll cycle (up to 10s) → manifest re-fetch → playbook fire. Per-deploy overhead is ~5-10s of additional latency. Deploys take minutes; this is rounding error. Operator-perceived "why is my deploy slower" should not surface; if it does, the polling cadence is a knob.
• Soak window cost (Phase 1, one week). Both code paths run in parallel; doubles webhook receiver invocations, doubles audit log volume during soak. Acceptable: the safety net is the point.
• A new failure class — the receiver and executor disagreeing on what's deployable. §4 makes the executor authoritative, but a misconfigured receiver could file tickets that the executor uniformly rejects, producing the "no deploys but no errors" mode. Mitigation: the receiver's audit log records every fire; the executor's audit log records every refusal; a dashboard cross-references the two. If the receiver fires and the executor refuses, operator sees the mismatch within minutes.

What we sign up to operate:

• A second binary (the receiver) on a small internet-facing host. CI artifacts, image scanning, deploy cadence — all the same shape as any other fleet service.
• The deploy_request ticket schema as a versioned contract. Schema changes ride a guideline-version bump per ADR-117; old executors must keep working when new fields appear.
• The executor's §4 trusted-filer allowlist as a piece of fleet config, refreshed in lockstep with new filer types. Adding a filer (e.g. a future "external partner CI fires deploys") is a deliberate ADR-level act, not a config tweak.
• The polling cadence and the "max-recent-wins per slug" coalescing rule. Today's defaults (10s, take-most-recent) are operator-tunable; the rule is the structural piece, the values are knobs.
• A dashboard for "warp deploy_request items in ready vs in_progress" so executor-offline drift is visible. Likely a one-pane addition to the existing factory release-status page (warp#1528).

Alternatives considered

• Status quo: keep ai-research as one local-only binary; never put it on the internet. Loses on the operator's stated goal. The operator wants the receiver path to not depend on a single laptop tunnel. Status quo blocks that desire indefinitely.
• Same binary, network-gated by Tailscale or similar. Easy in the short term: deploy ai-research on a public host, expose only the webhook intake path through Caddy, deny everything else. Loses on the structural axis: the SSH keys still live next to the public-IP listener. Compromise of the listener is full-fleet root. Network gating does not address the operator's actual concern ("too much security stuff to put on the internet"); it addresses the question they didn't ask ("how do I put the binary on the internet"). Reject — wrong-question fix.
• Microservice big-bang split. Extract every job (receiver, API server, executor, cloud control, approval gate) into its own binary at once. Architecturally pure; high implementation cost; multi-month refactor. Phasing matters more here than completeness — receiver/executor split delivers the security win, the API server and cloud control can split later if and when scale demands. Reject as primary path; revisit if/when fleet grows beyond one operator.
• GHA-files-warp-tickets-directly, no receiver binary. Each project's release.yml gains a final file-deploy-request step that POSTs the warp ticket. The receiver binary doesn't exist; ADR-161's gate cascade either lives inside each release.yml's bash step or doesn't run. Pros: zero new infrastructure; reuses GHA's existing trust model. Cons: distributes the trigger-evaluation logic across N release.yml files (every project must keep the loop guard + dedupe + opt-in checks in lockstep — or worse, drop them, breaking ADR-161's invariants); ties the deploy chain to GHA availability (today's billing incident proves this is a real failure mode); makes non-release triggers (manual deploys, monitoring-driven rollback, scheduled crons) require a different mechanism; loses the audit-log centralization. Considered as an addition, not a replacement. The receiver is the right home for centralised trigger logic; GHA-as-additional-filer is a clean Phase 2 augmentation if wanted (filer allowlist gains gha; release.yml's file-deploy-request step posts straight to warp, bypassing the receiver for release-event-only triggers; receiver still handles non-release events). Both filers feed the same queue; the executor doesn't change.
• Cloud-managed serverless executor (Cloud Run / Lambda). Receiver-as-Lambda is plausible but ansible needs persistent SSH context, durable image caches, and host-specific state — serverless is a poor fit for the executor side. Receiver-as-serverless is reasonable but adds a vendor dependency and a billing surface for what is currently a Hetzner-CAX-shaped problem. Reject for executor; defer for receiver until the Hetzner-CAX option is proven insufficient.
• Vault + per-fire credential issuance now. Gold-standard pattern: executor requests a TTL'd SSH cert from Vault per deploy; no long-lived keys at all. Reject for now — Vault rollout is its own project (estimate: 2-3 weeks); the receiver/executor split is the bigger win and unblocks the security audit independently. Phase 3 adds Vault when needed; the executor's credential-loading shape is already abstract enough to swap.
• A new dedicated webhook endpoint pattern (one URL per event type, instead of dispatch inside one URL). ADR-161 considered this and rejected for operational simplicity reasons; this ADR doesn't reopen that decision.
• Fold approval gate into the queue entirely (no separate operator-confirm step). The warp type: pause pattern (ADR-109/110) already handles operator confirmation for irreversible ops. Don't reinvent; reference. Destroy-* playbooks specifically use type: pause per Phase 0 of this ADR's migration.

Cross-references

• ADR-161 — trigger semantics for deploys (workflow_run.completed cascade). Preserved verbatim; the receiver binary inherits PR#847's implementation. This ADR cites ADR-161 as authoritative for "what fires"; this ADR is authoritative for "where the firing decision lives".
• ADR-109 / ADR-110 — type: claude step + observer: field + observe-and-file dominant pattern. Destroy-* playbooks gain type: pause in Phase 0 audit; the executor honours that gate before firing.
• ADR-111 — broker as model-routing layer. If the executor ever uses LLM judgment (e.g. for failure-mode diagnosis in the notification surface from ADR-161 §4), it goes through the broker. Today's executor doesn't call models; this is a future-state hook.
• ADR-117 — module guidelines. The deploy_request ticket schema is a fleet contract; schema changes ride a guideline-version bump.
• warp#186 — auto-deploy epic that ADR-161 implements. This ADR is the security/perimeter Phase running in parallel with ADR-161's trigger Phase.
• warp#1556 / PR#847 — ai-research webhook receiver implementing ADR-161. The receiver binary in §2 of this ADR is the same code, factored into its own entrypoint.
• warp#1528 — /release-status page (visibility half of warp#186). The "executor offline" failure mode in §6 surfaces here.
• warp follow-ups (filed alongside this ADR) — Phase 0 audit, Phase 1 binary split, Phase 2 receiver deployment.

Supersedes: none Superseded by: