ADR-112: Executor isolation — split web-facing factory from private executor

Status: Proposed Date: 2026-04-26 Related: ADR-079 (operational-approval-substrate), ADR-088 (destroy-needs-silence-snapshot-deregister-wait), ADR-089 (host-teardown-protocol), ADR-107 (gates-as-playbook-primitives), ADR-111 (broker-as-model-routing-layer)

Context

The factory today runs on localhost:9400: local-only, no auth, fine for development. As the runtime moves from rendering dashboards to executing privileged actions — Ansible against production hosts, SSH into managed servers, Hetzner provisioning, DNS cuts on gyrum.ai — the factory accumulates secrets that change the threat model entirely:

SSH private keys for every gyrum host
A Hetzner Cloud API token (provision and destroy capability)
Vault passwords (Postgres credentials, Cloudflare tunnel tokens, GHCR push tokens)
DNS API access (capable of cutting warp.gyrum.ai to anywhere)

Running that surface as an internet-exposed web app is the structural shape that makes a single web vulnerability a fleet-wide RCE plus infrastructure-destruction event plus DNS hijack. The operator named the worry on 2026-04-26: "the factory isn't protected yet so putting it on the internet is a concern". The two-process answer — keep the web-facing surface unprivileged, push the privileged surface behind no inbound network — is a standard isolation pattern, but only effective if it is structural rather than a habit anyone can break by adding a token to the wrong process.

ADR-079 already established the operational approval substrate (signed approval messages on a queue) for code-merge-style approvals. ADR-088/089 already established the destroy-needs-silence pattern of separating the decision (silence the alarms, snapshot, wait) from the action (delete). This ADR extends both lines: separate the decision (the operator clicking "approve" in the web UI) from the action (the playbook fire that touches infrastructure), with a network boundary between them.

Decision

The factory splits into two deployments with zero shared process state:

                  Operator (browser)
                         │
                         ▼ HTTPS
       ┌─────────────────────────────────┐
       │  factory-web                     │
       │  - UI (Svelte) + API (Go)        │
       │  - Read-only DB views            │
       │  - Approval queue (ADR-079)      │
       │  - Sub-agent telemetry surface   │
       │  ZERO SECRETS                    │
       │  Internet-exposed                │
       └────────────────┬────────────────┘
                        │ signed approvals via queue
                        ▼
       ┌─────────────────────────────────┐
       │  factory-executor                │
       │  - Playbook runtime (ai-research)│
       │  - SSH keys                      │
       │  - Hetzner token                 │
       │  - Vault passwords               │
       │  - DNS API tokens                │
       │  - gyrum-ai-broker (ADR-111)     │
       │  Private network only            │
       │  Pulls approvals + fires         │
       └─────────────────────────────────┘
                        │ ssh / api
                        ▼
                 Managed hosts + cloud

factory-web carries the user-facing surface — the SvelteKit UI, the API the UI talks to, read-only views of the database, and the approval queue (extending ADR-079's substrate to playbook-fire approvals). It is internet-exposed at factory.gyrum.ai. It holds no SSH keys, no Hetzner token, no vault passwords, no DNS tokens. A successful compromise of factory-web is embarrassing — leaked dashboards, manipulated read views — but does not yield infrastructure access.

factory-executor carries the privileged surface — the playbook runtime, all credentials, the gyrum-ai-broker (per ADR-111). It runs on a private host (factory-exec-01) with no inbound network surface; communication is outbound-only, long-poll against the approval queue. The executor pulls approval messages, validates the cryptographic signature, then fires the playbook. A successful compromise of factory-executor is fleet-wide RCE — but the executor has no internet-facing surface and is reachable only via VPN or bastion, raising the bar substantially.

The approval-queue contract extends ADR-079's shape:

# Approval message schema (queue payload)
playbook_id: <yaml-file-path>
inputs: { ... }
approved_by: <operator-key-id>
approved_at: <iso-8601>
signature: <ed25519-sig over canonicalised body>
expires_at: <iso-8601 — short window, default 5 min>

Signature verification is the first step of every executor fire — failure rejects the message and writes an audit-log entry. Every executor action emits {run_id, approval_id, signed_by, fired_at, result} to an append-only audit log; the operator can later prove which approval triggered which action.

DNS for factory.gyrum.ai resolves to factory-web only. The executor host appears in inventory but not in any public DNS or load balancer.

The interim configuration uses Cloudflare Access (warp #342) as the bridge while the operator decides whether factory-executor lives on rented infrastructure or on in-house hardware in a domestic environment; the long-term decision is unconstrained by this ADR.

Consequences

What becomes easier.

A web-compromise becomes survivable. The blast radius is the read views + the approval queue UI, not the SSH-key vault. Recovery is a redeploy, not a fleet-wide incident.
Audit gains a structural anchor. Every infrastructure mutation traces back through one approval message with a verifiable signature; the "who fired this?" question always has an answer with cryptographic backing.
The executor inherits the destroy-needs-silence pattern (ADR-088/089) for free. Both substrates already know the action runs through a separate, deliberate gate; this ADR just makes the separation enforced by network topology rather than by convention.
New privileged capabilities have an obvious home. When the runtime needs another credential type, it goes on the executor; the discussion of "should this go on the web?" is closed.
The broker (ADR-111) lives where it should. The single-routing-layer for model calls runs on the executor side, so its credentials and audit logs sit behind the same network boundary as the playbook secrets.

What becomes harder.

Two deployments to manage, monitor, and re-create. Mitigation: factory-executor is provisioned by the same provision-host playbook used for everything else (ADR-090's re-creatable infrastructure principle), and the operator surface in factory-web shows executor health, queue depth, and audit-log tail.
Approval-queue latency. A queued-then-fired approval adds 5–30 s versus an in-process fire. Acceptable for deploys (which take minutes anyway), unacceptable for sub-second cron triggers (which are not the workload this surface targets).
Failure modes multiply. Queue down means no fires; executor unreachable means no fires; bad signature means no fires. Mitigation: the queue is the same ADR-079 HA substrate already in use for code approvals, and executor health is itself a kind: pipeline warp item with health POSTs from the executor's heartbeat.
Bypass risk. A bug in the executor allowing an unsigned approval to fire collapses the whole isolation. Mitigation: signature verification is the first step of every fire and is itself a structural check (warp #335 P-C10 shape); the executor refuses to start if its verification path is unreachable.

What we sign up to maintain.

A new ansible role + compose stack for factory-executor. Same template as every other gyrum service deployment.
A second host (factory-exec-01) in the inventory, with its own backup schedule, monitoring, and re-provision playbook.
The signed-approval contract. Schema documented, signing-key rotation playbook documented, signature-verification step shipped as part of roles/gates/tasks/ (per ADR-107) so any playbook author can import_tasks it.
The audit log. Append-only, monthly rotation, retention policy documented; queryable from factory-web's read-only view but writable only by the executor process.

Alternatives considered

Keep one process, add auth in front of it. Considered and rejected — auth is the front door, not the structural barrier. Any RCE bug in the web layer reaches the SSH keys regardless of auth. The threat model demands process and network separation, not just an auth check.
Use a single process with capability dropping (separate Linux users). Rejected on substrate count and brittleness. A bug that escalates within one process — file-descriptor leaks, signal mishandling, library CVEs — defeats user-level isolation. The two-process boundary with a network gap is the standard answer for the threat class.
Run the executor on the operator's laptop on demand. Considered and rejected — it makes scheduled fires (cron, webhook) impossible, and ties operational availability to the operator's machine. The dedicated executor host removes that single point.
Use Hashicorp Vault as the secret holder, keep one factory process. Rejected as orthogonal — Vault solves credential storage, but the call path that pulls a credential from Vault and the call path that handles HTTP requests still live in the same process. Network isolation is the missing piece.
Defer this until after the next breach. Rejected on the obvious grounds. The blast radius is fleet-wide infrastructure destruction; "after the breach" is the wrong time to discover the boundary should have existed.

Supersedes: none Superseded by: leave blank until a later ADR reverses this one

References

ADR-079 — operational-approval-substrate (the queue + signature pattern this ADR extends to playbook fires)
ADR-088 — destroy-needs-silence-snapshot-deregister-wait (parallel pattern: separate decision from action)
ADR-089 — host-teardown-protocol (same separation principle on the destroy path)
ADR-090 — ai-as-scaffold-pipelines-as-runtime (re-creatable infrastructure principle that backs the "two deployments are fine" argument)
ADR-107 — gates-as-playbook-primitives (signature-verification step shipped under roles/gates/tasks/)
ADR-111 — broker-as-model-routing-layer (the broker lives on the executor side per this split)
warp #338 — implementation epic (this ADR is r1)
warp #321 — postmortem (silent-success patterns motivate the audit-log requirement)
warp #329 — structural-gates implementation epic (signature verification fits there)
warp #334 — type: claude (runs on executor not web)
warp #337 — broker (runs on executor side)
warp #342 — interim Cloudflare Access bridge while the long-term host-location decision settles