Decisions

ADR-078: Trigger-driven playbook orchestration

We extend the ADR-068 playbook runtime with a **trigger layer** — an additive `triggers:` section in the playbook YAML that declares how a playbook gets fired **without a human clicking Run**. Three trigger kinds land…

#078

ADR-078: Trigger-driven playbook orchestration

Status: Proposed Date: 2026-04-24 Related: ADR-067 (playbooks unified primitive), ADR-068 (playbook runtime), ADR-069 (playbook security), ADR-070 (playbook UI contract), ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-077 (agent coordination layer — in flight), ADR-050 (bearer auth for product metrics), ADR-049 (Alertmanager bearer auth at Caddy)

Decision (one paragraph)

We extend the ADR-068 playbook runtime with a trigger layer — an additive triggers: section in the playbook YAML that declares how a playbook gets fired without a human clicking Run. Three trigger kinds land as first-class citizens — cron (time-based, deterministic), webhook (event-based, external POST), and alert (Alertmanager routing) — with a fourth (polled_change) named and deferred. Each trigger is registered on playbook hot-reload against a (playbook_id, trigger_id)-keyed registry; a fire is a new playbook_run row stamped with trigger_kind and trigger_id (migration 003 adds the columns). Matching is deliberately per-trigger and dumb — label equality plus a regex on one field — and any logic beyond that belongs in the playbook, not the trigger layer. We ship in four phases: 2b cron-only (MVP, goal: fleet-health.yaml every 15 min), 2c webhook (Hetzner host-joined → onboarding), 2d alert (Alertmanager → triage), 2e polled change (inventory diff). This ADR specifies the shape; implementation is a follow-on tranche of PRs that does not land in this cycle.

Context

ADR-067 made the playbook a unified primitive. ADR-068 made it executable. ADR-076 made it bind to projects and hosts. ADR-077 (in flight) is making human and agent decisions legible inside a run via approval gates that park a run until the UI responds. None of them answered the question this ADR closes: how does a playbook get started without a human clicking "Run now"?

Today, every playbook_run row in the ai-research database was created by a human POSTing to /api/v1/playbook-runs from the <PlaybookRunner> button. That is the right shape for interactive work — kicking off a release, approving a destructive step. It is the wrong shape for the operational loop the vision demands.

The vision is a self-closing loop that runs for years. Three scenes that should be automatic and currently are not:

  1. New Hetzner server joins the fleet. The provisioning API fires a webhook. The system notices — no one typed anything. The host is hardened, catalogued, enrolled in monitoring, reported back.
  2. A feature merges. The release flow deploys to staging, verifies, observes for 15 minutes, then either promotes to prod or opens an issue. The merge was the event, the cycle ran from it.
  3. An alert fires. Alertmanager routes it to a triage playbook that collects diagnostics, proposes a rollback, and parks at a gyrum-approve step. A human clicks approve; the rollback runs. No terminal, no ssh, no typed commands.

Shared shape across all three: an event triggers a playbook. The playbook runs the way playbooks always run — same executors, SSE stream, audit trail, ADR-077 approval gates when needed. The only new capability is the door by which the run gets started.

Every run today starts because a human noticed something and clicked. That is the single slowest component in any loop: a sleeping operator is a thousand-times-slower dispatcher than a cron tick. The foundation to close this is already shipped (ADR-067/068/072/076 plus the Phase 1c UI + Phase 2 executor registry) or in flight (ADR-077). What's missing is the trigger layer — this ADR.

Why a separate ADR and not an amendment to ADR-068. ADR-068's surface is the runtime contract — executors, state machine, queue, SSE. That contract is settled. The trigger layer is orthogonal: the front door, not a change to how the runtime works after a run is enqueued. It also has its own security surface (public webhook endpoints, Alertmanager auth, cron fairness) that belongs outside the already-large ADR-069.

Relationship to ADR-077. They answer different questions:

Question ADR
How does a run start — what event fires it? ADR-078 (this)
How does a run decide mid-flight — when it needs a human or agent to judge? ADR-077

A concrete sequence: alert fires → ADR-078 matches and enqueues a triage playbook → playbook gathers diagnostics → reaches an approval step → ADR-077 parks the run, UI shows gyrum-approve card → human clicks → run resumes. Two ADRs, clean seam, one loop.

The seam: triggers fire unconditionally (no human in the loop to fire them — that's the point); actions may need approval (the playbook's approval steps are where ADR-077 engages). Committing a playbook YAML with a triggers: block is the implicit approval for the trigger to fire; the PR review is where consent was given. Firing at run-time does not require a second consent event.

Decision

1. Summary of gating choices

Five decisions are settled. Each is motivated below; the table is the quick index.

# Choice Rejected alternatives Gated by
1 Triggers are an additive YAML block, backwards-compatible side-car triggers.yaml file, a separate registration endpoint §2
2 Three kinds in scope: cron, webhook, alert; a fourth (polled_change) deferred one universal event model (CloudEvents), rules engine (CEL/JSONPath) §3
3 robfig/cron/v3 for cron, one goroutine per registered schedule k8s CronJob, external cron daemon, Temporal schedules §4.1
4 Per-trigger dumb matching (label equality + regex on one field) global rules engine, CEL, full JSONPath §5
5 Migration 003 adds trigger_kind + trigger_id to playbook_runs separate playbook_trigger_fires table, reuse actor for the trigger id §6

2. The YAML — additive to the ADR-068 spec

An existing playbook without a triggers: block behaves exactly as today: it runs when a human POSTs to /api/v1/playbook-runs. Adding triggers: turns on the doors. The block sits alongside inputs: and steps: inside runtime:.

Full shape, all three kinds:

---
kind: service_runbook
id: fleet-health-scan
title: "Fleet-wide health scan"
runtime:
  inputs:
    - name: scan_type
      type: string
      default: routine
  triggers:
    - id: every-15-min
      on: cron
      schedule: "*/15 * * * *"
      inputs:                      # static inputs for every scheduled run
        scan_type: routine

    - id: hetzner-host-joined
      on: webhook
      path: "/webhooks/hetzner/host-joined"
      secret_ref: "vault:hetzner_webhook_secret"
      payload_map:                 # JSON path → playbook input
        host_name: ".data.server.name"
        host_ip:   ".data.server.public_net.ipv4.ip"

    - id: critical-web-alert
      on: alert
      match:
        labels:
          severity: critical
          service: web
      inputs:
        alert_summary: "${alert.annotations.summary}"
        alert_runbook: "${alert.annotations.runbook_url}"

  steps:
    - id: snapshot
      type: http
      target: local
      method: GET
      url: "http://localhost:8080/api/v1/fleet"
      save_as: fleet
    # …
---

Each trigger is keyed by a trigger id: that is unique within the playbook. The (playbook_id, trigger_id) pair is the registry key (§6); renaming the trigger is a visible change in the YAML diff and in the audit stream.

3. The three kinds (and a named fourth)

3.1 Cron — time-based, deterministic, MVP

- id: every-15-min
  on: cron
  schedule: "*/15 * * * *"         # standard 5-field crontab
  timezone: UTC                    # optional; default UTC
  inputs:
    scan_type: routine

Semantics:

  • Standard 5-field crontab (minute hour day-of-month month day-of-week). The 6-field seconds-resolution form is not accepted; sub-minute playbooks are almost always wrong.
  • Every tick produces one run with trigger_kind: cron and trigger_id: every-15-min.
  • Static inputs: flow into the run's initial inputs. If the playbook's inputs: schema requires a field the trigger doesn't set, parse-time validation fails.
  • No catch-up. If ai-research was down during a tick, the tick is lost. Catch-up is a footgun (90-minute outage → 6 deploys in a minute) — documented rejection, see §7.4.

3.2 Webhook — event-based, external POST

- id: hetzner-host-joined
  on: webhook
  path: "/webhooks/hetzner/host-joined"
  secret_ref: "vault:hetzner_webhook_secret"
  payload_map:
    host_name: ".data.server.name"
    host_ip:   ".data.server.public_net.ipv4.ip"

Semantics:

  • Path prefix. Registered webhooks live under POST /api/v1/trigger/webhook/<path>. A leading /webhooks/ in the path: is stripped as a convenience, so the example resolves to /api/v1/trigger/webhook/hetzner/host-joined.
  • secret_ref: is a vault reference (ADR-069 §4). Plaintext lives only in the vault; the registry resolves at trigger-registration time and caches in memory for the replica's lifetime.
  • Inbound auth. Request must carry X-Gyrum-Trigger-Secret: <value>. Constant-time comparison. Missing or mismatched → 401, no hint.
  • HMAC variant for vendors that sign (Stripe, GitHub): hmac: {header: X-Hub-Signature-256, algorithm: sha256, secret_ref: …} instead of secret_ref:.
  • payload_map: is jq-ish single-path, not full jq. Each value is .foo.bar[0] — object keys and array indices only, no filters, no pipes. Malformed path → parse-time error; missing at fire-time → null input.
  • Raw payload escape hatch. The full parsed JSON body lands in inputs.__payload so the playbook can extract anything the trigger layer's mapping can't. Trigger-layer dumb, playbook-layer expressive.

3.3 Alert — Alertmanager routing

- id: critical-web-alert
  on: alert
  match:
    labels:
      severity: critical
      service: web
    annotations:
      auto_triage: "^true$"        # regex on the annotation value
  inputs:
    alert_summary: "${alert.annotations.summary}"
    alert_runbook: "${alert.annotations.runbook_url}"
    alert_fingerprint: "${alert.fingerprint}"

Semantics:

  • Alertmanager POSTs to a single endpoint (/api/v1/trigger/alert). Each element in alerts[] is evaluated independently; one POST can fire multiple registered triggers.
  • match: dialects: keys under labels: and annotations: are AND'd. A string value under labels: is exact; a value starting with ^ or containing .* under annotations: is a regex. Labels are usually enumerations (exact is right); annotations are free-text (regex fits).
  • Alertmanager auth. Same X-Gyrum-Trigger-Secret header, sourced from a platform-wide alertmanager_webhook_secret vault entry (not per-trigger). Alertmanager's webhook config points at the secret-bearing URL per ADR-049.
  • ${alert.X} in inputs: are templated against the alert object (a distinct namespace from ADR-068's ${inputs.X} / ${outputs.X}), evaluated once at fire-time to produce the run's initial_inputs.
  • resolved alerts. By default, a trigger matches only firing alerts. match.status: resolved narrows; status: any matches both. This is the most common Alertmanager footgun — explicit in the schema.

3.4 polled_change — named and deferred

This is the fourth trigger, specified here for the roadmap's sake and not shipped until Phase 2e. Shape (sketch):

- id: host-list-changed
  on: polled_change
  source: "http://localhost:8080/api/v1/catalog/hosts"
  every: 5m
  watch: ".hosts[].alias"          # a path into the source JSON
  fire_on:
    - added                        # new alias appears in the set
    - removed                      # alias disappears
    # - changed                    # not a Phase 2e primitive
  inputs:
    change: "${change}"            # {kind: added|removed, value: …, at: …}

Semantics (reserved):

  • The runtime polls the source: on the every: cadence, extracts the value of watch: from the response, diffs against the previous snapshot, and fires once per add/remove.
  • The diff snapshot lives in a new table (playbook_trigger_state, reserved for Phase 2e).
  • This is the cheapest path to "no webhook? no alert? no cron? — react to inventory change anyway". It is explicitly deferred because (a) implementing a correct diff engine with stable ordering and failure modes is its own design problem, and (b) the three shipped kinds cover the motivating scenes (§Context) already.

We call it out here so the trigger kind enum is complete from day one; adding polled_change later is additive.

4. Dispatcher architecture

4.1 Cron — robfig/cron/v3

We use github.com/robfig/cron/v3. Rationale: pure Go, zero cgo, actively maintained, used by prometheus-operator / gRPC / Kubernetes clients. Supports 5-field crontab (the only shape we accept), WithLocation(tz) for the timezone: field, and an EntryID handle for clean de-registration on hot-reload (§4.4). Port cost if it goes stale: ~200 lines (crontab parser + priority queue).

Rejected cron-layer alternatives: k8s CronJob (we don't run k8s, ADR-020); external cron daemon (splits source of truth — YAML plus crontab, two deploys per schedule change); Temporal schedules (ADR-068 §16 already rejected Temporal for runs; same reasoning).

Implementation shape: one *cron.Cron instance per replica, WithLocation(UTC) default, populated from the registry on hot- reload. Each schedule's Run() enqueues a playbook_run via the existing runtime (same code path as the UI button) with trigger_kind: cron.

Replica fairness. Multiple replicas would otherwise fire N runs per tick. The cron instance only enqueues if it can INSERT a row into a playbook_cron_leases table with schedule_key = <playbook_id>:<trigger_id>:<tick_unix_minute> as the PK. First replica wins the tick, the rest no-op. Classic cron-leader pattern without needing a separate leader-election protocol.

4.2 Webhook — HTTP handler

A single route group serves all registered webhook triggers:

POST /api/v1/trigger/webhook/*path

The handler:

  1. Looks up the registered trigger by full path (stripping the prefix). 404 if unregistered.
  2. Validates the X-Gyrum-Trigger-Secret header (or HMAC signature per §3.2) against the cached secret. 401 on mismatch.
  3. Checks the per-trigger rate limit (§7). 429 on exceeded.
  4. Parses the body as JSON. 400 on parse error.
  5. Evaluates payload_map: against the parsed body, merging resolved values into the initial inputs.
  6. Enqueues a playbook_run with trigger_kind: webhook, trigger_id: <id>. Returns 202 Accepted with {run_id: …, events_url: …} so the webhook sender can follow the run if it wants to.

The handler never executes steps — it enqueues. The 202 returns in milliseconds regardless of playbook runtime; webhook senders (Hetzner, GitHub) have tight timeout budgets and retry on slow 5xx.

4.3 Alert — Alertmanager handler

A single route:

POST /api/v1/trigger/alert

Behaviour mirrors §4.2 but the body shape is Alertmanager's webhook format:

{
  "version": "4",
  "groupKey": "…",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {"severity": "critical", "service": "web", "alertname": "…"},
      "annotations": {"summary": "…", "runbook_url": "…"},
      "startsAt": "…",
      "endsAt": "…",
      "fingerprint": "…"
    }
  ]
}

For each alert in alerts[], the handler walks every registered on: alert trigger and evaluates match:. Each matching trigger fires once per matching alert.

Deduplication uses the alert's fingerprint as an idempotency key in the existing ADR-068 §6 window table. Same (trigger, fingerprint) inside 5 minutes → returns the existing run id, preventing retry storms from spawning N triage runs.

4.4 Hot-reload

The existing loader (ADR-067 §6) watches dark-factory/playbooks/ and re-parses on change. This ADR extends it:

  1. Extract each playbook's triggers: block.
  2. Compute a stable key hash(playbook_id, trigger_id, spec_body). The body hash detects spec changes that keep the same id (schedule moved from */15 to */30).
  3. Diff against the registry: new → register, removed → deregister, unchanged → no-op.
  4. Reload is atomic per playbook: stage new set, swap behind a mutex, deregister stale. No tick sees a half-applied reload.

A trigger whose registration fails (invalid cron, duplicate webhook path, missing vault secret) is logged and the playbook marked not hot-loadable until fixed; the rest of the registry is unaffected — same discipline as ADR-067's playbook-level validation failures.

5. Matching rules — per-trigger, not a rules engine

We deliberately do not ship a rules engine. No CEL, no JSONPath- with-filters, no full jq, no PromQL-over-payloads. Each trigger gets:

  • Label equality (alert): exact match.
  • Regex on one field (alert annotations): value starting with ^ or containing .* is a regex.
  • Single-path extraction (webhook payload_map:): .foo.bar[0]; no filters, no pipes.

Matching lives at the trigger boundary and does almost nothing. The playbook body is where complex conditional logic goes — a when: step condition, a type: shell that parses the payload, a type: claude that classifies, a sub-playbook dispatch. The trigger's only job: did this event reach the right playbook at all?

Reasoning: a rules engine is a language; a language we operate requires a spec, a fuzzing harness, a security review, and ownership. We avoid adopting one until demand is proven. Second, if a trigger needs complex filtering, the right shape is "always fire, let the playbook decide". A skip: first step with a rich condition is reviewable, testable, and observable (the run appears in the audit even when it no-ops); a silent trigger-layer filter is invisible.

Exception: if one trigger handles 100x the traffic of the others and most fires no-op, the rate limiter (§7) throttles the noisy source. We accept that cost rather than import an engine.

6. Schema — migration 003

-- migration 003: trigger fields on playbook_runs

ALTER TABLE playbook_runs
  ADD COLUMN trigger_kind  text,               -- 'cron' | 'webhook' | 'alert' | 'polled_change' | NULL
  ADD COLUMN trigger_id    text;               -- author's trigger id; NULL for human-fired runs

CREATE INDEX ON playbook_runs (trigger_kind, trigger_id, created_at DESC)
  WHERE trigger_kind IS NOT NULL;

-- cron tick leadership (§4.1 replica fairness)
CREATE TABLE playbook_cron_leases (
  schedule_key     text PRIMARY KEY,            -- <playbook_id>:<trigger_id>:<unix_minute>
  won_by_worker    text NOT NULL,
  won_at           timestamptz NOT NULL DEFAULT now()
);

-- alert idempotency extends the existing window shape (ADR-068 §6)
-- no new table; we encode (trigger_id, fingerprint) as the key.

Choice to record trigger identity on the run, not in a separate playbook_trigger_fires table. Consequence: UI filters are a column and an index, not a new table and a join. A run is a run — the schema, events, UI card are identical to a human-fired run, with only trigger_kind distinguishing. A trigger_fired event in playbook_run_events (ADR-068 §8 kind extension) records the fire for audit.

Rejected alternative: a standalone playbook_trigger_fires table — one row per fire, including failed fires (401s, rate-limited). Nice property but cost outweighs benefit. Rejected fires are structured log entries instead; if we ever need queryable rejected fires, we add the table additively without disturbing the accepted-fire shape.

7. Security + rate-limiting

7.1 Shared-secret or HMAC per webhook

Every on: webhook trigger MUST declare secret_ref: or hmac:. Missing → parse validation fails, playbook rejected. An un-authenticated public POST is a misconfiguration the schema makes syntactically impossible.

Secrets live in the vault (ADR-069 §4). Rotation: (1) mint new secret under a versioned path (vault:hetzner_webhook_secret@v2), (2) update trigger YAML in a PR, (3) update the sender (Hetzner / GitHub UI), (4) retire the old path. The dual-valid window between steps 2 and 3 is brief and acceptable for our annual rotation cadence (ADR-028). A dual-secret registry lands later if rotations become frequent.

7.2 Alertmanager-wide secret

Alertmanager POSTs to one endpoint; one platform-level secret. Per- alert secrets would fight Alertmanager's fan-out. One endpoint, one secret, many per-trigger match blocks is the right shape.

7.3 Rate limits

Default: 10/min, 100/hour per trigger. Both enforced (per-minute catches bursts, per-hour catches slow leaks). Override per trigger:

rate_limit:
  per_minute: 60
  per_hour: 1000

Implementation: token-bucket keyed by (trigger_kind, trigger_id), in-memory with per-minute Postgres checkpoint (survives restart). Excess → 429 to the sender (or drop-and-log for cron / alert) and trigger_rate_limited_total metric. The cap protects the runtime from a misbehaving sender POSTing every second (3600 runs/hour, each blocking a worker slot).

7.4 No cron catch-up (documented)

If the runtime was down when a cron would have fired, the tick is lost. trigger_ticks_missed_total surfaces it; the next tick runs normally. Catch-up is a footgun — a 90-minute outage on a 15-minute deploy-project schedule would fire six deploys in a minute. "Must run" belongs on an external-scheduler-backed webhook, not cron.

8. Concurrency — previous run still in-flight

A 14:15 tick may arrive while the 14:00 run is unfinished. Three options: skip (drop new), queue (enqueue anyway), kill- previous (cancel in-flight, start new).

Default: skip for cron, queue for webhook / alert. Cron playbooks in Phase 2b are health checks where the in-flight run should own the cycle. External events are meaningful and should not be silently dropped.

concurrency: skip                  # skip | queue | kill-previous

Resolves Q1 (§13).

9. Observability

Metrics (via distill, ADR-050 / ADR-053):

Metric Type Labels Question
playbook_trigger_fires_total counter kind, trigger_id, playbook_id, result "how often does each trigger fire, and fail at the door?"
playbook_trigger_latency_seconds histogram kind "fire-to-enqueue latency"
playbook_ticks_missed_total counter trigger_id "cron ticks missed during outages"
playbook_registered_triggers gauge kind "live triggers right now"
playbook_trigger_rate_limited_total counter kind, trigger_id "which triggers are noisy"
playbook_webhook_auth_failures_total counter trigger_id "is someone probing endpoints?"

result label values: enqueued | rate_limited | auth_failed | match_miss. Cardinality safe per ADR-005 — trigger_id bounded by the catalogue; kind is closed.

Structured logs: every fire includes trigger_kind, trigger_id, playbook_id, fire_id (UUID). The fire_id also lands on the playbook_run row so a log line and a run correlate. Tracing: a playbook.trigger.fire span wraps registration → match → insert; context propagates into the enqueued run so webhook → run is one trace.

10. The UI — /playbooks/[id] Triggers section

ADR-070's playbook detail page grows a read-only Triggers section in Phase 2b; interactive (manual fire, disable toggle) later.

Triggers (3)
────────────────────────────────────────────────────────────
every-15-min          cron       */15 * * * *
  Last fired: 2m ago  Next tick: in 13m  24h: 96 fires
  [Fire now]

hetzner-host-joined   webhook    /webhooks/hetzner/host-joined
  Last fired: 3d ago  24h: 0    Auth failures 24h: 0
  [Fire now (simulate)]

critical-web-alert    alert      severity=critical, service=web
  Last fired: never

Per-row data sourced from playbook_runs filtered by (trigger_kind, trigger_id) plus §9 metrics. Fire now posts to POST /api/v1/playbook-runs with a synthetic_trigger: {kind, id} payload — the run is tagged as a trigger fire for audit even though the dispatcher was a human click. Resolves Q3 (§13); the gyrum- fire-trigger CLI (§11) is the automation-flavour counterpart.

11. Developer ergonomics — gyrum-fire-trigger <id>

A new devtool for local test-fires:

$ gyrum-fire-trigger fleet-health-scan:every-15-min
$ gyrum-fire-trigger fleet-health-scan:hetzner-host-joined \
      --payload-file test-fixtures/hetzner-host-joined.json
$ gyrum-fire-trigger fleet-health-scan:critical-web-alert \
      --alert-fixture test-fixtures/web-down.json

Behaviour: reads the playbook YAML, finds the trigger id, and (a) for webhook POSTs the --payload-file to the registered URL with the secret header, (b) for alert wraps the fixture in an Alertmanager envelope and POSTs, (c) for cron POSTs to the synthetic-fire endpoint. Prints the run_id for gyrum-tail-run. Usable by CI for integration tests of trigger-to-run wiring.

12. Input merging — trigger-wins

Q4 resolved: merge, trigger-wins.

A synthetic fire from the UI lets an operator type additional inputs. They merge with the trigger's static inputs:, and the trigger wins on key conflict. The trigger declaration is in reviewed YAML; an ad-hoc UI override of a reviewed field is the shape of a mistake. Operators edit the YAML if they need a different value.

For real (non-synthetic) fires there is no second source — cron has only static inputs, webhooks map payload fields, alertmanager templates alert fields. Merging is only interesting for synthetic fires; the rule is still trigger-wins.

13. Open questions — resolved or deferred

# Question Resolution
Q1 Concurrency: skip / queue / kill-previous? Resolved, §8. Default skip for cron, queue for webhook / alert. Knob per trigger.
Q2 Trigger history: playbook_run_events or a separate playbook_trigger_fires table? Resolved, §6. Use playbook_runs.trigger_kind/id + a trigger_fired event in playbook_run_events. No separate table.
Q3 How does an author test triggers locally? Resolved, §11. gyrum-fire-trigger <id> CLI + a Fire now UI button that emits a synthetic fire.
Q4 Trigger inputs: override or merge with user inputs? Resolved, §12. Merge with trigger-wins.
Q5 Should un-registered triggers leave orphan runs visible? Deferred. If a triggers: block is removed from a playbook YAML, historical runs stay with their trigger_id but no live registration. UI marks them as "trigger removed". Formal behaviour pinned in Phase 2e when polled_change lands.
Q6 Cross-replica rate-limit state consistency? Deferred. Phase 2b rate limits are per-replica. If we run 2+ replicas, a noisy sender could see 2x the cap. Good enough for MVP; tighten when we have actual multi-replica pressure.

14. Phased rollout

Extends ADR-068's Phase 2 with sub-phases b/c/d/e.

Phase Scope Exit criteria
2b — cron MVP on: cron only. robfig/cron/v3 wired. Cron-lease leader election. Migration 003 (all trigger_kind enum values reserved). gyrum-fire-trigger CLI. Metrics + read-only UI. fleet-health-scan.yaml runs every 15 min for a week; missed ticks only on deploys.
2c — webhook on: webhook + secret/HMAC + payload_map:. Hetzner host-joinedhost-onboarding.yaml. Vault wiring. First Hetzner host joins end-to-end with no human click.
2d — alert on: alert + Alertmanager endpoint. service-triage.yaml template. Integration with ADR-077 approval gates. Critical alert → triage → parks at approval → human approves → rollback runs.
2e — polled_change on: polled_change + diff state table. Inventory-diff on gyrum-catalog/hosts. Out-of-band catalog edit triggers onboarding.

Phase 2b is deliberately the smallest door — one kind, one wired playbook, one metric page. Prove registry, hot-reload, lease, UI, metrics under real load before adding surface.

Amended 2026-04-24 — ADR-080 names warp as the durable queue for Phase 2c onward. The direct-invocation path from Phase 2b is replaced by warp.Enqueue → worker claim → runtime execute, and the state machine grows a claimed phase. See 080-warp-as-durable-trigger-queue.md.

15. Invariants inherited from prior ADRs

Not negotiable here — they shape this ADR.

  • YAML + Go struct is source of truth (ADR-068 §9). triggers: is a new sub-struct on Runtime. go generate emits an updated JSON-Schema; editor LSP autocompletes trigger kinds.
  • A run is a run (ADR-068 §5). Trigger-fired runs share schema, events, and UI card with human-fired; only trigger_kind distinguishes. The front door is separate from the hallway.
  • SSE + POST, not WebSocket (ADR-068 §6). Trigger-fired events stream identically.
  • target: on every step (ADR-068 §7). Triggers don't change step semantics; a triggered target: local still hits the allow-list gate.
  • Secrets live in the vault (ADR-069 §4). Webhook secrets are vault references, never literal YAML values.
  • Idempotency is run-level (ADR-068 §6). Alert-fingerprint dedup (§4.3) reuses the existing 5-minute window; no trigger-specific dedup table.
  • Approval gates park the run (ADR-077, in flight). A trigger- fired run reaching approval behaves identically to a human-fired one — triggers fire the start, approvals gate the middle.

16. Consequences

Easier.

  • The human stops being the dispatcher. Runs fire from events; the operator steers the UI, not a terminal.
  • Cron playbooks are reviewable. Schedule in YAML is a PR diff, not a crontab on a host nobody can find.
  • Webhook onboarding is composable. Hetzner → host-onboarding is one trigger + one playbook, not three scripts and a Slack thread.
  • Alerts grow beyond paging. A critical alert pre-stages a rollback, runs diagnostics, opens a ticket — all before the human reads the page.
  • Audit is unified. Every fire is a playbook_run with a trigger_kind. "What made the system do X?" has a SQL answer.

Harder.

  • New public endpoints. /api/v1/trigger/webhook/* is a probe surface. §7.1 secrets and §7.3 limits gate it, but continued discipline is required.
  • Cron catch-up is wrong but tempting. Operators will ask for it; §7.4 is the answer. Budget one ADR amendment if demand persists.
  • Replica fairness. The cron lease table is a new operational concern; nightly cleanup job owns the bloat.
  • Hot-reload races. A trigger deregistered mid-fire must not strand a playbook_run. The staged-swap in §4.4 handles it but needs explicit tests.
  • Per-trigger YAML review. Adding a trigger is implicit consent for it to fire autonomously, forever. A heavier review decision than "should this playbook exist?".

What we sign up to maintain. The trigger-kind set (adding a fifth is an ADR). The matching-language scope (expansion to CEL / JSONPath is an ADR — see §5). The robfig/cron/v3 dependency (port: ~200 lines). Migration 003. The gyrum-fire-trigger devtool on ~/.gyrum/devtools.

17. Alternatives considered

  • Universal event model (CloudEvents). Rejected for Phase 2b. A single envelope is the right abstraction at 5+ trigger kinds; at 3, the common abstraction is more surface than value. Adopting CloudEvents later is additive — each current trigger becomes a source in that taxonomy.
  • Rules engine (CEL / JSONPath-with-filters). Rejected, §5. A language requires spec + fuzzing + review. Complex logic goes in the playbook.
  • External scheduler (systemd timers, GitHub Actions cron, external cron). Rejected — two deploys per schedule change, split audit, no hot-reload.
  • Temporal schedules. Rejected, same reasoning as ADR-068 §16 for Temporal runs.
  • Register triggers via API, not YAML. Rejected. Autonomous firing is a security-relevant decision; it must go through PR review.
  • One table per trigger kind. Rejected. Common fields dominate; kind-specific fields live in YAML.
  • Polled_change in Phase 2b. Rejected — correct diff engines are their own design problem. Ship the three well-understood kinds first.
  • Global rate limit. Rejected — one noisy trigger would starve the rest.

18. Review cadence

Re-review every 90 days (next: 2026-07-24) against: fire volume by kind (informs Phase 2e priority); rate-limit hits (persistent hot triggers signal misdesign); auth failures (probes); hot-reload correctness (stranded runs, skipped triggers); missed cron ticks (cumulative count against runtime uptime).

References

  • ADR-067 — playbooks unified primitive; triggers: is an additive field on the front-matter-plus-runtime model.
  • ADR-068 — playbook runtime; the queue, the state machine, the SSE stream that trigger-fired runs ride.
  • ADR-069 — playbook security; the vault that holds webhook secrets.
  • ADR-070 — playbook UI contract; the Triggers section on the playbook detail page.
  • ADR-072 — infrastructure as playbooks; the primary audience for cron triggers (fleet-health, drift-detect).
  • ADR-076 — project-to-host deployment binding; a triggered deploy-project knows where each repo lives.
  • ADR-077 — agent coordination layer (in flight); approval gates on triggered runs park the run until a human or agent decides.
  • ADR-049 — Alertmanager bearer auth at Caddy; the shape of the X-Gyrum-Trigger-Secret header for the alert endpoint.
  • ADR-050 — bearer auth for product metrics; related auth pattern for the webhook endpoints.
  • ADR-005 — cardinality labels vs fields; §9 metrics vetted against this.
  • github.com/robfig/cron/v3 — the cron library.
  • Operator framing, 2026-04-24: "today playbooks only run via human Run now clicks; triggers let the system cycle without terminal work."

Supersedes: none Superseded by: leave blank until a later ADR reverses this one