ADR-138: Activity-gated heartbeat replaces process-alive heartbeat

Status: Proposed Date: 2026-05-06

Context

Phase 0 of EPIC warp#1318 (activity-gated heartbeat). Today's session surfaced two failure modes that the current heartbeat shape cannot distinguish, both of which spam the production warp service and silently mis-attribute work:

• Orphan-process spam. A sweep of the host found 6 stale heartbeat loops left over from prior gyrum-start-work invocations — workdirs deleted, parent Claude Code session long since exited, but the detached setsid + nohup loop in lib/heartbeat.sh:heartbeat_spawn still POSTing /heartbeat every 240s against tickets that had already been completed or released. Pure API spam, plus the lease never lapses so a fresh agent cannot reclaim.
• Stalled-agent spam. Four swarm agents tonight (labels O, U, FF, II) hit the "agent dead but heartbeat keeps firing" pattern: the Claude Code session crashed or got wedged, no further tool invocations happened, but the heartbeat loop kept the lease green for hours. Operator had to manually pkill the loop and warp-release the ticket.

Both failure modes share one root cause: the heartbeat loop in lib/heartbeat.sh (warp#683 contract) sleeps 240s and POSTs unconditionally. It never checks "is the agent still doing work?" — only "am I, the loop, still alive?" That is the wrong signal. A heartbeat should mean the work is progressing, not bash is running.

Sister structural shape: warp#1352 (per-worktree LSP isolation) has the same start-work-spawned, must-be-reaped-on-exit, watchdog-covers-crash-case shape; the reaper discipline this ADR adopts is identical to the gopls watchdog shape that ticket lays out. Sister cognitive shape: warp#1342 (review-pr READY-TO-MERGE directive) — that ticket addresses the cognitive side of agent stalls (the agent forgets to merge); this ADR addresses the process side (the heartbeat lies about the agent being alive).

Decision

Heartbeats become activity-gated: the loop fires only if there is fresh evidence the agent is making progress, and the warp server expires leases on last_activity_at rather than last_heartbeat_at.

The shape:

• Primary signal: parent-process liveness. The heartbeat loop captures $PPID (the gyrum-start-work shell, which is in turn a child of the Claude Code session) at spawn time and stores it in a new heartbeat.parent.pid file under the worktree's .gyrum/ directory. Each tick, before POSTing, the loop runs kill -0 <captured-ppid>; if the parent is dead, the loop exits cleanly, removes its pid file, and lets the lease lapse. Cheap, robust, no Claude Code config dependency.
• Secondary signal: activity timestamp. A last-activity file under the worktree's .gyrum/ directory is touched by Claude Code's PostToolUse hook (via a small touch-activity.sh shipped under the devtools hooks/ tree). The heartbeat loop reads its mtime; if older than HEARTBEAT_ACTIVITY_STALENESS_SECS (default 900s = 15min), the loop skips the POST that tick. This catches the "parent shell still alive but agent wedged" case where the primary signal would lie.
• Server-side lease-lapse change. The warp service tracks last_activity_at separately from last_heartbeat_at. The lease-expiry job uses last_activity_at as its threshold (default 30min). Filed as a Phase 1 child against gyrum-labs/warp (sister to this ADR — same EPIC).
• Backward-compat path. Existing fire-and-forget heartbeats keep working during rollout — they continue to POST /heartbeat, the server keeps populating last_heartbeat_at, and during the transition window the lease-expiry job uses MAX(last_activity_at, last_heartbeat_at) so unmigrated heartbeats are not penalised. Cutover is by-host: a new heartbeat spawn writes a schema_version: 2 field on the POST, the server prefers last_activity_at for v2 heartbeats and falls back to last_heartbeat_at for v1. Once all hosts emit v2 (tracked via a dashboard query), the v1 fallback is removed in a follow-up.
• Reaper discipline. Every spawn-script that starts a heartbeat (today: start-work.sh calling heartbeat_spawn; surfaced below) MUST register a corresponding kill in its happy-path exit and in a trap for crash-path exit. The complete-pr path already calls heartbeat_kill from complete-pr.sh. Sister to LSP daemon reaping in warp#1352 Phase 2.
• Watchdog. A systemd user-timer (or cron-fallback for non-systemd hosts) sweeps every 5 minutes for orphan heartbeat processes whose worktree directory no longer exists, and SIGTERMs them. Mirrors the gopls watchdog pattern in warp#1352 L2.1. Filed as a Phase 2 child.

Inventory of heartbeat-spawning sites in the gyrum codebase (sweep of ~/.gyrum/devtools/, only production paths, tests excluded):

| heartbeat_spawn itself | lib/heartbeat.sh | the loop body | Add parent-liveness check + activity-mtime check before each POST; emit schema_version: 2 on the POST body. | | gyrum-complete-pr (cleanup) | complete-pr.sh at line 786 | calls heartbeat_kill | No change — the kill path is already correct. |

| direct POST helpers | lib/warp-common.sh | references /heartbeat for direct calls | No change — these are explicit operator calls, not spawn-loops. | | stale-claim sweeper | warp-stale-unlock.sh at lines 164, 267, 286 | reads last_heartbeat_at to detect stale claims | Update threshold logic to use last_activity_at once server v2 lands; until then, no change (it is read-only). |

warp-complete.sh and setup-project.sh reference heartbeats only in comments / docstrings; no spawn-site change needed.

Consequences

What becomes easier:

• A crashed Claude Code session releases its tickets in ≤30min without operator intervention. The orphan-sweep sister-fix becomes a watchdog (covers genuine zombies) rather than a recurring chore (covers the daily case).
• Operator can trust last_activity_at as a real progress signal — the dashboards in warp UI stop being noise.
• Reviewers and persona reviewers see a meaningful "stalled" state on tickets, distinct from "claimed but not heartbeating".
• Reaper discipline matches LSP daemon reaping (warp#1352), so the operational mental model is one shape, not two.

What becomes harder:

• Two-signal logic in the loop is more code than the current sleep-and-POST. The primary signal (parent PID) is one extra kill -0 per tick — negligible. The secondary signal needs a Claude Code PostToolUse hook installed on each developer machine; gyrum-setup will own that, but it is a new dependency on Claude Code's hook system that did not exist before.
• The server-side lease-expiry change is a coordinated cutover. The backward-compat window described in the Decision (server reads MAX(last_activity_at, last_heartbeat_at) until all hosts emit v2) is the migration path; the dashboard query that confirms "all hosts emit v2" must be live before the v1 fallback is removed.
• Hosts without systemd (the contractor sandbox case) need a cron fallback for the watchdog. Filed as part of the Phase 2 child; a missing watchdog is not a correctness issue (the activity-gated loop self-exits on parent death) but a defence-in-depth gap.
• The PostToolUse hook is best-effort — if Claude Code is upgraded and the hook contract changes, the secondary signal silently degrades to "always stale" and the loop stops POSTing. The watchdog (Phase 2) plus the primary signal (parent PID) keep this from being a correctness regression, but it is a known fragility we are signing up to monitor.

Phasing / migration path:

• Phase 0 (this ADR). Lock the semantics. No code changes.

• Phase 1a (gyrum-labs/devtools). Retrofit lib/heartbeat.sh with parent-PID + activity-mtime gating; emit schema_version: 2; update start-work.sh to wire the signals. Keep v1-shape POSTs as fallback.
• Phase 1b (gyrum-labs/warp). Add last_activity_at column; update lease-expiry to read MAX(last_activity_at, last_heartbeat_at) during transition.
• Phase 2. Watchdog (systemd timer + cron fallback). Orphan-worktree sweep.
• Phase 3. Once schema_version: 2 adoption hits 100% on the dashboard query, remove MAX(...) fallback; lease expiry uses last_activity_at only.

Alternatives considered

• Inotify watch on the worktree directory. Cheap and accurate for "files changed", but Claude Code's tool surface includes plenty of work that does not touch files in the worktree (web fetches, RAG queries, broker calls). Inotify would mark those as inactivity and prematurely lapse the lease. Activity-timestamp via PostToolUse covers all tool invocations, not just file edits.
• Pure activity timestamp (no parent-PID check). Simpler, one signal. Lost because the secondary signal depends on Claude Code's hook system being correctly wired; if the hook is missing or breaks (Claude Code upgrade, sandboxed runner without hook support), the loop would never POST and every ticket would fail-stop on first lease window. The primary signal (parent PID) is the no-config baseline that survives missing hooks.
• Pure parent-PID liveness (no activity timestamp). Catches orphaned-process spam but misses the wedged-agent case (parent shell still alive, agent stalled). Tonight's evidence is 6 orphan + 4 wedged — the wedged case is the larger share, so the secondary signal pays for itself.
• Server-side timeout reduction (drop lease window from 30min to 5min). Catches the stall faster but spam-amplifies — every legitimate long-running step (test suite, AI review, broker call) now has to heartbeat ≤5min or get reclaimed mid-flight. Activity-gating keeps the 30min window and gates whether to POST, which is the correct shape.
• Operator-driven kill via pkill -f heartbeat. The status quo. Lost because tonight's 6+4=10 manual interventions in one session is the failure mode this ADR exists to fix.

Supersedes: none (extends warp#683 heartbeat contract) Superseded by: leave blank until a later ADR reverses this one