ADR-152: Rate-limit graceful degradation — wrapper-side detection plus server-side claim safety net

Status: Accepted Date: 2026-05-06

Context

Phase 0 of EPIC warp#1568 (EFF-8: sub-agent rate-limit graceful degradation). Tonight's session (2026-05-06) surfaced a failure mode that the existing dispatch wrapper and warp claim contract cannot recover from cleanly: three swarm agents (labels RRR, SSS, TTT) hit Anthropic's server-side rate-limit at the moment of dispatch. Each agent died at 0 tokens with API Error: Server is temporarily limiting requests · Rate limited, never reached its first PostToolUse invocation, and never POSTed a single /heartbeat. Each left:

• A stuck warp claim with no live heartbeat — the lease window (default 15min, see ADR-138's last_activity_at fallback) does eventually lapse, but for the operator watching the board the ticket reads "in flight" with no signal of life until manual triage.
• An empty PR or no PR — the dispatch wrapper opens the draft PR before invoking Claude Code, so the half-state confuses any subsequent agent that picks up the ticket and finds a branch with no commits.
• No diagnostic trail — neither the warp ticket nor the PR carries a "blocked: rate-limited" signal; the operator has to grep the dispatch logs to discover the agent died loudly with an Anthropic 429.

Per-RL-hit cost tonight: ~5 min operator triage per stuck claim. Three hits in one minute = ~15 min of operator time gone before the pattern was named. As fleet dispatch volume scales (warp#1497 swarm-dispatch-template), the pattern compounds.

Sister structural shape: warp#1318 / ADR-138 (activity-gated heartbeat) — that EPIC fixes silent-death-AFTER-success, where the agent ran for a while then crashed with the heartbeat loop still firing. EFF-8 fixes silent-death-ON-DISPATCH, where the agent never started doing work because Anthropic refused the first request. The two share the lease-lapse semantics but differ in which signal arrives first: ADR-138 uses parent-PID + activity-mtime to detect a wedged-but-spawned agent; ADR-152 uses dispatch-wrapper failure-output classification to detect an agent that never spawned a useful tool call. Sister cognitive shape: warp#1342 (stall pattern, review-pr READY-TO-MERGE directive) — same theme of swarm robustness, different surface.

The EPIC body sketched three contract surfaces — wrapper-side exit code, server-side claim field, audit-log line — but as prose, not as a normative spec. The original Phase 0 ADR (merged 2026-05-06 as warp#1572) proposed one shape; Phase 1 (devtools PR #349, warp#1568) and Phase 2 (warp PR #258, warp#1635) shipped the same day with several principled deviations from that proposal. This Accepted revision retroactively codifies what actually shipped so future work cites the contract that exists in code, not the contract that existed in proposal.

Decision

Rate-limit-on-dispatch is a first-class signal: the dispatch wrapper detects it via post-failure output classification, auto-releases the warp claim, posts a structured comment, and exits 75 (EX_TEMPFAIL) so the parent orchestrator can distinguish transient throttling from real failure; the warp claim endpoint additionally accepts an auto_release_on_silence_minutes field as a server-side belt-and-suspenders.

The shape — three contract surfaces, one ADR. Each invariant below is normative: deviations require an ADR amendment in the same PR that introduces them.

1. Wrapper-side detection (Phase 1 — gyrum-labs/devtools, PR #349)

The dispatch wrapper (spawn-agent.sh's spawn_dispatch) treats Anthropic rate-limit responses on the spawned Claude Code process as a structurally distinct exit, not a generic failure. Four invariants:

Invariant 1 — RL identification corpus

A failure is classified as rate-limited iff its captured combined output matches any of:

• The canonical CLI exit string: contains Server is temporarily limiting requests AND case-insensitive rate[ _-]?limited.
• HTTP 429 status: contains \b429\b AND case-insensitive rate[ _-]?limited|rate_limit.
• Anthropic SDK JSON error type: matches "type"\s*:\s*"(rate_limit_error|overloaded_error)".

The corpus is intentionally narrow. Anything outside it is treated as a non-RL failure and falls through to the existing exit-1 path. The narrow corpus is a guardrail against the false-positive class the EPIC pitfalls explicitly forbid: a transient network blip, a tool-call timeout, an unhandled panic — all would otherwise trigger an unwanted auto-release of partial work.

Invariant 2 — usage_limit is explicitly NOT rate-limited

The classifier rejects any output containing case-insensitive usage[ _-]?limit before any positive match attempt. Usage-limit is a different recovery class — clears at midnight UTC, not in minutes — and the wrapper-side auto-release / comment / exit-75 contract does not apply. A usage-limit-shaped failure surfaces as a regular non-zero exit; the operator handles it manually.

This is pitfall #2 from the EPIC body, lifted to a normative invariant: any future change that conflates the two recovery classes is a contract break.

Invariant 3 — best-effort reset-time extraction

The wrapper attempts to parse a reset-at timestamp from the failure output, in this order:

1. HTTP header form: anthropic-ratelimit-<bucket>-reset:<ws><iso8601> (e.g. anthropic-ratelimit-tokens-reset, anthropic-ratelimit-requests-reset).
2. JSON body form: "reset_at"\s*:\s*"<iso8601>".

Empty result is acceptable — the warp comment falls back to a "now-ish" wall-clock phrasing. Reset-time is operator-facing context, not control-flow input; the auto-release fires regardless of whether a reset-at was extracted.

Invariant 4 — structured exit code 75 (EX_TEMPFAIL)

On RL detection, spawn_dispatch returns exit 75 (EX_TEMPFAIL from sysexits.h). Generic agent failures continue to use exit 1. The exit-75 path is reserved for this signal so the parent orchestrator can distinguish "transient upstream throttling — try again later or pick a different ticket" from "real spawn failure — investigate". Exit 0 continues to mean clean success.

Why 75 and not the proposal's 87: EX_TEMPFAIL carries the same semantic ("temporary failure; user is invited to retry") in a value that is already standard across the Unix sysexits convention. Picking 87 would have been an arbitrary value with no shared meaning outside this contract; picking 75 makes the signal legible to any caller that already understands sysexits.

Invariant 5 — comment FIRST then auto-release

On RL detection the wrapper performs side effects in this exact order:

1. POST a structured comment to the warp ticket: POST /api/v1/items/<id>/comment with body containing the literal prefix blocked: rate-limited at <iso-ts> — sub-agent worker hit Anthropic rate-limit on dispatch (warp#1568 / EFF-8). followed (when extracted) by auto-resume eligible after <reset-at> and a closing paragraph clarifying that auto-release is NOT auto-resume.
2. POST the release: POST /api/v1/items/<id>/release (no body).

Both calls are best-effort: failures log to stderr but do not propagate (the wrapper's exit-75 already signals the RL upstream). Comment is posted first because the audit trail must survive a release-call flake — a stuck claim with a comment is recoverable by warp-release; an auto-released claim with no comment is invisible.

The leading literal blocked: prefix is the operator-facing convention so warp UI dashboards can grep a stable prefix. The trailing warp#1568 / EFF-8 cross-links the EPIC.

Invariant 6 — no auto-resume

The wrapper does not re-attempt dispatch. The auto-release frees the lease; a fresh agent with headroom may pick up the ticket from Ready, but the choice to dispatch again is operator-driven (or scheduler-driven if/when one exists). Lifted into the ADR so a future Phase 1.5 cannot quietly add auto-resume without an ADR amendment.

2. Server-side claim safety net (Phase 2 — gyrum-labs/warp, PR #258)

The warp claim endpoint accepts an optional new field that lets the server release a stuck claim when the wrapper-side detection itself fails (wrapper killed before exit-75 could be emitted; network partition between wrapper and warp; OOM / segfault before the detection branch runs):

Invariant 7 — nullable per-claim threshold

Migration 030_auto_release_on_silence.sql adds nullable auto_release_on_silence_minutes INTEGER on warp_items. Three properties hold:

• NULL = legacy behaviour. The cron sweep falls back to the universal 15-minute LeaseDurationMinutes default. Every existing in-progress row preserves the legacy path byte-identically; no in-flight claim is silently rebound by the migration.
• Non-NULL N = per-claim threshold. The cron sweep COALESCEs the per-row column against the default in its GREATEST(last_activity_at, last_heartbeat_at) + interval ... < now() freshness check, so the per-row interval takes precedence when set.
• Validation at decode-time. The /claim handler rejects zero / negative values with HTTP 400 invalid_silence_minutes so a malformed body is loud rather than silently persisting a useless threshold. There is no upper-bound check — operators with legitimate long-haul work (e.g. a 1440-minute supervised dispatch) are supported.

Why nullable not defaulted: a NOT NULL column with a default would silently flip every in-flight claim into the new auto-release rule on deploy day, reclaiming long-running operator-paused work without consent. The opt-in shape preserves backwards compatibility without a separate feature flag.

Invariant 8 — silent_death audit distinct from lease_expired

When the cron sweep reclaims a claim because the per-claim silence threshold (not the universal lease) fired, it writes event=claim.auto_released audit rows with two distinct shapes:

• Audit action/reason: silent_death (the operator-facing label for "agent died silently before its wrapper-side rate-limit detection had a chance to run").
• Audit details.reason: the string silent-death (kebab-case, written into the JSONB details payload alongside the per-row threshold).

Universal-lease reclaims continue to write the legacy lease_expired shape unchanged. The two distinct values let a future telemetry layer (E8.3, deferred) count silent-death incidents without scraping prose, and let operator triage filter the EPIC-relevant subset of reclaim events.

The two shapes (snake_case audit field, kebab-case details payload) are intentional: the audit field is keyed on the existing snake_case audit-action enum convention; the details payload keeps the operator-readable kebab-case shape that already appears in warp UI views.

3. Client-server contract (the composition guarantee)

The wrapper-side and server-side paths layer correctly:

• In the common case (wrapper exits cleanly), Invariant 5 fires first: the wrapper posts the comment and releases the claim within ~1 second of the RL detection. The server-side timer never fires because its preconditions (silence > threshold) are not met — the explicit release supersedes the silence path entirely.
• In the silent-death case (wrapper killed before Invariant 5 runs — operator SIGKILL, kernel OOM, network partition between wrapper and warp), Invariant 7's per-claim threshold catches the residue. The audit row's silent_death shape (Invariant 8) distinguishes this from the universal-lease fallback, so operator triage and future telemetry can quantify the silent-death rate independent of the loud-failure rate.
• The two paths are not redundant. Phase 1 alone leaves silent-death uncaught for up to 15 minutes (universal lease). Phase 2 alone forces every loud-RL incident to wait auto_release_on_silence_minutes (typically 10) before recovery. Together they bound recovery latency: <1s common case, ≤N minutes worst case.

Defence-in-depth, same shape as ADR-138's parent-PID + activity-mtime layering and ADR-083's rule-promotion defence-in-depth.

4. Phasing summary (as shipped)

The original ADR-152 proposal also called for gyrum-start-work to set auto_release_on_silence_minutes: 10 by default and expose --auto-release-silence-minutes <n> / --no-auto-release-on-silence flags. Those wiring changes did not ship in Phase 1; the field is currently set per-call by agents that opt in. Deferred: add the start-work wiring and flags as a follow-up ticket so the safety net layers automatically rather than per-claim. Tracked separately so this ADR closes against what actually exists in code.

Consequences

What becomes easier:

• A dispatch RL-hit is recovered in <1s without operator intervention in the common case (wrapper detects, releases, comments). The 5-min-per-hit triage cost from tonight's session goes to zero for the loud-failure path.
• Operator triage gains stable signals at three layers: warp UI dashboards filter on blocked: comment prefix (Invariant 5), audit logs filter on event=claim.auto_released action=silent_death for silent-death (Invariant 8), and parent orchestrators distinguish exit 75 from exit 1 (Invariant 4). All three are machine-parseable, which makes Phase 3 telemetry a thin layer on existing data rather than a parsing project.
• The wrapper-side and server-side paths layer correctly per Section 3: in the common case the wrapper's exit-75 path runs first and the server-side timer never fires; when the wrapper is killed before it can emit, the server-side timer catches the residue. Defence-in-depth, same shape as ADR-138 and ADR-083.
• The EFF-8 EPIC's contract is now locked against the implementations that exist; future RL-shape changes are contract-breaking and require an ADR amendment in the same PR. Phase 1 and Phase 2 reviewers (and any Phase 3 work) verify against this ADR rather than re-deriving the contract from PR diffs.

What becomes harder:

• The classifier's narrow corpus (Invariant 1) requires Phase 1 to keep up with Anthropic's response-shape evolution. If Anthropic introduces a new rate-limit-shaped failure (a fourth JSON error type, a different CLI string), the wrapper either misclassifies or fails to detect. Mitigation: the regex set lives in one file (spawn-agent.sh's spawn_is_rate_limited) with a comment pointing back to this ADR; Anthropic-shape drift gets its own warp ticket with a clear "extend the corpus" scope, and the change rides an ADR amendment so the contract stays in sync.
• The exit-75 constant is now load-bearing in the dispatch contract. Any wrapper that calls the dispatch wrapper (today: parent orchestrators that interpret swarm-dispatch exit codes) must respect it. Repurposing the value requires an ADR amendment.
• The auto_release_on_silence_minutes field broadens the warp claim API surface. Per ADR-117 (module guidelines), the warp HTTP handler guideline must be updated when the field semantics change; per ADR-115 (principle-aware reviewers) the persona reviewer for any future change to this surface fetches both the principle (claim API stability) and the guideline (handler shape) before reviewing.
• The two audit shapes for silent_death (snake_case action, kebab-case details payload — Invariant 8) require the future telemetry layer to key on whichever shape its consumer needs. The two-shape choice is intentional but the duplication is a small surface-area cost.

Phasing / migration path (as shipped):

• Phase 0 (this ADR). Original proposal merged 2026-05-06 as warp#1572. This Accepted revision (warp#1634) reconciles the contract against what shipped same-day in Phase 1 and Phase 2. No code changes.
• Phase 1 (gyrum-labs/devtools PR #349, merged 2026-05-06). Wrapper-side detection (Invariants 1–6).
• Phase 2 (gyrum-labs/warp PR #258, merged 2026-05-06). Server-side timer + audit (Invariants 7–8).
• Phase 3 (deferred). Silent-death-rate dashboard consuming Invariant 8's audit distinction. Read-only consumer; no contract changes; lands when fleet RL-hit volume justifies the work.
• Wiring follow-up (deferred). gyrum-start-work opting every claim into auto_release_on_silence_minutes: 10 by default with --no-auto-release-on-silence opt-out. Ticket to be filed.

Alternatives considered

• Auto-release on any agent failure, not just RL. Simpler — no failure-shape classification, just "agent process exited non-zero → release the claim". Lost because the false-positive class is too wide: a legitimate test-suite failure, a transient network blip mid-build, an unhandled panic in a tool — all would cause the agent to release a claim it had real partial work on. Operator's recovery path then is harder, not easier, because the partial work is on a branch the next agent does not know about. The EPIC body's pitfall list explicitly warns against this; the ADR keeps RL as the one signal (Invariant 1).
• Server-side-only safety net (no wrapper-side detection). Drop Phase 1; just ship Phase 2 and let the server-side timer eventually catch every silent-death case after auto_release_on_silence_minutes minutes. Lost on latency: with auto_release_on_silence_minutes: 10 the operator waits 10 minutes per RL-hit instead of <1s. With three RL-hits in one session that is 30 minutes of stuck-board state versus 3 seconds. The server-side path is the safety net, not the primary path.
• Wrapper-side-only detection (no server-side safety net). Drop Phase 2; rely on the wrapper to emit exit-75 reliably. Lost on the failure modes the wrapper cannot self-recover from: SIGKILL, kernel OOM during the wrapper's own auto-release POST, host-side network partition between the wrapper and warp. Tonight's evidence had clean wrapper exits, but the larger fleet will eventually hit one of those edge cases; the server-side timer is the audit-trail-leaving fallback. Same defence-in-depth shape as ADR-138.
• Auto-resume on auto-release. When the wrapper detects RL-hit, after RL_RESET_AT, redispatch automatically. Lost because re-attempt policy is operator-driven for now: the fleet does not yet have a scheduler that can reason about which tickets are worth re-attempting versus which were RL-hit because they were lower-priority and should remain queued. Auto-resume is a future feature, gated on a scheduler ADR that does not yet exist; this ADR explicitly forecloses adding it without the scheduler ADR landing first (Invariant 6).
• Detect RL by HTTP response code on a probe call before dispatch. Instead of dispatching and watching output, have the wrapper POST a tiny /v1/messages probe before invoking Claude Code; if the probe returns 429, skip dispatch. Lost on cost (one probe per dispatch is wasteful — 95%+ of dispatches succeed) and on shape (the probe and the actual agent invocation are different rate-limit pools in Anthropic's accounting; a passing probe does not guarantee a passing dispatch). The post-failure classification path catches the actual failure that matters.
• Reduce the warp lease window globally to 5 minutes. Catches the silent-death case faster without any new code. Lost the same way ADR-138 lost the same alternative: every legitimate long-running step (AI review, broker call, test suite) would have to heartbeat ≤5min or get reclaimed mid-flight. Activity-gating + per-claim opt-in is the correct shape; this ADR layers on top of it.
• Bounded silence-minutes range [5, 60] (the original Phase 0 proposal). Reject thresholds outside the range with HTTP 422. Lost during Phase 2 implementation: operators with legitimate long-haul supervised work (1440 minutes for a day-long dispatch) are valid use cases, and a bounded range would force them to either disable the safety net entirely or game the bound. The decode-time > 0 check (Invariant 7) keeps the loud-rejection of malformed values without foreclosing legitimate operator use.
• Exit code 87 (the original Phase 0 proposal). Use an arbitrary value outside the sysexits range and outside gyrum-validate-playbook's reserved 64–78. Lost during Phase 1 implementation: 75 (EX_TEMPFAIL) carries the same semantic in a value that is already standard across Unix sysexits. The cost (the value falls inside gyrum-validate-playbook's reserved range) was reconciled by carving out 75 explicitly as the EFF-8 RL signal in the wrapper docs.
• Branch the wrapper between rate_limited and usage_limit_hit (the original Phase 0 proposal). Both branches auto-release; only the operator-facing comment differs. Lost during Phase 1 implementation: usage-limit has a fundamentally different recovery contract (clears at midnight UTC, not minutes) and conflating the two — even with different comment text — invites the operator to misjudge recovery timing. Invariant 2 explicitly rejects usage-limit shape upstream of the auto-release path; the operator handles usage-limit manually.

Supersedes: none (extends warp#683 heartbeat contract; layers on ADR-138's activity-gated heartbeat; reconciles the original Phase 0 proposal merged as warp#1572 against the implementations that shipped in warp#1568 / devtools PR #349 and warp#1635 / warp PR #258) Superseded by: leave blank until a later ADR reverses this one