ADR-083: Rule promotion engine — auto-hoisting prose rules to devtools

Status: Proposed Date: 2026-04-24 Related: ADR-077 (session-independent agent coordination layer), ADR-078 (trigger-driven playbook orchestration), ADR-079 (dark-factory approval substrate), ADR-080 (warp as durable trigger queue)

Numbering note. The user brief named this "ADR-082". Slot 082 was filled earlier today by 082-experiment-experiments-url-split.md (landed via PR #345 ahead of this draft). Per the README numbering rule ("Never reuse a number, even for a superseded record"), this decision takes the next free slot — 083. Inbound links in the brief that cite "ADR-082 rule promotion" should resolve here.

Decision (one paragraph)

Candidate rules surfaced from the findings journal, memory feedback files, admin-merge log, and agent-report stream can be auto-drafted into devtools checks, and — once proven — auto-filed as approvals under ADR-079's substrate for one-click human promotion. The engine runs in three phases (surface → auto-draft with simulation → auto-file). It reuses existing infrastructure: findings.jsonl as the pattern source, the ADR-079 approval directory as the gate, the /operate Pending Approvals card as the UI, and the Phase 3b/3c approval watcher as the executor. It has a mandatory deny-list (narrative, architectural, judgment decisions) enforced as code with test coverage, a kill-switch (gyrum-rule-demote), a recursion guard (cannot promote rules about itself, throttled to 2/week, disableable via env var), and a phased rollout where each phase is gated on data from the prior one. No part of the engine ships in this ADR — this is design; implementation follows in sequenced PRs against devtools.

Context

Today's observation — the structural-over-prose rule

Memory feedback (feedback_structural_over_prose.md, codified earlier in the 2026-04-24 session) ranks reliability layers:

The session's lived pattern is consistent: every prose rule drifted at least once; every structural rule held. The codified consequence is "when a new invariant is named, push it down a layer." This ADR is the first downstream consequence of that memory.

The hand-hoist pattern

During the 2026-04-22..2026-04-24 window, prose rules that had proved themselves were hoisted into devtools by hand:

Each hoist took 30–90 min of human attention: recognise the pattern, write the check, test it, land it. The pattern is legible enough that a machine can do the recognition; the judgment about whether a pattern should become a check is still human, but the sorting and drafting is mechanical.

User direction

"can that happen automatically — a really good idea once pressure tested goes into devtools automatically?"

Yes — but only with a conservative pressure-test bar, a deny-list for the categories where pattern-matching is dangerous, a kill-switch, and a recursion guard. This ADR is the design.

Non-negotiables inherited from prior ADRs

From ADR-079: operational approvals live in-repo as YAML files, not in ~/.gyrum/ journals. From ADR-077: agent-visible surfaces should read from the existing journals, not replicate them. From the structural-over-prose memory: the engine must itself be structural — a prose rule saying "promote carefully" would be the weakest possible mechanism for promoting prose rules.

Decision

1. Pressure-test bar

A candidate rule must meet all five of these conditions before the engine will draft a promotion. The bar is enforced in code, not in a reviewer's head.

1. Repeated occurrence — the candidate pattern has fired in ≥3 incidents across ≥2 sessions. Single-session bursts do not qualify, no matter how many incidents — a single session can be the outlier.
2. Stable resolution — every one of those incidents was resolved the same way (same fix, same structure). Carve-outs, special cases, or context-dependent resolutions disqualify the candidate — the engine is looking for a mechanical rule, and a rule with exceptions is still judgment.
3. Mechanical verifiability — a cheap programmatic check exists: grep, an API call, a file hash, a git predicate. The check runs in <5s on a developer machine. If the check would require an LLM call to evaluate, the candidate fails this condition.
4. Low blast radius on misfire — a false-positive prompts or warns; it does not silently block legitimate work. The engine refuses to draft any check whose failure mode is "human cannot continue" without an explicit override path.
5. Not in a judgment category — see §4. The Gate 2 category filter is enforced at read time, before any of conditions 1–4 are evaluated against a candidate.

A candidate failing any one of the five is rejected with a structured explanation recorded back to the candidate journal, so the human reviewer sees both the match and the disqualifier.

2. Three phases

Phase 1 — Surface (weekly cron)

A new CLI gyrum-rule-candidates scans four sources:

• ~/.gyrum/findings/findings.jsonl — cross-agent findings (the durable record from ADR-077 Phase 1).
• <project>/memory/feedback_*.md and the user-memory directory — the prose rules that keep getting re-emphasised.
• ~/.gyrum/admin-merges.log — the admin-override record (which carries why the gate was overridden and is therefore a rich source of "the gate was wrong, not the PR" patterns).
• ~/.gyrum/agent-reports.jsonl — once #237 (agent-report) ships, the structural reply stream from every completed agent run.

Scoring function:

score = occurrence × stability × mechanizability

where occurrence is the count from condition (1), stability is the fraction of incidents with the same resolution (must be 1.0 to clear condition (2)), and mechanizability is a classifier output in [0, 1] based on whether the resolution can be expressed as a grep/API/hash check.

Output: dark-factory/docs/rule-promotion-candidates-<YYYY-MM-DD>.md, one row per candidate, sorted by score descending. A human opens it, reviews, and decides which to promote. Phase 1 is output-only — no PRs drafted, no approvals filed.

A concrete example from today: stacked-PR detection (#241) would have scored high in Phase 1 — it hit ≥3 incidents this session alone (admin-merged a stack, found drift, had to re-order), the resolution was the same each time ("refuse to push if base is an open PR"), and the check is a one-line gh pr view $base --json state call.

Phase 2 — Auto-draft (on-demand via gyrum-rule-promote <id>)

A human picks a candidate from the Phase 1 output and runs gyrum-rule-promote <id>. The engine:

1. Generates the check implementation (grep rule, hook script, API call) and the devtools test.

2. Opens a PR against devtools in the style of the existing hand-hoists (#183, #188, #236, #240, #241) — same file layout, same test fixture shape.

3. Runs simulation: replays the new check against 90 days of git/PR history for every repo the check applies to. Measures:

   • Accuracy — percentage of the known past incidents the check correctly flagged.
   • False-positive rate — percentage of legitimate commits/PRs the check would have incorrectly blocked.
   • Regression risk — percentage of prior merged PRs that would have been retroactively blocked.

4. Writes simulation results into the PR body as a YAML block:

   simulation:
     window: 2026-01-24..2026-04-24
     repos_scanned: [devtools, dark-factory, distill-gyrum-ai]
     accuracy: 0.97       # 97% of known incidents correctly flagged
     false_positive_rate: 0.01
     regression_risk: 0.00
     prior_prs_retroactively_blocked: 0
   

5. Tags the PR rule-promotion and requests the standard three-persona review.

The human reviews the code and the simulation numbers, then merges normally. No admin-merge — the rule-promotion engine earns no override privileges.

For stacked-PR detection, Phase 2 simulation over the last 90 days would have flagged the 4 stacked pushes actually seen without flagging any single-branch push. Accuracy 1.0, false-positive 0.0.

Phase 3 — Auto-file via ADR-079 (only for proven candidates)

A candidate that (a) passed Phase 2 and landed, (b) hit the accuracy threshold (default ≥95%), (c) false-positive rate (default ≤2%), and (d) zero catastrophic retroactive regression, is eligible for auto-file on next run.

"Auto-file" means: the next time a pattern of the same shape is surfaced in Phase 1, the engine writes an approval YAML directly to dark-factory/approvals/pending/<id>.yml with class: auto-promote. The file surfaces in the /operate Pending Approvals card (shipped today per ADR-079 Phase 3a). A human clicks Approve; the Phase 3b endpoint flips the status; the Phase 3c watcher (approval-watcher) merges the devtools PR, records the execution in approvals/executed/, and archives.

Net path: pattern → candidate → drafted PR + sim → approval YAML → one-click promotion. Human final say survives unchanged; the hoisting overhead disappears.

Stacked-PR detection, at Phase 3 maturity, would flow: Phase 1 surfaces the pattern on Monday; auto-draft runs the simulation Monday evening; the YAML lands in pending/ Tuesday morning; the operator sees it on the /operate card at 08:05, clicks Approve, and the devtools PR merges. Total human wall-time: ~15 seconds.

3. The self-hosting move

No new infrastructure is built for the engine. It reuses:

The factory promoting its own rules uses its own approval mechanism. The engine is an ADR-079 client, not an ADR-079 extension. If ADR-079's substrate is ever replaced, the rule-promotion engine migrates by pointing at the new substrate — no glue code is owned by this ADR.

4. Four-gate defense-in-depth (amended 2026-04-24)

Amendment note (2026-04-24, task #244). The original draft named a single deny-list check as the mechanism that prevents judgment rules from auto-promoting. One gate is one drift away from a regression. This section is replaced with a four-gate model: each gate is independently structural, no single gate is trusted alone, and a human appears only at Gate 4. The deny-list categories (formerly the whole of §4) survive inside Gate 2 — they are now one filter in a stack of four, not the stack itself. See §9 for the open question this closes.

A candidate must pass all four gates in order. Each gate runs at a different point in the candidate's lifecycle, defends against a different failure mode, and is enforced structurally — not as reviewer prose.

Gate 1 — Write-time (structural default)

Layer: structural default. Defends against: uncategorised findings silently entering the candidate pool. If a finding never carries a category, a later gate has no metadata to filter on. Implementation: gyrum-record-finding refuses to write an entry without an explicit --category flag. The tool fails closed — no default category, no interactive fallback. Callers that want a finding journalled must classify it at write time. Status: SHIPPED as devtools #91 / v0.1.59. Prerequisite: none (already landed; this ADR cites it as the substrate Gate 2 depends on).

Write-time categorisation is a structural default (not a check) because the only way to bypass it is to not call the tool at all — and any code path that bypasses the tool is out-of-scope for the engine by construction (the engine only reads entries it knows are categorised).

Gate 2 — Read-time (structural check)

Layer: structural check. Defends against: judgment-category findings being scored as promotion candidates. An accepted finding tagged narrative or architectural must not reach the scoring function. Implementation: Phase 1's scan (gyrum-rule-candidates) filters by category == structural-eligible before applying the §1 scoring function. Entries with any other category — including the six judgment categories listed below — are excluded from the candidate pool at read time, not rejected at promote time. The filter lives at the top of the scan loop; a finding that slips through is a bug in the filter, not a close call.

The six judgment categories must never pass Gate 2:

1. Narrative / taste — rules sourced from words like "feels", "clunky", "fragmented", "clean". Pattern-matchers cannot evaluate aesthetic claims.
2. Architectural seams — anything that would introduce a new module boundary, a new service, a new data store, or a new trust boundary. These require ADRs, not hoists.
3. Approval-override reasons — admin-merge justifications are case-by-case judgment. A pattern of "we overrode for reason X" does not license auto-promoting "always override for reason X".
4. People decisions — who reviews, who approves, who owns. The matcher checks CODEOWNERS, reviewer-list, and approver changes.
5. Product direction / strategy — anything touching a customer-visible promise, pricing, product name, or roadmap.
6. Context-dependent — any rule whose correctness depends on information the pattern-matcher cannot see (team headcount, ongoing contract, time-of-day, phase of a migration).

The categories are encoded as predicates in devtools/rule-promotion/category-filter.yml (renamed from deny-list.yml to reflect that the filter is now read-time, not promotion-time). For each category, a fixture in devtools/rule-promotion/testdata/filtered/<category>/ contains a past incident that quantitatively meets §1 conditions 1–4 but must still be excluded because its category disqualifies it. The engine must filter each fixture; a filter regression fails CI.

Status: Phase 1 MVP. Prerequisite: Gate 1 shipped (the scan has nothing to filter on until entries carry categories).

Gate 3 — Simulation-time (structural check)

Layer: structural check. Defends against: a structurally-eligible rule that is still wrong — high false-positive rate, retroactive regression, or noisy against real history. Passing Gate 2 means the rule is in a promotable category; it does not mean the rule is correct. Implementation: Phase 2's simulation (gyrum-rule-promote) replays the candidate check against 90 days of git/PR history for every repo the check applies to and refuses to proceed unless false-positive rate < 2%, accuracy ≥ 95%, and zero catastrophic retroactive regression. The thresholds are configurable in devtools/rule-promotion/config.yml; the existence of the gate is not. Status: Phase 2 per §7 rollout. Prerequisite: Phase 1 (Gates 1–2) producing ≥3 hand-promoted candidates over 4+ weeks.

A candidate that fails Gate 3 is rejected with the simulation numbers recorded back to the candidate journal. The rejection is data, not drama — the same candidate may pass on a later scan if the underlying pattern changes (e.g. a formerly-noisy check is scoped tighter).

Gate 4 — Approval-time (structural + human)

Layer: structural default + human judgment. Defends against: a rule that passed Gates 1–3 but that a human would still reject on contextual grounds the pattern-matcher cannot see. Gates 1–3 are necessary but not sufficient; Gate 4 is the sufficiency gate. Implementation: Phase 3 files a class: auto-promote approval YAML to dark-factory/approvals/pending/<id>.yml under ADR-079's substrate. The /operate Pending Approvals card surfaces it; a human clicks Approve; ADR-079's Phase 3b/3c watcher merges the devtools PR. The human cannot be bypassed — the approval-watcher refuses to execute without an explicit approved status flip, which only the approve endpoint can write. Status: Phase 3 per §7 rollout. Prerequisite: Phase 2 (Gate 3) producing ≥2 successful promotions that stayed landed

14 days with zero demotions.

The human appears at exactly one point in the pipeline: Gate 4. Gates 1–3 are fully structural; the human is spared the routine sorting that the original hand-hoist pattern consumed (§ Context), and the human's attention is concentrated on the single decision the machine cannot make — the contextual yes/no. This is the design's productivity promise.

Why four, not one

A single gate — the original draft's deny-list — is a single point of failure. If the deny-list misses a category, every subsequent step of the pipeline runs against a misclassified input. Four gates are four independent filters: a miss in any one is caught by the next. The layering is not paranoia; it is the structural-over-prose principle applied to the engine that enforces it. No single gate trusted alone; each gate structural; human at exactly one layer.

5. Kill-switch (mandatory)

gyrum-rule-demote <rule-id> reverses any auto-promoted rule in one command. The demote:

1. Removes the check from devtools (revert commit on the promoting PR, or a new commit that disables the hook).
2. Logs the demotion to ~/.gyrum/rule-promotion/demotions.jsonl with a reason (required, freeform) and the triggering agent/user.
3. Bumps the devtools/VERSION patch level, so consumers notice.
4. Files the demote itself as an approval: a YAML lands in approvals/pending/<id>-demote.yml with class: auto-demote. Same one-click mechanism as promotion, in reverse.

A rule cannot be auto-promoted a second time for 30 days after a demotion (rate-limit on re-promote) to prevent flap loops.

Structural rigidity without an escape hatch is worse than prose drift. Every auto-promotion must be reversible by a single command and a single click.

6. Recursion guard

The engine is a rule about rules. It is itself subject to the engine it implements. Three guards:

1. Engine kill-switch — GYRUM_RULE_PROMOTE_DISABLED=1 in the environment halts Phase 1 cron, Phase 2 draft, and Phase 3 auto-file. The env check is the first line of each phase entry point. The kill-switch cannot itself be auto-promoted or auto-removed.
2. Meta-rule block — the Gate 2 category filter (see §4) has an explicit category for "rules about rule-promotion". The engine cannot surface, draft, or auto-file a candidate whose target path is devtools/rule-promotion/** or whose scanned source includes findings.jsonl entries tagged rule-promotion.
3. Throttle — a hard cap on promotions per week, default 2. The cap is configurable in devtools/rule-promotion/config.yml and enforced structurally: Phase 3 refuses to file a third approval inside a 7-day window. The throttle applies only to auto-files (Phase 3); Phase 2 human-merged promotions are uncapped.

7. Rollout

The engine ships in four phases. Each phase gates the next on observed data, not on a calendar — a later phase may not ship until the prior phase has produced evidence that the current one behaves as designed. The phase-to-gate correspondence (per §4) is:

Each phase lands exactly one gate. A phase may not ship without its predecessor gate in place (e.g. P1 cannot ship without P0; P2 cannot ship without P1). The gate order is not a suggestion — it is the dependency graph.

Phase 4 is deferred explicitly because it introduces feedback on the engine's own decisions, which is recursion-adjacent. The recursion guard handles hard cases; Phase 4 handles soft cases and needs its own ADR when the time comes.

8. Relationship to existing ADRs

• ADR-077 (coordination layer) — the engine reads its journals (findings.jsonl, agent-reports.jsonl). The §4 Gate 2 read-time filter is the named consumer of the findings journal's category field; ADR-077 owns the field, this ADR owns the filter. No structural change to ADR-077 is required, but a follow-up cross-reference note in ADR-077 naming ADR-083 §4 Gate 2 as the downstream reader will keep the two documents in lockstep. (The ADR-077 file itself lives elsewhere in the monorepo; the cross-ref PR is tracked separately from this amendment.)
• ADR-078 (triggers) — the weekly Phase 1 cron is a trigger under ADR-078's grammar. No changes to ADR-078.
• ADR-079 (approval substrate) — the engine files Phase 3 promotions as class: auto-promote approvals. A new approval class is added to policies/approval-classes.yml by a follow-up PR. No structural change to ADR-079.
• ADR-080 (warp queue) — once Phase 2c of ADR-080 lands, the engine's simulation runs can move from the local gyrum-rule-promote invocation to a warp-queued job with deduplicated idempotency keys. Purely additive; Phase 2 ships usable before ADR-080 Phase 2c lands.
• Nothing superseded. This ADR is purely additive across the existing substrate.

9. Open questions

1. Phase 3 thresholds. Default accuracy ≥ 0.95 and false_positive_rate ≤ 0.02 are placeholders. Tune after Phase 2 data lands — likely the false-positive cap is the tighter gate in practice.
2. Auto-demotion trigger. How does the engine detect "this rule fired wrongly N times" after landing, to surface a demote candidate? Same scoring function but on the landed-rules population, or a separate signal from operator feedback?
3. Graduation. Should rules that auto-promoted AND survived 90 days without a demotion be moved from "can demote via a single PR" to "needs an ADR to demote"? This would mirror the ADR immutability rule: the rule earned durability.
4. Backfill of prior hand-hoists. #183, #188, #236, #240, #241 are canonical examples — should their provenance be backfilled into the candidate journal so the engine has a cold-start corpus, or left untagged so the engine starts fresh from live data?
5. Multi-repo scope. The engine is specified to draft PRs against devtools. Rules that belong to a single project (e.g. a distill-gyrum-ai-specific lint) are out of scope — but some candidates will be ambiguous. What's the decision rule for "devtools-global vs project-local"?
6. Simulation corpus staleness. The Phase 2 simulation window defaults to 90 days. Repos with low PR velocity may not have enough history to simulate meaningfully. Below what PR-count does the simulation refuse to run (and force a human judgement)?
7. Cross-project contamination. If a rule is proven on project A and auto-filed as devtools-global, project B inherits it without ever having experienced the pattern. Acceptable (devtools is shared by contract) or requires opt-in per repo?

Resolved since original draft

• Q (original) — Single-gate deny-list sufficient? RESOLVED 2026-04-24 (task #244): no — replaced with the §4 four-gate defense-in-depth model. One gate is one drift away from a regression; four independent structural gates (write-time, read-time, simulation-time, approval-time) with a human only at Gate 4 is the shipped design. See §4.

Consequences

Easier:

• Prose rules that prove themselves reach devtools without a human having to spot the pattern and sit down to write the check. Today's 30–90 min per hoist compresses to ~15 seconds of human wall-time once Phase 3 is live.
• Promoted rules come with simulation numbers in the PR body — the reviewer decides with quantitative evidence, not gut feel.
• The engine is legible: every rejection is explained, every promotion is auditable, every demotion is recorded. The findings journal and the approvals directory together carry the full provenance.
• Self-hosting means one substrate to operate (ADR-079) for both human approvals and engine-filed approvals — no second-queue problem.

Harder:

• Someone operates the Gate 2 category filter (formerly named the deny-list). Categories will shift as the system matures, and a mis-categorised candidate (narrative sneaks into "structural-eligible" slot) is a correctness bug in the engine, not a taste disagreement.
• The simulation harness is non-trivial. Replaying 90 days of git history against a proposed check is an N × M operation where N = repos and M = commits. Phase 2 needs a time/cost budget from the start.
• Recursion guard is easy to specify and hard to prove absent. The meta-rule block relies on path and tag matching; a candidate that disguises itself (e.g. a generalisation that also applies to rule-promotion) could slip. Test coverage on the deny-list helps, but perfect coverage is not achievable.
• New approval class (auto-promote) expands ADR-079's policy file surface. Governance of policies/approval-classes.yml (ADR-079 §8) must cover it before Phase 3 ships.

Signed up to operate:

• Review the Phase 1 output weekly. The weekly cron is a promise; if the output goes unread for a month, the engine's value is evaporating.
• Keep the deny-list fixtures current. Each time a candidate is rejected by a human that the engine accepted (or vice versa), the fixture set should gain an entry — this is how the engine stays honest.
• Honour the throttle. If the throttle is hit, that is a signal to inspect the engine, not to raise the cap. Raising the cap without understanding why three rules are queuing for promotion in a single week is how runaway systems happen.
• Ship each phase behind its data gate. Phase 2 does not ship on enthusiasm; it ships on 4+ weeks of Phase 1 showing the engine ranks candidates the way humans do.

Alternatives considered

• Do nothing — keep hand-hoisting. The baseline. Rejected: the structural-over-prose memory explicitly names the drift problem, and the 2026-04-24 session's five hoists show the cost. The pattern is mechanical; leaving it manual is a choice to burn human attention on routine sorting.
• One-phase auto-promote — skip Phase 1 output-only + Phase 2 human-merged, go direct to auto-file. Rejected: zero pressure-test data before ceding the merge button is how the engine promotes its first bad rule. The three-phase rollout is not bureaucracy — each phase is the prior phase's proof.
• Skip simulation, rely on review. Rejected: a review of a check without a simulation is a reviewer guessing the check's accuracy. The simulation turns the review into a calibration exercise — the reviewer checks the numbers, not intuits them.
• Hand-promote forever; encode the bar as a human checklist. Rejected by the structural-over-prose memory itself — a checklist is prose, and prose drifts. The engine is the structural expression of the same bar.
• New substrate for rule-promotion (separate journal + CLI). Rejected: ADR-079 is the designated operational-approval substrate. Adding a second substrate for engine-filed approvals fragments the operator's mental model and builds the same drift this ADR is trying to fix — two places for the same kind of thing is worse than one place with a new class.
• LLM-judge instead of mechanical check. Rejected: condition (3) explicitly excludes LLM-evaluated checks. An LLM judge at promotion time converts a structural check into a prose one wearing a different costume, and gives up the "cannot drift" property that motivates the whole ADR.

Supersedes: none. Superseded by: leave blank until a later ADR reverses this one.

Reference PRs:

• Phase 1 implementation — follow-up PR (TBD) against devtools adding gyrum-rule-candidates + weekly cron.
• Phase 2 implementation — follow-up PR against devtools adding gyrum-rule-promote + simulation harness.
• Phase 3 implementation — follow-up PRs against devtools and dark-factory wiring the auto-promote approval class into the /operate card and the approval watcher.

Reference memory: feedback_structural_over_prose.md — this ADR is the first downstream consequence of that memory.

Canonical worked example: stacked-PR detection (#241). Phase 1 surfaces it from 3 findings + 1 admin-merge-log entry; Phase 2 drafts a gh pr view $base --json state check, simulates against 90 days (accuracy 1.0, false-positive 0.0); Phase 3 auto-files the approval; human approves in 15 seconds on the /operate card.