ADR-116: gyrum-knowledge-base RAG (retrieval-augmented generation) service

Status: Proposed Date: 2026-04-26 Related: ADR-067 (playbooks-unified-primitive), ADR-090 (ai-as-scaffold-pipelines-as-runtime), ADR-111 (broker — sibling primitive), ADR-112 (principles tier — first-class indexed source), ADR-115 (principle-aware reviewers — first major consumer), ADR-117 (module guidelines — second major consumer)

Context

Today every sub-agent brief has to embed the relevant ADRs, principles, memory rules, and CLAUDE.md sections verbatim, because the agent has no canonical place to fetch them on demand. The result is bloated context, expensive token usage, and briefs that drift from the source within hours of a documentation update. PR reviewers (ADR-115) face the same problem from the opposite side: they need to find the principles relevant to the current diff and have no semantic-search surface to ask. Operators searching across 100+ ADRs, weeks of session logs, and hundreds of warp items rely on grep over directory layouts they have to know in advance.

An audit of gyrum-labs/* (100 repos, 2026-04-26) found no dedicated RAG (retrieval-augmented generation) or knowledge-base service. The closest matches (ai-research, autoresearch) are research-orchestration shapes, not context-indexing. The decision is captured in warp #343.

This pairs structurally with the broker (ADR-111): the broker is where model calls run (single point for routing, fallback, cost, audit); the RAG is what model calls know (single point for docs, ADRs, principles, memory). Together they let any agent issue "send a model call (broker) with the right context (RAG)" as a clean primitive.

Decision

gyrum-knowledge-base is a standalone service — sibling repo to gyrum-ai-broker, not a feature of it — that indexes the fleet's structural knowledge and exposes it via a semantic search API. The repo decision (new vs. extending broker) lands on new repo because the operational profile differs: the broker is light routing; the RAG is heavy storage plus indexing plus periodic re-embedding.

Indexed sources, defined in warp #343 r3:

• ADRs: dark-factory/docs/decisions/*.md (~115 today)
• CLAUDE.md sections: every repo, parsed by markers (gyrum-warp-section, gyrum-claude-step-v1, etc.)
• Fleet READMEs: every repo's README.md
• Session logs: dark-factory/docs/sessions/YYYY-MM-DD.md (chronological narrative)
• Principles: warp items with kind: principle (ADR-112)
• Memory: operator's ~/.claude/projects/.../memory/*.md via opt-in read-only mount
• Warp item descriptions: full text plus tag facets

Embeddings flow through the broker on the embed-passage@v1 route, so the model choice is policy-controlled, fallback-aware, and audited like any other call. Storage is pgvector on the factory-executor box (no internet exposure, ADR-conformant with warp #338). Re-indexing fires nightly via cron and on-merge via webhook from any indexed repo.

The query API is GET /api/v1/knowledge/search?q=<text>&filter[source]=adr|claude-md|principle|warp-item&filter[date_range]=<from>:<to>&top_k=5, returning ranked passages with {score, source_url, citation, snippet}. Auth is bearer-token (same scheme as warp) with per-token rate limits. A gyrum-knowledge-search CLI wrapper covers operator-shell queries; the factory /knowledge page covers the browser surface with facet filters and a "last index time / source counts / drift detection" header.

Consequences

What becomes easier.

• Sub-agent briefs shrink. The brief says "fetch any context you need from kb.gyrum.ai/search?q=..." instead of pasting four ADRs verbatim. Briefs become routing instructions, not context dumps.
• Reviewers (ADR-115) get a real "principles applicable to this diff" query path, not a curl warp/items?type=principle returning everything.
• Operator semantic search becomes a real surface: "what did we decide about vault?" returns ranked passages with ADR + principle citations across the fleet.
• Memory files in the operator's home directory become fleet-visible (opt-in, read-only). Today they are local-only and sub-agents cannot see them; tomorrow they are queryable like any other source.
• Documentation upkeep (warp #309) gets a measurable payoff: the docs are not just refreshed, they are findable and cited by every agent that needs them.

What becomes harder.

• A new service to run: re-indexing schedule, embedding cost, vector store, query API uptime, auth, rate limits. Mitigation: the corpus is small (~3000 vectors) and the load profile is gentle; this is not a hard service to operate, but it is one more.
• Storage cost is small but non-zero. ~575 documents × ~5 chunks ≈ 3000 vectors at typical embedding dimensions is well under 100 MB; trivial in absolute terms, real in service-budget terms.
• Stale-index risk: nightly re-index plus on-merge webhook is the answer for indexed repos, but memory files and operator-home sources need an explicit refresh trigger.
• Query-quality drift: the retrieval prompts and chunk strategy (warp #343 r4 — by markdown header for ADRs/CLAUDE.md, by paragraph for sessions, by warp-item field for principles) need tuning as the corpus grows. Mitigation: a query-quality regression set anchored to known-good queries from ADR-115 reviewer flows.

What we sign up to maintain.

• The gyrum-knowledge-base repo, its CI, its deployment, its bearer-token issuance, and its re-index cron + webhook.
• The chunk-strategy rules per source type (header-split for markdown, paragraph for sessions, field-split for warp items).
• The embed-passage@v1 broker route and its model pin.
• The /knowledge page on factory and the gyrum-knowledge-search CLI wrapper.

Alternatives considered

• Extend gyrum-ai-broker with a knowledge-base sub-feature. Rejected: different operational profile (heavy storage vs. light routing), different release cadence, different scaling shape. The broker is a hot path on every model call; the RAG is a periodic re-index plus a cool-path query. Co-locating couples deploy risk for no real benefit.
• Use a hosted vector DB (Pinecone, Weaviate Cloud). Rejected: the corpus is tiny and the query rate is low; hosted is overkill on cost and adds a fleet dependency on a third-party uptime. pgvector on the executor box matches the ADR-138-class isolation profile and is operationally trivial.
• Skip embeddings — full-text search via Postgres or ripgrep. Rejected: the consumer pattern is "find principles applicable to this diff" and "what did we decide about X" — both semantic, both poorly served by lexical match. Lexical search returns an ADR if its title contains the word; semantic returns it if its content answers the question.
• Index only ADRs (smaller scope). Rejected: the consumer set spans principles, memory, session logs, warp items, and CLAUDE.md sections. A scope-limited RAG forces every other source back into the verbatim-embed-in-brief pattern, defeating the whole point.

Supersedes: none Superseded by: none yet