ADR-165: GHCR retention policy + tag taxonomy — 3-layer auto-prune defense

Status: Proposed Date: 2026-05-07

Related: warp#1748 (this ADR's ticket, Phase 0 / G0.1), warp#1743 (parent EPIC: GHCR cost reduction), warp#1735 (sister category-axis ADR — category: warning ticket class for L3 drift), warp#1733 (sister budget tile: GitHub Actions %-used in factory dashboard), warp#1721 (sister structural EPIC: self-hosted-only fleet CI — same ADR-then-substrate shape), warp#1551 (sister structural EPIC: sitemap-as-source-of-truth), warp#1406 (rollover precedent: 1-week soft-warn before hard-fail), ADR-157 (sister gate semantics), ADR-163 (sister bidirectional CI gate).

Context

Operator audit 2026-05-07 surfaced 4,455 PRIVATE container versions across the three highest-volume repos on GHCR (GitHub Container Registry, the OCI image registry that backs ghcr.io): ghcr.io/gyrum-labs/warp (1,455), ghcr.io/gyrum-labs/warp-ui (1,440), ghcr.io/gyrum-labs/ai-frontend (1,560). The pattern is "tag every CI (continuous integration) build, never prune" — every PR, every merge, every release adds new image versions; nothing removes them. Private GHCR storage is ~$0.25/GB-month plus ~$0.50/GB outbound bandwidth; with ~3 years of accumulated tagging across the fleet, the bill is no longer rounding error.

Operator considered switching to Docker Hub Pro ($9/mo) and rejected it: Docker Hub's 200-pull / 6-hour authenticated rate limit would intermittently break CI on busy days. The structural answer is to stay on GHCR and add the retention discipline that should have been there from day one — automatic, observable, and reversible-on-purpose.

The shape to import is the one already pinned by ADR-157, ADR-159, and ADR-163: commit canonical truth (here the tag taxonomy), wire a bidirectional gate that fails-on-drift (here the L3 weekly audit cron), and give edge cases an audited-override path (here a per-package "long-tail" allow-list metadata flag). Today nothing in the substrate refuses an unbounded version count; once this ADR lands, the warp#1743 substrate phases ship the gates and the bill drops ≥80% month-over-month without further operator input.

Decision

The GHCR retention contract has five pinned components: the tag taxonomy itself, the L1 GH-native retention rule, the L2 release.yml tag-discipline rule, the L3 drift-audit cron + warning-ticket auto-file, and the secrets-baked-in audit gate that precedes any image's visibility flip to public. Every fleet repo's release.yml MUST conform to the taxonomy after the rollover window; the L3 cron files a category: warning ticket (per warp#1735) on threshold breach; visibility flips to public ride the secrets-audit gate, never operator judgement alone.

1. Tag taxonomy

Every image pushed to GHCR MUST carry exactly one tag from the canonical taxonomy:

vN.N.N is the load-bearing tag — it names a release and never moves. :sha-XXXX is the audit trail — every CI build keeps one for 30 days so a sha-bisect across a recent regression is always possible. :latest is convenience for docker pull and never load-bearing; if it expires, no production reference breaks because production refs use :vN.N.N. Any other pattern (:dev, :nightly, :branch-XYZ, custom artefact tags) is rejected by the L2 release.yml lint (component 3).

2. L1 (Layer 1) — GH-native retention rule

GHCR's org-level retention rule is the load-bearing prune. Operator applies once via GH (GitHub.com) Settings → Packages → Container retention with the rule set:

• Keep all versions matching v[0-9]+.[0-9]+.[0-9]+ indefinitely.
• Keep last 30 days of all other versions.
• Honour per-package "long-tail" metadata flag (a custom GH package label — see component 6) that exempts a specific package from the 30-day window.

The retention rule fires nightly on GH's side; no fleet-side cron is required for the prune itself. L3 (component 5) only watches that the rule is firing as expected.

3. L2 (Layer 2) — release.yml tag-discipline rule

Every fleet repo's release.yml MUST emit both :vN.N.N (for L1's keep-forever bucket) and :sha-XXXX (for L1's keep-30d bucket) on every release. :latest is permitted but optional. CI workflows that produce throwaway images (per-PR builds for e2e tests) MUST use :sha-XXXX only and MUST NOT use :vN.N.N-shaped tags — version numbers belong to releases, not to CI noise.

A devtools-templated lint job (substrate phase G2.1, separate ticket) parses release.yml files in the PR diff, fails the PR if a job pushes a tag that doesn't match the taxonomy. Same shape as ADR-163's self-hosted-only-validate gate — a small bash script + 6-test-case suite + soft-warn rollover.

4. L3 (Layer 3) — weekly drift-audit cron

A weekly cron runs factory-cli ghcr audit (substrate phase G3.1, separate ticket) which counts versions per repo and emits structured output. The cron logic:

• For each fleet repo: count versions on the corresponding GHCR package.
• If count > threshold (initial: 100 versions per repo; tunable via factory-cli config), file a warp ticket with category: warning (per warp#1735).
• Ticket body names the repo, the count, the threshold, and the most-likely cause (retention rule disabled / new tag pattern slipping through / package not yet on the rule).
• If counts are clean: no ticket; cron logs success.

The 100-version threshold is calibrated for "L1 prune is firing": :vN.N.N accumulates ~10/year, :sha-XXXX accumulates ~30/month and prunes after 30d, :latest is one-or-zero — total expected ~40-50 in steady state. 100 is a 2× headroom that flags genuine drift without false-positive noise.

5. Soft-warn rollover

The L2 lint and L3 cron both enter service in soft-warn mode for 1 week per repo (per warp#1406's precedent — title-format-gate's soft-warn-then-hard-fail pattern). During the soft-warn window the gate logs the violation but exits 0; after the window flips, the gate hard-fails. The window flip is per-repo (a date in the workflow YAML) so heavily-modified repos can move at their own pace. Operator can bump the rollover date by editing the workflow.

6. Secrets-baked-in audit gate

Phase 4 of warp#1743 flips selected images from private to public. Public GHCR storage is FREE (no per-GB billing) so the cost saving on high-volume images is ~95-100%. But public images are world-readable forever; an image that bakes a secret into an env var or label is a credential leak that no rotation closes.

Before any image's visibility flips to public, a factory-cli ghcr secrets-audit <package> MUST run and MUST emit a clean verdict. The audit:

• docker inspect every kept version of the image.
• Greps env vars, labels, and known-config-file paths for the substring patterns: AWS (Amazon Web Services) access keys, GitHub tokens, JWT (JSON Web Token) secrets, Slack tokens, base64-shaped secrets > 32 chars.
• Fails on any match; outputs the matching version + field for remediation.

The audit is mandatory. Operator authorisation alone does not suffice — the structural gate is the audit pass. If the audit surfaces a match, the image stays private until the offending build is excluded (or the package gets a fresh-rebuild-with-redacted-history workflow, separate work).

The factory-cli ghcr secrets-audit subcommand is substrate phase G0.2 (separate ticket).

Consequences

Every fleet repo's release.yml conforms to the taxonomy after the rollover window. The org-level retention rule prunes ≥80% of today's accumulated versions (the keep-30d bucket dominates since :vN.N.N is the rare event). The L3 cron files warning tickets when drift returns — a future repo authored by an agent who didn't read this ADR fails the gate immediately rather than accumulating quietly until the next bill audit.

Phase 4 (selective go-public, 4 children: warp + warp-ui + ai-frontend + ai-research) closes the bill on the highest-volume repos. The secrets-audit gate prevents the foreseeable failure mode where an operator flips visibility on an image with a long-baked secret. Per warp#1743 acceptance, month-over-month GHCR bill drops ≥80% once Phases 1-4 land.

The 100-version threshold is intentionally generous for the soft-warn window. If the L3 cron files no warning tickets in week 2 of operation, a successor ADR can tighten to 60. Tightening is cheaper than loosening — false negatives (drift accumulating quietly) are the failure mode this ADR exists to prevent.

The audited-override pattern lives in the per-package "long-tail" metadata flag rather than in YAML (Yet Another Markup Language) comments because GHCR retention rules don't read YAML; the rule mechanism is per-package metadata. Same audit principle as the comment-anchor patterns in ADR-157 and ADR-163, different storage substrate.

Alternatives considered

Switch to Docker Hub Pro ($9/mo). Rejected: Docker Hub's 200-pull / 6-hour authenticated rate limit would intermittently break CI on busy days, particularly during overnight swarm runs that pull tens of images in quick succession. The cost saving is real but the rate-limit risk is a structural CI fragility we would buy in exchange. GHCR has no equivalent rate limit.

Single-layer defense — GH-native retention only. Rejected: GH's retention rule is a black box; if it stops firing (UI (user interface) bug, policy edit, package metadata mismatch), nothing surfaces the regression until the next manual bill audit. The L3 weekly drift cron exists to catch that failure mode within a week rather than a month or quarter. Defense-in-depth is the same shape as ADR-083 (rule promotion) and ADR-157 (sitemap-validate's two-track gate).

Per-repo retention rules instead of org-level. Rejected: scaling to N repos means N rule applications, N runbooks, N future-author "did anyone apply this here" failures. Org-level applies once and survives new-repo creation by default. The trade-off is a single rule shape across all repos; the per-package "long-tail" metadata flag absorbs the rare per-repo edge case.

Aggressive prune (e.g. keep last 7d of :sha-XXXX). Rejected for v1: 30d aligns with GH's 30-day undelete window so a too-aggressive prune is recoverable; 7d would not be. A successor ADR can tighten after L3 cron data confirms steady-state version count is well under threshold.

Mass-flip everything to public to bypass private storage cost. Rejected: blanket-public is a credential-leak vector for any image with a baked-in secret across its history. The §6 secrets-audit gate is the structural answer — selective go-public per Phase 4 with an audit gate, not blanket flip.

Migrate to ECR (Amazon Elastic Container Registry) / private docker registry self-hosted. Rejected: registry self-hosting is a high-touch substrate (HA (high availability), backups, GC (garbage collection), auth) that the current fleet substrate doesn't have; ECR is non-trivial AWS account management. GHCR is already integrated with the GitHub-native CI flow; the marginal cost of correct retention is a one-time runbook + script + cron, not a registry migration.

References

• warp#1748 — this ADR's ticket (Phase 0 / G0.1).
• warp#1743 — parent EPIC.
• warp#1735 — sister category-axis ADR (warning category for L3 drift tickets).
• warp#1733 — sister Actions budget %-used tile.
• warp#1721 — sister structural EPIC (self-hosted-only fleet CI).
• warp#1551 — sister structural EPIC (sitemap-as-source-of-truth).
• warp#1406 — rollover precedent.
• ADR-117 — module guidelines tier (this ADR is fleet-tier; per-repo guidelines layer on top).
• ADR-157 — sister gate semantics (comment-anchor + reason validation).
• ADR-159 — sister bidirectional CI gate.
• ADR-163 — sister structural ADR (same shape, different surface).