ADR-064: gyrum-catalog library + hexagonal architecture

Status: Accepted Date: 2026-04-22 Related: ADR-060 (catalog-driven infrastructure), ADR-061 (extract pipelines + projects libraries)

Context

ADR-060 introduced capture-layer experiments (exp-315 Hetzner audit, exp-316 GitHub repo audit, planned exp-317 Prometheus drift, exp-318 Caddy routes, exp-319 fleet rollup). Each is a standalone shell script under experiments/exp-NNN-slug/, orchestrating its own plug-ins, writing its own JSON envelope, handling its own errors.

ADR-061 extracted the pipeline engine and project schema into standalone Go + TypeScript libraries (gyrum-pipelines, gyrum-projects). The catalog work needs the same treatment for the same reasons: every new audit experiment is re-implementing the same orchestration, envelope shape, secret-scrubbing, schema versioning, and target-type taxonomy.

The operator's framing on 2026-04-22 review of the audit experiments:

with the experiments to extract the catalog, how about a common library or hex architecture design so there's more of a consistency that can be used cataloguing the different services

The risks of not extracting:

• Envelope drift. exp-315 already had to define its own envelope; exp-316 followed but with subtle differences (e.g. target.type casing). Without a library, exp-330 will diverge again.
• Secret-scrubbing inconsistency. ADR-060's boundary rule says catalogs never carry secrets. Today each experiment script must enforce this in its own way. A common library can scrub by default.
• Plug-in reuse. A docker-containers plug-in is identical whether it runs inside exp-315 (Hetzner audit) or inside a future exp-330 (Kubernetes node audit). Without a registry, copy-paste is the only sharing mechanism.
• Schema migration pain. When schemaVersion bumps from 1.0 to 2.0, every consumer that hand-parses the JSON breaks. A typed library makes the migration mechanical.

Decision

Extract a third standalone library — gyrum-labs/gyrum-catalog — alongside gyrum-pipelines and gyrum-projects (per ADR-061). The catalog library is the canonical implementation of:

• The catalog envelope schema (matching docs/specs/catalog-spec.md)
• The Plug-in interface (Run(ctx, target) → Result)
• The Runner that orchestrates plug-ins against a target
• Cross-cutting concerns: secret scrubbing, timing, error capture, schema versioning, target-type taxonomy
• Storage adapters (filesystem, blob, git-versioned, in-memory-for-tests)

gyrum-catalog ships Go (canonical) + TypeScript (mirror), follows the same GitHub-standard skeleton as ADR-061, ships TDD-first.

Why a third library, not folded into gyrum-pipelines

Catalogs and pipelines are conceptually distinct:

• A pipeline composes work units (any work, generic). The runner doesn't care what the work produces.
• A catalog is a specific shape of work output — a versioned, structured observation of a target. The catalog library has cross-cutting concerns (secret scrubbing, schema version) that pipelines don't.

A catalog uses pipelines internally (the orchestration of plug-ins is a pipeline-style flow), but the catalog domain is its own thing. Folding it into gyrum-pipelines would either pollute the pipelines library with catalog-specific logic, or hide the catalog domain inside pipeline names. Cleaner separation: pipelines is the workflow primitive, catalog is the observation primitive that USES the workflow primitive.

Hexagonal architecture

The library is structured as a hexagon:

                         ┌─────────────────────┐
                         │       Domain        │
                         │  (no dependencies)  │
                         │                     │
                         │  Catalog, Target,   │
                         │  Plugin (interface),│
                         │  Result, Envelope   │
                         └──────────┬──────────┘
                                    │
                  ┌─────────────────┴─────────────────┐
                  │                                   │
              ┌───▼───┐                          ┌────▼────┐
              │ Ports │                          │  Ports  │
              │  (in) │                          │  (out)  │
              └───┬───┘                          └────┬────┘
        ┌─────────┴─────────┐               ┌────────┴────────┐
        │   RunCatalog      │               │     Storage     │
        │   ListPlugins     │               │     Scrubber    │
        │   ValidateSchema  │               │     ClockSource │
        └─────────┬─────────┘               └────────┬────────┘
                  │                                   │
        ┌─────────┴─────────┐               ┌────────┴────────────────────┐
        │    Adapters in    │               │       Adapters out          │
        ├───────────────────┤               ├─────────────────────────────┤
        │ CLI runner        │               │ Filesystem storage          │
        │ HTTP API          │               │ Blob (S3-compat) storage    │
        │ Pipeline-block    │               │ Git-versioned storage       │
        │ (in gyrum-pipelines│               │ In-memory storage (tests)   │
        │  via gy-block)    │               │ Regex secret scrubber       │
        └───────────────────┘               │ Real-clock / mock-clock     │
                                            └─────────────────────────────┘

Domain (zero dependencies)

package catalog

// Target identifies what is being captured. The taxonomy is closed —
// new types are minor-version additions to this package.
type TargetType string
const (
    TargetHetznerServer TargetType = "hetzner-server"
    TargetGitHubRepo    TargetType = "github-repo"
    TargetFleet         TargetType = "fleet"
    TargetPrometheus    TargetType = "prometheus-instance"
    TargetCaddyHost     TargetType = "caddy-host"
)

type Target struct {
    Type TargetType
    ID   string
}

// Plugin is the unit of capture. Every probe implements this.
type Plugin interface {
    Name() string
    Version() string
    Run(ctx context.Context, target Target) (Result, error)
}

// Result is what a plugin returns. It MUST be JSON-serializable.
// The scrubber walks the tree before serialization.
type Result struct {
    Data any
}

// PluginResult is one slot in the catalog envelope.
type PluginResult struct {
    Name        string
    Version     string
    OK          bool
    DurationMs  int64
    Result      Result
    Error       string
    Stderr      string
}

// Catalog is the envelope. Stable schema; matches catalog-spec.md.
type Catalog struct {
    SchemaVersion           string
    Target                  Target
    CapturedAt              time.Time
    CapturedBy              string
    CapturedBySchemaVersion string
    Plugins                 []PluginResult
    Warnings                []string
}

The domain has zero external dependencies. No HTTP libraries. No SSH libraries. No filesystem. Pure types + interfaces.

Ports (interfaces over external concerns)

// Storage persists catalogs. Implementations: filesystem, blob, git, memory.
type Storage interface {
    Put(ctx context.Context, c *Catalog) (CatalogRef, error)
    Get(ctx context.Context, ref CatalogRef) (*Catalog, error)
    List(ctx context.Context, target Target, opts ListOpts) ([]CatalogRef, error)
    SetBaseline(ctx context.Context, target Target, ref CatalogRef) error
    Baseline(ctx context.Context, target Target) (*Catalog, error)
}

// Scrubber removes secrets from a catalog before serialization. Default
// implementation: regex over common patterns (PEM headers, GitHub PAT
// shapes, JWT shapes, AWS keys). Pluggable so consumers can add their
// own patterns.
type Scrubber interface {
    Scrub(c *Catalog) (*Catalog, []ScrubFinding, error)
}

// ClockSource — for test determinism on CapturedAt.
type ClockSource interface {
    Now() time.Time
}

Core (use case)

// RunCatalog is the central use case: take a target + a list of plugins,
// orchestrate, return a Catalog. Pure given the ports — no I/O directly,
// only through interfaces.
type RunCatalog struct {
    Storage  Storage
    Scrubber Scrubber
    Clock    ClockSource
    CapturedBy string  // experiment id, e.g. "exp-315-hetzner-server-audit"
}

func (r *RunCatalog) Run(ctx context.Context, target Target, plugins []Plugin) (*Catalog, error) {
    started := r.Clock.Now()
    cat := &Catalog{
        SchemaVersion: "1.0",
        Target:        target,
        CapturedAt:    started,
        CapturedBy:    r.CapturedBy,
        CapturedBySchemaVersion: "1.0",
    }

    for _, p := range plugins {
        slot := r.runOne(ctx, p, target)
        cat.Plugins = append(cat.Plugins, slot)
    }

    scrubbed, findings, err := r.Scrubber.Scrub(cat)
    if err != nil {
        return nil, fmt.Errorf("scrub: %w", err)
    }
    if len(findings) > 0 {
        for _, f := range findings {
            scrubbed.Warnings = append(scrubbed.Warnings, fmt.Sprintf("scrubbed: %s", f.Pattern))
        }
    }

    if _, err := r.Storage.Put(ctx, scrubbed); err != nil {
        return nil, fmt.Errorf("storage.Put: %w", err)
    }

    return scrubbed, nil
}

func (r *RunCatalog) runOne(ctx context.Context, p Plugin, target Target) PluginResult {
    started := r.Clock.Now()
    result, err := p.Run(ctx, target)
    elapsed := r.Clock.Now().Sub(started).Milliseconds()
    if err != nil {
        return PluginResult{Name: p.Name(), Version: p.Version(), OK: false, DurationMs: elapsed, Error: err.Error()}
    }
    return PluginResult{Name: p.Name(), Version: p.Version(), OK: true, DurationMs: elapsed, Result: result}
}

The core is deterministic given its ports. Tests inject mock Storage, mock Scrubber, mock Clock — no I/O. Tests run in microseconds.

Adapters

Adapters are external implementations of ports. Each is its own subpackage so consumers import only what they need (small bundle in TS, narrow Go import graph):

pkg/
├── catalog/                  # Domain (no deps beyond stdlib)
├── runner/                   # Core use case (depends on catalog only)
├── scrubber/
│   └── regex/                # Default Scrubber implementation
├── storage/
│   ├── memory/               # In-memory (tests, smoke runs)
│   ├── filesystem/           # Files in a directory
│   ├── git/                  # Filesystem + auto-commit
│   └── blob/                 # S3-compat
├── plugins/                  # Reference plug-ins (callers can write their own)
│   ├── docker/               # docker-containers plug-in
│   ├── ports/                # listening-ports plug-in
│   ├── systemd/              # systemd-services plug-in
│   ├── disk/                 # disk-usage plug-in
│   ├── packages/             # installed-packages plug-in
│   ├── caddy/                # reverse-proxy-config plug-in
│   ├── github/               # github-* plug-ins (workflows, branch-protection, ...)
│   └── prometheus/           # prometheus-targets plug-in
├── targets/                  # Target-type registry, validation
├── ssh/                      # SSH executor used by server-side plug-ins
├── ghapi/                    # GitHub API client used by github plug-ins
└── cli/                      # CLI adapter — `gy-catalog run --target ... --plugin ...`

How plug-ins compose with this library

A plug-in is a Go type implementing catalog.Plugin. The SSH port is a two-interface design — Executor hands out per-host Session handles, and Session owns the remote /tmp/audit-* workspace for the run's duration:

// Executor connects to a target host; returns a Session handle.
type Executor interface {
    Connect(ctx context.Context, host string) (Session, error)
}

// Session is the per-host handle that owns the /tmp/audit-* workspace
// for the run's duration. Close is responsible for cleaning up remote
// state regardless of success/failure.
type Session interface {
    PrepareRemoteDir(ctx context.Context) (string, error)
    CopyFile(ctx context.Context, localPath, remotePath string) error
    Run(ctx context.Context, cmd string) (stdout []byte, err error)
    Close() error
}

Example plug-in using the port:

package docker

type Plugin struct{ Exec ssh.Executor }

func (p *Plugin) Name() string    { return "docker-containers" }
func (p *Plugin) Version() string { return "1.0" }

func (p *Plugin) Run(ctx context.Context, target catalog.Target) (catalog.Result, error) {
    if target.Type != catalog.TargetHetznerServer {
        return catalog.Result{}, errors.New("docker plug-in requires server target")
    }
    sess, err := p.Exec.Connect(ctx, target.ID)
    if err != nil { return catalog.Result{}, err }
    defer sess.Close() // cleans up /tmp/audit-* on the remote
    out, err := sess.Run(ctx, `docker ps --format '{{json .}}'`)
    if err != nil { return catalog.Result{}, err }
    parsed, err := parseDockerPS(out)
    if err != nil { return catalog.Result{}, err }
    return catalog.Result{Data: parsed}, nil
}

Bash shell scripts in experiments/exp-NNN-slug/plugins/ STAY (they're the on-target executables); the Go side wraps them as the Plugin interface implementation. Or — once the library matures — plug-ins can be pure Go, using the SSH adapter to run arbitrary commands directly. Both styles coexist; bash plug-ins are easier to write and ship to the target, Go plug-ins are easier to test in isolation.

How experiments use the library

After extraction, an experiment shrinks dramatically. exp-315's audit-server.sh becomes either:

• A thin wrapper that exec's gy-catalog run --target hetzner-server --id $SERVER --plugins docker,ports,systemd,disk,packages,caddy --out $EXP_DIR, OR
• A 30-line Go program in the experiment dir that imports the library and registers its plug-ins

Either way, the orchestration logic (timing, error capture, envelope, scrub, storage) lives in the library, NOT in the script. New experiments contribute plug-ins and target types, not orchestration code.

TypeScript mirror

The TypeScript side has the SAME domain types and a read-only consumer for:

• Validating catalogs at the wire boundary (frontend reads /api/v1/catalogs/... JSON; types come from the library)
• Running scrubber client-side (defense-in-depth — never trust the server forgot to scrub)
• Diff visualisation: audit:diff block output is structured; the TypeScript side renders it natively without re-parsing
• Type-safe import wizards: see ADR-061's gyrum-projects mirror pattern

Cross-language fixture parity (per ADR-061) is enforced via shared fixtures/ and a parity.yml workflow.

Repository skeleton (gyrum-catalog)

Follows the GitHub-standard layout from ADR-061 § "Repository layout" exactly (LICENSE, CONTRIBUTING, SECURITY, CODE_OF_CONDUCT, .github workflows, docs/concepts/userguide/tutorials/examples/reference, tests/, fixtures/, ts/, pkg/). Specifically for gyrum-catalog:

gyrum-labs/gyrum-catalog/
├── README.md                 # Quick start: define a plugin, run, get JSON
├── docs/
│   ├── concepts.md           # Domain, ports, adapters, plugins
│   ├── userguide.md          # Full reference
│   ├── tutorials/
│   │   ├── 01-first-plugin.md
│   │   ├── 02-custom-storage-adapter.md
│   │   ├── 03-cross-target-plugin.md   # one plug-in working on multiple Target types
│   │   ├── 04-catalog-diff-and-baseline.md
│   │   └── 05-rag-context-from-catalog.md   # RAG use case (mirrors ADR-061's pipeline RAG tutorial)
│   ├── reference/
│   │   ├── envelope-schema.md            # mirror of catalog-spec.md
│   │   ├── plugin-interface.md
│   │   ├── target-types.md
│   │   └── scrubber-rules.md
│   └── faq.md
├── pkg/                      # Go canonical impl
├── ts/                       # TypeScript mirror
├── fixtures/                 # Shared catalog JSONs (valid + invalid)
├── examples/                 # End-to-end runnable demos
└── ...                       # standard GitHub-standard files

Migration path

Same pattern as ADR-061: small PRs, application repos depend on the library once it exists, application code shrinks.

Each step independently shippable. Application repos work throughout.

Consequences

Easier:

• New audit experiments are smaller — they contribute plug-ins, not orchestration
• Envelope drift becomes impossible — the library owns the envelope; can't invent a different one
• Secret scrubbing is on by default — opt-OUT is explicit, not opt-IN
• Cross-target plug-ins become possible (one plug-in works for both Hetzner servers and dedicated bare-metal, etc.)
• Visualisation: TypeScript consumers get typed access to the catalog tree
• The catalog domain is independently testable; mock storage, mock scrubber, mock clock — no real I/O in unit tests

Harder:

• Three libraries to coordinate (pipelines, projects, catalog) and their release cadences
• Cross-library refactors: a target-type addition in catalog might warrant a block addition in pipelines
• Schema migration discipline: bumping SchemaVersion requires a TypeScript mirror update + parity-test fixture rewrite

What we sign up to maintain:

• Catalog envelope as a stable contract (catalog-spec.md remains the human reference; the library is the machine reference; CI parity test ties them)
• Plug-in interface stability — any signature change is a major version bump on gyrum-catalog
• The hexagon: never let infrastructure (HTTP, SSH, FS) leak into the domain. Adapters are the boundary; static analysis can enforce.
• Target-type taxonomy (closed enum). Adding a target type is an explicit contribution, not a YOLO string.

Decisions that evolved during implementation

The ADR's original SSH-port sketch was a one-shot Exec.Run(ctx, host, cmd) -> out. When the SSH adapter actually shipped in gyrum-catalog v0.0.8 (gyrum-labs/gyrum-catalog#7) it settled as a richer two-interface design: Executor.Connect(ctx, host) -> Session, where Session exposes PrepareRemoteDir / CopyFile / Run / Close.

This shape matches exp-315's real needs — scp-then-run-then-cleanup — and gives Session.Close a natural home for remote /tmp/audit-* cleanup, without the caller threading a directory path through every Run call. The one-shot sketch would have forced either a separate port for file transfer, or leaky per-call host+workdir parameters on every invocation.

This is not a breaking change to ADR-064's thesis. The hexagonal architecture (domain / ports / adapters) is unchanged; only the SSH port's signature evolved between sketch and implementation.

Alternatives considered

• Keep orchestration in each experiment. Rejected: drift surface area scales with experiments. We already see drift in just two scripts (exp-315 vs exp-316).
• Fold catalog into gyrum-pipelines. Rejected: catalog has its own domain (envelope, scrubber, target taxonomy) that pollutes pipelines. The libraries SHOULD be small and focused; this is a third focus area.
• Use a generic data-pipeline library (Apache Beam, Prefect, Dagster). Rejected: those are full systems with their own runtime requirements. catalog is small, opinionated (single-target capture, not data ETL), and benefits from being purpose-built.
• Pure Go library only, no TypeScript mirror. Rejected: same reasoning as ADR-061 — schema drift between Go server and TS frontend is the default state without a shared library; we are not paying for it.

How this hooks into ADR-058/059/060/061/062/063

• ADR-058 (5-mode UI): catalog viewers live in Infra mode; tab on each server detail page rendered by gyrum-ui from gyrum-catalog's TS types
• ADR-059 (project-first IA): a project's catalog membership comes from joining its repos field with capture-by-target
• ADR-060 (catalog-driven infrastructure): this ADR is the implementation layer for ADR-060's two-layer model
• ADR-061 (libraries extraction): same skeleton, same TDD, same GitHub-standard. Three libraries side-by-side, coordinated via shared fixtures and shared review discipline
• ADR-062 (design system / Web Components): catalog tree visualisation is a <gy-catalog-view> Web Component; uses gyrum-catalog's TS types
• ADR-063 (tenant scoping + portability): each catalog carries a tenant: ... field (already in the envelope schema); export-import bundles serialise per-tenant catalogs

Open questions for follow-up

• Plug-in versioning policy across the registry. When docker-containers bumps from v1.0 to v2.0, does the runner accept both, or refuse old? Lean: accept both for one minor release window, then drop. Document policy in docs/reference/plugin-interface.md once written.
• Caching. Some plug-ins are cheap (1ms), some are expensive (installed-packages SSHes + greps a long list). Cache key is (target, plugin-version, ttl). Out of v1; revisit when audits run per-minute.
• Streaming results. Long captures (fleet rollup) might benefit from streaming partial results to the consumer instead of buffering. v1 is buffered; revisit when fleets exceed ~100 targets.
• WASM: Could the Go core compile to WASM and let the TS side run the same orchestration locally (e.g. for "audit my own browser" type use-cases)? Cute but speculative. Out of v1.