ADR-038: Tamper-evident audit stream is a separate sink, sibling to the app audit logger

Status: Accepted Date: 2026-04-21

Context

pkg/audit already exists: a structured, asynchronous audit logger backed by a Store interface, designed for services that need a high-throughput record of who did what and when. It writes through batches, targets ~10k events/sec, and exposes a Query method for admin/investigation surfaces.

That package answers "how do I log an audit event into my database without blocking the request?" It does NOT answer "how do I produce a tamper-evident record for billing or compliance reviews?" The two questions sound similar; they're actually different primitives:

• The app logger is OK to lose events on buffer-full backpressure ("audit logs should not crash the service"). Acceptable for ops queries. Unacceptable for an auditor reconstructing what happened.
• The app logger is OK to share a retention policy with the rest of the service's audit table. Compliance audit streams often need 7-year retention and a different access-control surface.
• The app logger writes into the same database the attacker may reach via an RCE. An attacker who can DELETE FROM audit_events has the same logs as the compliance auditor.

The security tranche docs (security-standards, security-tripwires) call for a tamper-evident record for auth, billing, and admin actions. This is the new primitive.

Decision

Ship pkg/audit/sink as a SIBLING sub-package of pkg/audit. The two coexist; services wire both.

pkg/audit/sink provides:

• Sink interface: Record(ctx, Event) error, Close() error.
• Event: Timestamp, Category, Actor, Target, Outcome, Details.
• NewStderrSink(): writes JSON lines to stderr with the literal AUDITLOG prefix so Alloy/Promtail can route audit lines into a dedicated Loki stream with its own retention.
• NewFileSink(path): appends JSON lines to path for services that need a retention policy separate from container stderr.

Hash-chain design lives in ADR-039.

Consequences

Easier:

• Services can wire both logger and sink. A typical setup records every auth event into the DB (for user-facing "recent activity") AND into the tamper-evident sink (for auditor review).
• The two packages have independent evolution: the app logger can change its Store interface without disturbing the compliance sink.
• Log pipeline splits cleanly: app logs go to the default Loki stream, AUDITLOG-prefixed lines go to a long-retention stream.

Harder:

• Two audit primitives to explain. README and tranche docs need to be clear about when to call which. We accept this over the alternative of forcing one primitive to cover both concerns badly.
• Double-writing (DB + sink) in hot paths costs ~2 mutex lock-release cycles per event. Measured: <2µs; negligible against request latency.

Signed up to operate:

• A log-pipeline contract with Alloy/Promtail: lines beginning AUDITLOG route to the audit stream. The literal is a stable string exported as sink.StderrPrefix. Changing it requires coordinated updates to deploy configs.
• Separate retention lanes in Loki/object storage per regulatory regime — owned by the observability tranche, referenced from here.

Alternatives considered

• Single pkg/audit with tamper-evident mode as a Store impl — rejected. The API shapes diverge: the app logger is async with batching, the sink is synchronous and ordered. Forcing them into one interface made both ugly.
• Add a HashChain() Store wrapper to pkg/audit — rejected. The chain has to be computed BEFORE the event crosses the Store interface (batching would obscure the chain boundaries). Hashing post-hoc from the DB is trivial to tamper with during the window between insert and hash.
• Cloud-native audit service (AWS CloudTrail-like) only — rejected as the only answer. Useful for deploy-level API calls; fine-grained app events need an in-process primitive. We may ship events to such a service AS WELL.

Supersedes: none Superseded by: —