ADR-003: JSON-per-line on stderr, not logfmt or protobuf

Status: Accepted Date: 2026-04-21

Context

Once we committed to slog (ADR-002) and a Loki-backed stack (ADR-001), the on-wire line format still had a choice. slog ships both JSONHandler and TextHandler; third-party handlers offer logfmt, protobuf, MessagePack, and others. The format decision propagates through three surfaces:

Loki's parser. LogQL has | json and | logfmt built in; anything else needs a custom parser.
Human emergency path. When Loki is unreachable, the only log access is docker logs <container> on the VPS. Whatever comes out has to be grep-able by a half-awake operator at 3am.
Structure preservation. Nested fields (error with cause chain, redacted attrs, groups) need a format that can represent them without collapsing.

Decision

Every log line is a single-line JSON object on os.Stderr. The line is produced by slog.NewJSONHandler with the library's ReplaceAttr hook and wrapped by the message-scrubber decorator (see ADR-007). No other format is offered by the library.

Shape:

{"ts":"2026-04-21T14:02:11Z","level":"INFO","msg":"server starting","service":"distill","version":"0.1.4","commit":"abc1234","release_track":"stable"}

Consequences

Loki parses it natively. | json in LogQL gives queryable access to every field without a parser definition. {service="distill"} | json | err_category="upstream_timeout" works from day one.
docker logs <container> | jq '...' works for emergency grepping. A human with no stack access can still pull useful structure out of a log file. The jq recipes in the playbook assume this shape.
Nested structure survives. Groups, error-cause chains, and redacted attrs all round-trip through JSON cleanly. logfmt would flatten or drop them; protobuf would require a schema registry.
Line size is larger than logfmt. Keys repeat per line ("service":"distill" on every event). We accept the bytes in exchange for the human readability and the free Loki parser. Compression on the Loki chunk side absorbs most of the cost.
One line per event is a hard rule. The docker json-file driver assumes newline-terminated records; a multi-line JSON payload would be split into separate entries. slog's JSONHandler guarantees single-line output.
stderr, never stdout. stdout is reserved for program output (test harnesses, tools, piped CLIs). Mixing logs into stdout muddies the contract; stderr is universally "diagnostic byte stream".

Alternatives considered

logfmt (key=value key=value). Lighter on the wire, easier to scan visually. Rejected: escaping rules for values with spaces or quotes are fiddly, structured values (arrays, nested groups) collapse to strings that need re-parsing, and LogQL's | logfmt parser is more forgiving but less expressive than | json. The format survives casual eyeballing; it does not survive structured querying.
Protocol Buffers. Fastest to serialise, smallest on the wire, schema-enforced. Rejected: opaque to humans, requires a schema distribution story, and docker logs <container> outputs bytes nobody can read without tooling. Operator ergonomics outweigh the serialisation win at our volume.
MessagePack / CBOR. Same tradeoff as protobuf without the schema story. Rejected for the same reason.
Custom framed format. A bespoke format would let us optimise for exactly our fields. Rejected on "boring is better" grounds — JSON is the default every tool in the pipeline understands, and the cost of custom framing is a parser in Loki + a parser in every operator's head.

Supersedes: none Superseded by: