← Decisions operate mode →

Decisions

ADR-005: Bounded labels, unbounded fields — cardinality discipline

Loki indexes by *labels*. Every distinct combination of label values creates a new stream; chunks are allocated per stream. High-cardinality labels (user IDs, request IDs, raw URLs) fragment streams, explode chunk…

#005

ADR-005: Bounded labels, unbounded fields — cardinality discipline

Status: Accepted Date: 2026-04-21

Context

Loki indexes by labels. Every distinct combination of label values creates a new stream; chunks are allocated per stream. High-cardinality labels (user IDs, request IDs, raw URLs) fragment streams, explode chunk counts, and degrade query performance for everyone on the cluster. Prometheus has the same failure mode for metric label cardinality when Phase 2 lands.

The failure mode is catastrophic-on-a-delay: a service can add a user_id label, ship the change, run fine for a week as the user set grows, then melt the cluster when it crosses a cardinality threshold. By the time symptoms appear, weeks of chunks exist in the bad shape and cleanup is expensive.

This decision sets the rule every service follows when choosing between a Loki label and a log field. The rule is also implemented in Alloy's relabel pipeline: most attributes stay as fields; only a named list is promoted to labels.

Decision

Labels are for things we group by; fields are for everything else. An attribute becomes a Loki label only if all three of the following are true:

  1. The set of values is bounded by construction (not by current traffic).
  2. The bound is a small number a human can name — under 20 for comfort, under 100 as a hard ceiling.
  3. A dashboard panel or alert query needs to group by this attribute.

The bounded label set as of Phase 1:

  • service (one per service)
  • release_track (stable or canary — see ADR-004)
  • level (DEBUG/INFO/WARN/ERROR)
  • container_name, compose_project (derived by Alloy from docker metadata)

Phase 2+ will add a small, named list of additional labels (status_class, err_category) — each one is an ADR amendment or a new ADR, never a unilateral change.

Everything else — request_id, user_id, trace_id, probe_id, route, url, referrer, anything free-text — is a field. Fields are queryable in LogQL via | json | foo="bar" at parse time. Not indexed, still searchable.

The rule is stated in prose at observability-architecture.md#labels-vs-fields and observability-standards.md#cardinality-rules; the gyrum-go library documents the same rule at pkg/observ/README.md#cardinality-discipline.

Consequences

  • Dashboards stay fast. Queries filter on a small, indexed label set; content search happens inside chunks already narrowed by labels. Query latency stays flat as log volume grows.
  • Investigators filter by field at query time. | json | user_id="u_123" is slower than a label filter but still sub-second on a label-narrowed stream. The ergonomics cost is acceptable.
  • Every new label requires an ADR or amendment. Adding tenant_id as a label (for example) needs explicit sign-off. This is deliberate friction — the cost of the wrong label is large enough that "one more label won't hurt" is never the right framing.
  • Alloy's relabel pipeline is load-bearing. The promotion from JSON field → Loki label happens in Alloy, not in the library. A service that accidentally emits user_id gets it dropped back to a field on ingest; the label set cannot grow unless Alloy's config grows with it. See observability-architecture.md, step 4.
  • Field-query cost is paid by the reader, not the cluster. This is the correct distribution — a rare high-cardinality investigation costs more than a common label-filter query, and the expensive one is the one we do less often.

Alternatives considered

  • Label everything structured; let Loki sort it out. Maximum query flexibility. Rejected outright — this is the exact failure mode Loki documents, and "let the database sort it out" has burned every team that tried it. Cardinality explosions do not reverse cheaply.
  • Fields only; no Loki labels at all. Maximum safety. Rejected: without labels, every query full-scans chunks. {service="distill"} would have to become | json | service="distill" and query latency balloons. The goal is bounded labels, not zero labels.
  • Case-by-case judgement, no written rule. Relies on every service author understanding Loki internals. Rejected: the failure mode is silent-until-catastrophic, and "judgement" is not an audit trail when a new label is proposed. The rule is written down so a PR reviewer can cite it.
  • Entropy-based auto-promotion (promote fields that have low cardinality). Rejected: "low cardinality today" is not "bounded by construction", and the promotion would backfire the first time a service's user count or feature-flag set grows.

Supersedes: none Superseded by: