ADR-004: Ship release_track as a first-class bounded label from day one

Status: Accepted Date: 2026-04-21

Context

Canary deployments are a Phase 3+ feature: split traffic between two replicas of the same service, compare error rate and latency between them, promote the new build when the canary is clean. The machinery to split traffic doesn't ship yet, but the machinery to compare the two tracks in dashboards has to be in place the moment it does — retrofitting a new log label across historical dashboards and alert queries after the fact is expensive.

Dashboards and alert queries referencing a label are easy to write on day one ({release_track="canary"}) and hard to rewrite after a fleet of services has been emitting without one. Loki's index is label-structured; the historical queries we lose when the label didn't exist before cannot be recovered.

A retrofit would also mean a coordinated upgrade across every service, a re-cut of every dashboard, and a period where "canary" means different things in different panels. Better to pay the trivial cost now.

The library already reads the RELEASE_TRACK env var at package init and attaches it to every log line — see gyrum-go pkg/observ/README.md (commit 7b5006f).

Decision

Every log line carries a release_track field. The library reads RELEASE_TRACK once at package init, defaults to stable, and attaches the value as a first-class attribute alongside service, version, commit. Alloy promotes release_track from a log field to a Loki label during ingest (see observability-architecture.md#labels-vs-fields).

The bounded values are stable and canary. Any other value is accepted by the library but will not match the Phase 3+ canary dashboards and is therefore a configuration bug.

Consequences

• Canary-vs-stable LogQL works today. {service="distill", release_track="canary"} | json | level="ERROR" returns sensible results the moment a canary replica comes up, without any code change to the service. The queries in the playbook assume this.
• All Phase 1 dashboards plan for the axis now. Panels that plot error rate include a release_track split in their group-by even when there is no canary replica. When one appears, the panel already shows it without a dashboard edit.
• The label exists with a near-zero operational cost. A single bounded attribute with two values adds trivial cardinality pressure on Loki — this is exactly what ADR-005's bounded-label rule permits. At the projected fleet size, the total label-value explosion from release_track is 2× the existing stream count, which stays far below the comfortable ceiling.
• Services are committed to the env-var contract. A service that does not read RELEASE_TRACK (e.g. one that predates the library) emits logs without the field, and its lines will be ungrouped in canary dashboards. The migration is cheap (import the library), but it is a commitment to uniform runtime identity across the fleet.
• Phase 3+ work is unblocked. When traffic splitting lands, the only new work is the splitter itself — dashboards, alerts, and the comparison story already exist.
• Convention over enforcement. The library reads the env var at init; a misconfigured deploy that omits RELEASE_TRACK gets the stable default rather than a crash. A canary accidentally labelled stable will be invisible as a canary. Deploy scripts must set the env explicitly; the standards call this out.

Alternatives considered

• Retrofit release_track when canary deploys arrive. Simpler now, deferred work later. Rejected: migrating historical dashboards and alert queries across multiple services is expensive and repetitive, and the Phase 5 alerts need release_track in their group-by to avoid false-positives during rollout. Doing it once, up front, is materially cheaper than doing it under deploy pressure.
• Use version alone as the canary signal. A canary replica has a newer version than stable, so in theory | version="0.2.0" replaces | release_track="canary". Rejected: version is high-cardinality over time (every release cuts a new value); promoting it to a Loki label would violate ADR-005. release_track stays bounded by construction.
• Read the env var at log-site instead of init. Would permit a mid-process flip. Rejected: slog handler construction caches the attribute at WithAttrs time, and a mid-process flip would be an operational footgun (which replica is "canary"? it just changed). A single read at init matches the intended deploy model.
• Use a canary=true|false boolean label instead of a string track name. Rejected: boolean labels close the door on a third track (e.g. shadow, experimental) that we are likely to want later. A small enum is cheaper than a retrofit from boolean to enum.

Supersedes: none Superseded by: