ADR-001: Self-hosted observability stack (Loki + Prometheus + Alloy + Grafana + Tempo)

Status: Accepted Date: 2026-04-21

Context

Phase 1 of the observability rollout ships a structured-logging library (github.com/gyrum-labs/gyrum-go/pkg/observ — see gyrum-go pkg/observ/README.md). Before services can emit logs, we need somewhere to send them, and we need to commit to a backend shape now because later phases (metrics, dashboards, alerts, tracing) depend on the choice.

Constraints:

Log volume at current cadence is already above the Grafana Cloud free-tier cap (50 GB logs / 10k metric series). Paid Cloud is billed per-ingest and scales with every new service.
We run services on our own VPS fleet. Outbound from the factory network to a third-party ingest endpoint is a policy and cost surface we would rather not open.
The factory model requires multiple services to share one observability substrate; per-service vendor signups are not acceptable.
We need full control over retention, redaction behaviour, and query surface — log content can contain regulated data even after in-process redaction (see ADR-007).

Decision

We self-host Loki + Prometheus + Alloy + Grafana + Tempo on the observability VPS. All five components run in the same docker-compose stack. Services emit to local docker stderr; Alloy tails the docker JSON log files on each host and ships to Loki over the internal network. The full flow is documented in observability-architecture.md.

The library writes slog JSON to stderr and contains no push client, no vendor SDK, and no HTTP dependency on the backend. Every vendor-specific choice in the chain terminates at Alloy.

Consequences

We operate the stack ourselves. That means disk capacity planning for Loki chunks, Prometheus TSDB sizing, Alloy upgrades, and on-call for the observability VPS itself. Guardian heartbeats the stack from outside so a dead Loki does not hide behind its own dashboards.
The migration path to Grafana Cloud is a config swap on Alloy — no service redeploy, no code change. See observability-architecture.md#migration-path-to-grafana-cloud. This is the reason the library is stderr-first.
Cost is flat (VPS lease) rather than per-ingest. New services land on the stack without changing our monthly bill. We can reason about unit economics without a metered meter in the middle.
Data sovereignty is total. Regulated fields that leak past the in-process redactor never leave our infrastructure. The compliance story is "we don't log that data", but the stack backstops it.
The stack is bigger than Grafana Cloud from day one. Four server components plus Tempo (idle) is more moving parts than a hosted offering. We accept that in exchange for the three bullets above.

Alternatives considered

Grafana Cloud free tier. Easiest to stand up. Rejected: the free-tier caps are below our current volume, so "easy" converts to "outage in week two". Paid Cloud solves ingest but opens the data-sovereignty and per-ingest-billing surfaces above.
Datadog / Splunk. Mature, full-stack, zero ops burden. Rejected on cost: per-host + per-GB pricing at our service count makes the monthly spend a meaningful chunk of the factory budget, and the lock-in is hard to reverse.
DIY Elasticsearch / ELK. Well-known shape, many ops engineers know it. Rejected on ops burden: Elasticsearch cluster operations (shard rebalancing, JVM tuning, version upgrades) consume more attention than Loki's object-store-backed design, and the Loki label model matches our cardinality discipline (see ADR-005) better than ES full-text indexing.
Just stderr + docker logs + grep. Zero infra. Rejected because "grep across five hosts by SSH" does not survive Phase 5 alert-driven triage, and because log retention via docker's default json-file driver is 300 MB × 3 files per container — not enough.

Supersedes: none Superseded by: