← Decisions operate mode →

Decisions

ADR-031: The observability stack runs on the same VPS as the apps it observes

[ADR-001](001-self-hosted-observability-stack.md) committed us to self-hosting Loki + Prometheus + Alloy + Grafana (+ Tempo in Phase 6). It did not answer *where* the stack runs.

#031

ADR-031: The observability stack runs on the same VPS as the apps it observes

Status: Accepted Date: 2026-04-21

Context

ADR-001 committed us to self-hosting Loki + Prometheus + Alloy + Grafana (+ Tempo in Phase 6). It did not answer where the stack runs.

Two shapes were live options:

  1. Dedicated observability VPS (separate machine, separate IP). Industry-standard layout. Isolates the blast radius of an observability backend that falls over from the apps it is meant to be watching.
  2. Co-resident with the app VPS. Single host runs both the Dark Factory app containers and the observability compose stack.

At our current scale (one Hetzner VPS, ≤10 apps, one developer) option 1 means provisioning a second VPS whose sole user is the first VPS, with a cross-host network path we would also own. The observability stack's monthly cost would roughly double.

The infrastructure/server-bootstrap/ kit is the script that materialises whichever choice we make. The layout decision locks in for that kit's lifespan: stage 04 creates gyrum-obs-net as an internal docker network, stage 08 attaches Loki, Prometheus, Alloy and Grafana to it. Only Grafana is exposed via Caddy at grafana.gyrum.ai — the other three are reachable only from inside the docker network.

Decision

We run the observability stack on the same VPS as Dark Factory apps for now. Isolation is at the docker-network layer, not the host layer:

  • gyrum-obs-net is an internal docker network (stage 04, --internal). Loki/Prom/Alloy are unreachable from the public internet and the app docker network (dark-factory-net).
  • Grafana is the only service with a port published on 127.0.0.1:3000; Caddy (already running on :443) reverse-proxies grafana.gyrum.ai to it.
  • Alloy reads the docker json-file logs at /var/lib/docker/containers/ — which only exists on the same host. This is the dependency that makes co-residence free; a cross-host Alloy would need the docker log API over TCP or a separate log forwarder.

Consequences

  • One VPS bill, not two. Same host, same disks, same SSH surface.
  • Blast-radius risk. A runaway Loki ingest can starve CPU or disk from the apps it is watching. Mitigation: --memory 512m / --cpus limits on each compose service (stage 08 sets restart: unless-stopped, disk quotas are the next lever we pull if we see contention). Full isolation is a migration away — see below.
  • The stack can't observe its own host dying. If the VPS is down, Grafana is down. We rely on Guardian (external heartbeat) to page us when the whole host is unreachable. This is the same failure mode as ADR-001's "Guardian heartbeats the stack from outside" clause — the co-residence decision makes the heartbeat non-optional, not merely prudent.
  • Retention tightens. Loki holds 7 days hot on the shared disk (see config/loki.yml); Prometheus holds 30 days. Both are half of what we'd run on a dedicated obs host with room to spare.
  • Migration path. When we outgrow the shared host, stage 08 moves to a new VPS with no app code change: Alloy's loki.write "default" endpoint flips from http://loki:3100 to http://obs.internal:3100, and services keep writing slog JSON to stderr as today (ADR-001's stderr-first design was chosen precisely to make this cheap). The app VPS keeps Alloy (as a forwarder), loses Loki/Prom/Grafana.
  • Network hardening. The --internal docker network is the only thing keeping Loki's ingest endpoint off the public internet. UFW doesn't block docker-published ports by default, so a future change that accidentally adds ports: "3100:3100" to Loki would expose it immediately. Stage 10's verify step does not yet check for this — it's called out as follow-up work.

Alternatives considered

  • Dedicated observability VPS. Industry-standard, cleaner blast-radius story, roughly 2× monthly cost at our scale. Revisit when either (a) the shared host is CPU-bound by Loki compactions or (b) we run more than one app VPS (observability is then per-host anyway, so centralising onto one obs VPS becomes strictly the right move).
  • Grafana Cloud. Rejected by ADR-001 for cost and data sovereignty.
  • Observability containers alongside app containers on the same docker network (no internal net). Would let us skip the gyrum-obs-net plumbing. Rejected: a misconfigured publish on Loki/Prom would expose the whole log store to the internet, and the internal-only docker network is the single control that prevents that.

Supersedes: none Superseded by: