ADR-092: Host/Project separation — two-images-per-project, host owns edge

Status: Accepted Date: 2026-04-25

Context

Tonight's first end-to-end deploy of warp to warp.gyrum.ai failed for hours and required a manual quick-path because the existing pipeline conflated two things that should never have been one. provision-and-deploy-warp.yaml runs 14 steps that mix host setup (apt installs, ufw, observability scrape registration, security daemons) with project setup (git-clone the warp source, render the warp .env from vault, bring up the warp compose stack). The ansible warp role does the same: tasks/docker.yml is host-shaped, tasks/clone.yml and tasks/env.yml and tasks/compose.yml are project-shaped, and tasks/admin_token.yml is project-shaped. There is no role for TLS termination at all — the design quietly assumed each project would terminate its own edge via a cloudflared container in its compose file, which is why warp/docker-compose.yml carries a cloudflared service nobody configured.

The collision shows up as four concrete failures, all rooted in the same missing cut:

warp-clone task hangs forever because the warp repo is private and the host has no GitHub credentials. The host-shaped role is reaching across the cut to do project-shaped work (fetching project source).
warp:local image is unbuildable on the host without source on disk, because the project compose file references a build-from-context image but the project pipeline assumes that source is on the host. The project's image identity is leaking into the host's filesystem.
cloudflared is in warp's compose but never configured, because the role assumed TLS was a project concern. DNS at warp.gyrum.ai is a direct A record; nothing was opening :443. The project was carrying edge responsibility it should never have had.
No path to add a second project to the same box. Putting distill on warp-01 alongside warp would mean re-doing every host decision (caddy config, ufw rules, observability registration) inside distill's compose. The host pipeline has no notion of "I am hosting multiple projects" because the project pipeline is the only thing that ever touches the host.

The fleet has been growing past the one-project-per-box assumption for weeks (distill, buzzy, hello-world all queued). Without a structural cut, every new project will re-invent host setup, and we will keep finding the same four failures.

Decision

A gyrum deployment is two pieces with a one-shape interface between them:

HOST is a VPS plus the shared runtime every project on that box needs: Docker, Caddy (with auto-LE — automatic Let's Encrypt certificate issuance and renewal — fronting every vhost on the box), ufw + auditd + Falco + AIDE for security, node-exporter + promtail for observability, and a GHCR (GitHub Container Registry) deploy token logged in. The host knows nothing about any specific project. ssh into a freshly-provisioned host and grep for "warp" and you should find nothing.
PROJECT is a docker-compose stack that declares its own services (one backend container, one frontend container, plus any datastores like postgres), an .env requirements list, and a hostname binding. The project does not ship Docker, does not ship Caddy, does not ship its own TLS strategy, and does not know which box it lands on.
The interface between the two is one shape: the project hands the host a (hostname, upstream_container, upstream_port) triple. The host's caddy gains a vhost. The host pulls the project's images from GHCR. The host brings up the project's compose stack on the host's docker daemon. The host routes /{*} for that vhost to the upstream container over the shared docker network.

Three rules follow from this cut and are non-negotiable for every project shipped on the fleet:

Two images per project, always. Every project publishes exactly two images to GHCR: ghcr.io/gyrum-labs/<project> (the backend Go binary or equivalent) and ghcr.io/gyrum-labs/<project>-ui (the frontend, typically nginx:alpine serving a static SvelteKit build). The frontend is never embedded in the backend via go:embed or equivalent. The frontend never ships as files on disk via rsync. If a project genuinely has no UI today, it ships an nginx:alpine serving a one-page placeholder so the contract holds and the next deploy of the same project doesn't require a topology change.
Host owns edge — projects MUST NOT bundle caddy, cloudflared, nginx-front, or any TLS-terminating service in their compose. Caddy is a host-piece, version-pinned by the host pipeline, shared across every project on the box. Cloudflare Tunnel, if reintroduced, is a host-piece (a per-host cloudflared service the host's pipeline manages) — never a per-project one. A project compose file that declares a public-facing port (any container with ports: exposing 80, 443, or any other host-bound port) is malformed and the project-deploy pipeline rejects it.
Project↔host interface is the triple (hostname, upstream_container, upstream_port). Nothing else crosses the cut. The project does not template a Caddyfile; it states its hostname binding declaratively (in the project's catalog entry, validated by gyrum-validate-playbook). The host does not read the project's source; it pulls images by tag and runs the project's compose file unmodified. The triple is the boundary; everything else stays on its side.

Service placement within a project

Not every service in a project's compose stack has the same placement constraints. Three classes:

Backend + frontend are atomic — must co-locate on the same host, always. They share the project's docker network and the project's caddy vhost, which only works on a single docker daemon. Splitting them would require either two caddy vhosts with the UI's API base hardcoded to a public URL, a public network hop with extra TLS and CORS overhead, or a service mesh — none of which justify the complexity at fleet scale. They also benefit from atomic deploy: a UI release that needs a corresponding backend release should ship as one transaction, not two phases across hosts. No flexibility here.
Database is the one service whose placement flexes, via a profiles: declaration in the project's compose plus a db: block in the project's deploy manifest. Three modes, all driven by the same compose file:
- mode: local (default) — enables the compose's local-db profile, postgres spins up alongside the project on the same host, DATABASE_URL points at postgres:5432 over the docker network. Self-contained; survives hcloud server delete only if the host has a Hetzner volume mounted at the data path.
- mode: shared — omits the local-db profile, DATABASE_URL points at the fleet's shared postgres host (dark-factory-db.gyrum.internal:5432) with a per-project database and role provisioned by the host pipeline. Cheaper on memory at fleet scale, single backup pipeline, but shares failure domain across every project that uses it.
- mode: external — escape hatch. DATABASE_URL comes from a vault var pointing wherever (managed Hetzner postgres, AWS RDS, a developer's local pg). The project doesn't care.
Off-the-shelf supporting services (caches, message queues if added later) follow the same profile-and-manifest pattern as the database. Keep one compose file per project; topology variation lives in the manifest.

The default is mode: local because at current fleet scale (single team, 10 projects, fast iteration with frequent host re-provisioning), blast-radius isolation outweighs the shared-DB cost savings (€20/month per project not paid). Shared mode is the right escape hatch when (a) the project is genuinely lightweight and per-project pg overhead isn't worth it, (b) the project explicitly needs to share data with another project (rare; usually a sign the projects should merge), or (c) cost matters more than blast radius for the specific deployment. The fleet's dark-factory-db host becomes a real shared-DB target only when a follow-up ADR defines its contract: per-project database + role provisioning, firewall rules limiting :5432 to known fleet hosts, backup pipeline, and noisy-neighbour mitigation. Until that ADR lands, mode: shared is documented but not operational.

The compose file pattern that supports all three modes:

services:
  <project>:
    image: ghcr.io/gyrum-labs/<project>:${<PROJECT>_TAG}
    environment:
      DATABASE_URL: ${DATABASE_URL}    # supplied by deploy manifest
  <project>-ui:
    image: ghcr.io/gyrum-labs/<project>-ui:${<PROJECT>_UI_TAG}
  postgres:
    image: postgres:15-alpine
    profiles: ["local-db"]              # only spins up when profile enabled
    environment:
      POSTGRES_USER: <project>
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: <project>
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

The deploy manifest (lives in gyrum-catalog per project) carries db: { mode: local|shared|external, ... }; project-deploy.yaml computes the right --profile flags + env vars from it. Same compose file, three deployment shapes, zero conditionals in the project repo.

Data persistence

Where postgres runs and how durably its bytes survive are independent. The deploy pipeline is idempotent over container images and never touches data; data lives in volumes that outlive containers. The manifest carries a sibling db.persistence: field choosing one of three shapes:

persistence: ephemeral (default for mode: local) — postgres data lives in a docker named volume on the host's root disk. Survives container recreation and docker compose down && up; lost on hcloud server delete along with the host. Right for dev/iteration projects that re-provision frequently and have nothing worth keeping between sessions.
persistence: durable — host has a Hetzner block volume attached at /srv/<project>-data (the existing data_volume_gb input on the provision pipeline), and postgres bind-mounts its data dir to it. Survives container recreation, host restart, and hcloud server delete; the next provision re-attaches the same volume by name via hcloud volume attach. Right for projects with users or other state worth preserving across host turnover.
persistence: off-host — implicit when db.mode: shared or db.mode: external. The project's host has no DB bytes at all; the data's durability is the responsibility of whatever runs the postgres instance.

Default-of-defaults: local + ephemeral for new projects (zero ceremony, nothing to clean up if you delete the host), graduating to local + durable the moment the project has users. Operators flip it in the manifest; nothing in the project repo changes.

Persistence is not the same as backups. Even durable mode loses you the data to a corrupted volume or an accidental DROP TABLE. A separate backup pipeline (pg_dump to Hetzner Object Storage on a daily cron, with retention) is deferred to its own ADR; this ADR only covers "does data survive normal deploy + provision lifecycle."

Consequences

What becomes easier:

Two projects on one box becomes a 5-minute job, not an hour. The host pipeline runs once per box; the project-deploy pipeline runs once per project per box. Adding distill alongside warp on warp-01 is just a second run of project-deploy with a different triple. Caddy gains a second vhost via an additive snippet in /etc/caddy/sites/distill.caddy; nothing about warp is touched.
Re-imaging a project never touches the host. A new warp release is a docker compose pull && up -d against unchanged host state. Caddy reload is a SIGHUP, not a config rewrite. No apt, no ufw, no observability re-registration.
Re-imaging the host never touches projects. Caddy version bumps, security stack updates, and observability stack changes are host-pipeline reruns; the project compose stacks above them are unaffected as long as the docker network names and the GHCR auth survive (both are part of the host contract, not the host implementation).
Tonight's four failures are structurally extinct. No project clones source on the host (rule 1: images come from GHCR). No project terminates its own TLS (rule 2: host owns edge). No project's compose file has phantom cloudflared services (rule 2 again). Adding a second project does not require host knowledge (interface is the triple).
Frontend release cadence is decoupled from backend release cadence by default. Two images means two version pins, two GHCR digests, two rollback paths. A UI hotfix doesn't require a backend release; a backend hotfix doesn't require a UI release. We pay the two-image overhead always so we never have to refactor when the cadences actually diverge.

What becomes harder:

Every project must publish a frontend image even if it has no UI. The "ship a placeholder nginx:alpine with a one-page index.html" requirement is real overhead for backend-only services (e.g. internal observability collectors). The alternative — exempting some projects from the rule — re-introduces the conditional shape that this ADR exists to kill. We accept the placeholder tax as the cost of the rule being unconditional.
The fleet now has two playbook surfaces to maintain instead of one. provision-host.yaml and deploy-project.yaml are separate, with a non-trivial composition (provision-and-deploy-<project> becomes a thin wrapper that calls both). ADR-093 and ADR-094 will define each half; until those land, the existing per-project playbooks remain in place but are deprecated.
Caddy becomes a fleet-wide single point of failure per host. A bad caddy config file or a Let's Encrypt rate-limit hit takes down every project on the host, not just one. We accept this — caddy is operationally simpler than per-project edge — and mitigate with: (a) gyrum-validate-playbook validates the caddy snippet before deploy, (b) caddy config reload is SIGHUP not restart, (c) host pipeline holds back caddy version bumps that change config syntax.
Backend and frontend version skew is a real failure mode now that they ship independently. A frontend that calls a backend endpoint the deployed backend doesn't yet implement will 404 in production. We accept this and mitigate with: (a) the contract suite (warp/contract/) gates the OpenAPI spec, (b) deploy-project.yaml verifies both images come from compatible builds before flipping caddy, (c) operators can pin both image tags explicitly when forward-compat matters.
Existing projects need migration. Warp's role today knows about clone, env, compose, and admin-token tasks; the host parts (docker.yml) need extracting and the project parts need to keep working through the migration. ADR-094 will spell out the staging.

What we have signed up to operate:

A host ansible role, owned by dark-factory, maintained as the canonical "what's on a freshly-provisioned box".
A project-deploy ansible role, owned by dark-factory, parameterized over (project_name, hostname, image_tags, env_vars_from_vault, compose_file_path).
A release.yml workflow template in gyrum-create-repo that publishes both images to GHCR on push to main and on version tags.
A caddy snippet template at dark-factory/ansible/templates/caddy/project-vhost.caddy.j2 that the project-deploy role renders for each project.

What we revisit:

If the per-project frontend-image overhead becomes painful enough that operators start exempting backend-only projects, this ADR comes back and we reconsider — probably by carving out a "backend-only project" subtype with its own contract. Not before then.
If the fleet ever grows to the point that one caddy per host is a real bottleneck (it won't, for any realistic gyrum fleet size), we revisit edge architecture.

Alternatives considered

Keep the conflated pipeline; just add credential distribution and TLS roles inside the existing warp role. Loses on the "second project to same box" test: every new project re-implements every host concern. Rejected — this is the path that produced tonight's four failures, and the fix would be re-doing them per project.
One image per project (option a in tonight's discussion) — embed the frontend in the backend via go:embed or equivalent. Lower complexity, single image, single GHCR pipeline. Lost because the rule has to be uniform across the fleet to prevent per-project drift, and embedding doesn't generalize to non-Go projects, doesn't support independent UI release cadence, and re-introduces the "what serves /?" ambiguity at the project level. The ergonomic win for warp specifically didn't justify a fleet-wide compromise.
Project ships its own caddy/cloudflared inside its compose. What the warp role half-implemented. Loses on the "two projects on one box" test: each project's TLS terminator wants port 443, they collide on the second project. Rejected — host owns edge is the only shape that scales past one project per box.
Per-project Hetzner box (one project, one VPS, forever). Avoids the multi-tenancy question entirely. Rejected on cost: gyrum has ~10 projects queued; that's ~€420/month minimum even on the smallest dedicated VPS, and most projects sit idle most of the time. Multi-tenancy via shared host is the right cost shape.
Kubernetes / nomad / fleet orchestrator. Over-engineered for current scale (single-digit boxes, single-digit projects per box). Reconsider when we have a real reason to need pod scheduling, secret rotation, or multi-region. The cut in this ADR is intentionally orchestrator-agnostic — provision-host.yaml could output a k8s node instead of a docker host without changing the project↔host interface.

Supersedes: none directly. Amends ADR-072 (Infrastructure as playbooks) by naming the host/project axis explicitly. Amends ADR-076 (Project-to-host deployment binding) by defining the interface triple. Tightens ADR-090 (AI as scaffold, pipelines as runtime) by giving the runtime a structural shape. Superseded by: {{leave blank until a later ADR reverses this one}}