ADR-089: Host security telemetry, response, and forensics lifecycle

Status: Accepted Date: 2026-04-25 Related: ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-087 (cloud-init bootstrap), ADR-088 (observability lifecycle)

Decision

Every gyrum-managed host runs a security telemetry stack — auditd for filesystem and syscall events, Falco for eBPF container events, AIDE for binary integrity, rkhunter for rootkit scan, fail2ban for auth brute-force — all routed to Loki via Promtail with source-labelled streams that Grafana turns into dashboards and Alertmanager turns into routed pages. High-severity events also forward to an external log relay the primary host can't tamper with, so forensics survive post-compromise cleanup. The provision pipeline owns bootstrap; the destroy pipeline owns snapshot-and-ship of security logs before delete.

Context

What harden already covers (ufw + fail2ban + unattended-upgrades) is attack-surface reduction. The gaps it leaves are not subtle: no detection of privilege escalation, no integrity monitoring of /usr/bin or /etc/passwd or pulled container images, no forensics after a compromise (logs live on the compromised host and walk out the door when the attacker rotates them), and no response automation beyond fail2ban's IP-ban hammer. Real security engineering asks four questions in sequence: who sees (detection signal), what's preserved (forensic record), what happens between compromise and human response (automated containment), and what does the attacker get even if they succeed (blast radius). ADR-087 closed the bootstrap-fragility class of bug; this ADR closes the no-detection class.

Threat model

The table is the spec. Every row has a detection source, a response policy, and a forensic record — three columns for three independent failure modes (we missed it; we saw it but didn't act; we acted but can't reconstruct what happened). A row with empty cells is a known unknown, not a hidden gap.

Signal taxonomy and routing

Three sinks, each with a different retention horizon and a different tamper-resistance profile:

Every security signal source streams to Loki AND mirrors to the external relay. The difference between the two streams is the compromise window: anything in Loki but not the relay was dropped post-compromise; anything in the relay but not Loki was generated during the compromise window after Promtail-to-Loki was severed. Diffing the two is one of the first forensic moves after an incident.

What cloud-init does vs what the pipeline does

Cloud-init installs binaries (auditd, falco-ce via the official Falco apt repo, aide, rkhunter, trivy, s3cmd — all stateless, version-pinned in the cloud-init template). The pipeline renders configs with host labels, Loki URLs, and external-relay endpoint and ships them in. Same split as ADR-088 observability and ADR-087 bootstrap: cloud-init = "make the box look like a baseline gyrum host", pipeline = "wire this gyrum host into the rest of the fleet".

The three new provision steps

1. security_bootstrap — render and deploy auditd rules (the big six watchpoints: /etc/passwd, /etc/shadow, /etc/sudoers, /root/.ssh/authorized_keys, /usr/bin/docker, /var/run/docker.sock), enable Falco systemd service, take initial AIDE baseline (aide --init && mv aide.db.new aide.db), enable rkhunter weekly cron. Idempotent: re-running on a host already bootstrapped diffs the rendered configs against on-host configs and only restarts services on actual change.
2. register_security_observability — Promtail configs for each source (/var/log/audit/audit.log, /var/log/falco/events.log, /var/log/aide/aide.log, /var/log/fail2ban.log), Prometheus scrape targets for the security exporters, Alertmanager routes for source=auditd severity>=high. Mirrors ADR-088's register_observability shape.
3. security_verify — self-test: from the ops user, attempt su root and confirm an audit event fires in /var/log/audit/audit.log within 5 seconds; attempt docker exec <container> /bin/bash and confirm Falco fires in /var/log/falco/events.log. If either probe fails to produce its expected event, fail the step. This is a structural check that security monitoring is actually working, not assumed-working — silent monitoring failure is the most expensive kind because nothing alerts you to the absence of alerts.

The new destroy step

snapshot_security_logs — at destroy time, BEFORE delete_server: tarball /var/log/audit/, /var/lib/aide/, /var/log/falco/, and /var/log/fail2ban.log; upload to Hetzner Object Storage bucket gyrum-forensics with object metadata destroyed-host=<name>, deleted_at=<ts>, retention=365d. Client-side encrypted with a key not stored on the host so Hetzner can't read it and a compromised host that exfils its own snapshot can't read what it sent. If the compromise surfaces three months later, forensics survive host deletion — without this step the most natural attacker move (provoke a destroy-and-replace) is also the most thorough cleanup of evidence.

Beyond Grafana — what else security signals reach

Loki and Grafana are the searchable plane; they are not the only sink. Four additional destinations earn their place in the routing table:

• Cloudflare Logs — Cloudflare Logpush ships inbound HTTP request and response metadata into Loki under source=cloudflare. L7 attacks (SQLi probes, oversized POSTs, abusive UA strings) are visible at the edge before they touch the host.
• Hetzner Cloud audit log — a cron pulls /api/v1/audit-logs every 15 minutes and ships the deltas to Loki under source=hcloud-audit. Catches "attacker rotated the API key" or "someone else created warp-02 outside the playbook" — control-plane compromises that never leave a trace on the data plane.
• External log relay (Phase 2) — write-only SFTP or S3-compatible endpoint in a separate Hetzner project with credentials the warp host doesn't know. For now we document the architectural slot; the implementation is its own follow-up.
• Alertmanager → Slack and email — severity routing: critical → Slack #ops-critical plus pager, high → Slack #ops-warnings, medium → daily email digest. Never only to Grafana — a dashboard nobody is looking at is not a control. A weekly security digest email from Grafana lands in the operator's inbox as an audit trail of "I looked at this".

Consequences and risks

Positive: real detection beyond "was the door locked"; forensics survive a hostile delete (the destroy snapshot is the kill chain's escape valve); the security_verify self-test ensures monitoring is not silently broken; the matrix in §Threat model is checkable — for any new threat we can ask which row it lands on, and a row that comes up empty is a gap on record.

Negative: +5–10% CPU on a cpx22 (auditd's syscall log is the hot path; cpx21 is borderline at deploy-time peaks) — this argues for ccx23 as the new default size, which sibling PR #645 in ai-research is already pushing toward. AIDE baseline must be re-computed after every legitimate apt upgrade or the next weekly diff floods the alert channel; we wire the unattended-upgrades post-hook to call aide --update for that. Cloud-init time grows ~30s for the additional package installs, on the wrong side of the no-noticeable-change line for a one-off provision.

Risks: Promtail config tampering by a compromised root would re-route or drop events before Loki sees them — partially mitigated by the external relay running its own collector; AIDE is itself a binary on disk and can be patched out by an attacker who knows what they're doing — partially mitigated by hashing the AIDE binary in the AIDE database; Falco signatures lag novel exploits — mitigated only by stock-rule freshness (falco-rules-update cron); the external relay is infra we don't have yet — Phase 2 work, the slot is reserved.

Phases

• Phase 1 — warp (this shipment): auditd, Falco, AIDE, rkhunter, fail2ban; Loki routing; provision and destroy pipeline integration; self-test.
• Phase 2 — external log relay: separate Hetzner project, write-only SFTP or S3-compatible endpoint, second Promtail destination on every host.
• Phase 3 — SIEM-shape normalization: re-encode events to OCSF format on the relay side so a downstream SIEM (or our own query layer) doesn't have to know about auditd's wire format vs Falco's JSON vs AIDE's plain text.
• Phase 4 — automated response: on a high-severity event, isolate the compromised host via Hetzner firewall rule API rather than just alert, with a 60-second human-cancel window before the rule lands.

Alternatives considered

• Stay at fail2ban + ufw + unattended-upgrades only. This is what harden already does, and it covers attack-surface reduction. It cannot detect privilege escalation, container escape, binary tampering, or credential theft, and it produces no forensic record once the host is destroyed. The asymmetry between "we have a firewall" and "we have no detection" is exactly the gap that gets a fleet to its first real incident with no forensics on hand.
• Wazuh as an all-in-one SIEM agent instead of auditd + Falco + AIDE separately. Wazuh bundles roughly the same capabilities under one daemon and one config tree. We rejected it because the bundled mode hides which tool produced which event (Wazuh re-emits in its own envelope), the agent is one extra moving part on every host, and the audit log we want is the auditd kernel-emitted record — not Wazuh's interpretation of it. The separate-tools path keeps each signal source first-class in Loki under its own source= label and makes Promtail the only mediator between host and observability plane.
• Host-only logging — no external relay. Simpler, fewer moving parts, and Loki retains 90 days. We rejected it because the threat model explicitly includes "attacker drops events before they ship to Loki", and a host-only sink is exactly the configuration that loses to that threat. The external relay is reserved as Phase 2 not because it is optional but because the architectural slot must be present from day one.
• Push-only auditd → syslog → external collector, skip Loki entirely for security signals. Tempting because it short-circuits the "Promtail tampering" risk by sending audit events out of the host before any local store. We rejected it as the primary path because it gives up cross-correlation with metrics and application logs in Grafana — the most common security investigation move is "what was warp doing at the moment auditd fired?", which needs both streams in the same query language. The external relay covers the tamper-resistance need without sacrificing Loki's queryability.

Supersedes: none Superseded by: (leave blank until a later ADR reverses this one)

Related

ADR-072 (infrastructure as playbooks), ADR-076 (project-to-host deployment binding), ADR-087 (cloud-init bootstrap), ADR-088 (observability lifecycle).