Platform — Observability

Monitoring & Observability

Find out about problems before the client does. Prometheus scrapes host, container, database and cache metrics; Grafana visualizes them; Alertmanager emails the admin the moment something looks wrong — and autoheal restarts containers that are hung, not just crashed.

Prometheus + Grafana + Alertmanager Self-hosted, no data leaves your server Opt-in per client Auto-healing containers

What gets monitored — host, containers, database, cache

Every component already runs a Docker healthcheck today — but a healthcheck only tells you "up" or "down". The monitoring stack adds real numbers on top: how much CPU/RAM/disk is left, how each container is behaving individually, and actual Postgres/Redis internals instead of a bare ping.

node-exporter: host-level CPU, RAM, disk and network — disk is the one that matters most, since it protects against the daily backup dump silently filling the volume
cAdvisor: per-container CPU/RAM/network, so you can see exactly which of the six containers is consuming resources, not just the host total
postgres_exporter / redis_exporter: real metrics — connections, slow queries, memory, eviction rate — not just "the process answered a ping"
Structured JSON logs (already in place via slog → stdout → Docker json-file) stay as-is for now — this stack doesn't add a separate log-aggregation layer yet
Everything scrapes on a 15-30 second interval into a local Prometheus TSDB — no data is sent to any third-party service

Proactive alerting, not reactive firefighting

Alertmanager routes alerts by email through the same SMTP relay the backend already uses for notifications — no new mail-sending code, no new secrets to manage beyond the existing Ansible Vault entries. Two concrete problems this closes today:

Disk-full-before-backup-fails

Daily pg_dump backup (02:00 UTC) has zero success/failure notification today — a silently broken cron would only be discovered when a restore is actually needed
DiskSpaceLow alert fires well before the volume actually fills — the admin gets time to clear old dumps or resize before the backup itself fails

Hung containers get restarted automatically

Docker's own restart: unless-stopped already recovers a container that crashes — that part already works
autoheal covers the gap: a container whose process is alive but whose healthcheck reports unhealthy N times in a row gets docker restart automatically
ContainerRestartLoop alert: if autoheal keeps restarting the same container too often, that's a signal of a real bug, not a transient blip — and the admin is told

Alert rules planned for the initial rollout: InstanceDown, PostgresDown/RedisDown, DiskSpaceLow, HighCPU/HighMemory, HealthCheckDegraded, NotificationQueueBacklog.

Security model — only Grafana faces the internet

Prometheus, Alertmanager and every exporter live exclusively inside the existing "axiom" Docker network — none of them publish a port to the host. The single deliberate exception is Grafana, reachable only through a dedicated nginx vhost.

Grafana — the one exposed component

Separate nginx server block, TLS via the same certbot mechanism already used for client sites
Basic auth at the nginx layer, in addition to Grafana's own login — two layers for negligible extra cost

Everything else — internal-only

Prometheus (9090), Alertmanager (9093), node-exporter, cAdvisor, postgres_exporter, redis_exporter: no ports: entry in docker-compose.monitoring.yml, reachable only from other containers on the axiom network
Credentials — Grafana admin password, SMTP access for Alertmanager — stored in Ansible Vault, the exact same pattern already used for every other client secret

docker-compose.monitoring.yml (excerpt)
services:
prometheus:
image: prom/prometheus:latest
networks: [axiom]
# no "ports:" — internal only
grafana:
image: grafana/grafana:latest
networks: [axiom]
# exposed only via nginx vhost + TLS + basic auth
Ansible var (per client)
monitoring_enabled: true
grafana_admin_password: "{{ vault_grafana_admin_password }}"

How it's rolled out — opt-in, Docker-based, no Kubernetes

Every Axiom client runs on its own single VM managed by Ansible, not a Kubernetes cluster — introducing a full orchestration platform just to get self-healing/observability "for free" would be disproportionate at this stage. The monitoring stack is deliberately opt-in per client rather than switched on everywhere by default.

Toggled with a single Ansible variable, monitoring_enabled (false by default) — the same pattern as existing resource-tuning variables like docker_backend_cpus
Ships as its own docker-compose.monitoring.yml alongside the main stack, with make monitoring-up / monitoring-down / monitoring-status / monitoring-logs for local development
Resource cost is transparent up front: roughly +1.1 vCPU and +1.2-1.5 GB RAM total, cAdvisor being the heaviest single piece — it can be dropped on smaller tiers while keeping node-exporter and the DB/cache exporters
Current status: a working local prototype (the compose file and Ansible role skeleton exist and run), not yet deployed to a live client — this is a newer capability, not a battle-tested one yet

Grafana/Prometheus dashboards are an ops tool for the team running the platform — they are not part of the client-facing web/office product UI.

Resource cost, per component

Prometheus: ~0.3 vCPU, 256-512 MB RAM (15-day retention)
Grafana: ~0.2 vCPU, 150 MB RAM
cAdvisor (heaviest, optional on smaller tiers): ~0.3 vCPU, 150-200 MB RAM
Alertmanager, node-exporter, both DB exporters, autoheal: ~0.3 vCPU, ~120 MB RAM combined