Observability
The single pane for fleet health — heartbeat telemetry, real uptime %, backup-success, alerts, and per-server restore-drill posture.
Observability
The Observability page is the one place to answer "is the fleet healthy, and what needs a human right now?" It pulls together heartbeat telemetry, real historical uptime, backup posture, and the alerts raised from live fleet state.
Access needs the observability.view privilege; without it the page shows a locked
state.
Telemetry cards
Three cards sit at the top, each showing the latest value and a 24-hour sparkline:
| Card | What it measures |
|---|---|
| Fleet uptime | Percentage of expected agent heartbeats that actually landed, over the last 24h. |
| Backup success | Rolling backup-success rate across the fleet. |
| Storage | Total backup-repository size across all Nodes. |
If no agents have reported yet, the page shows an honest "No telemetry yet" empty state instead of fabricated numbers — the cards only appear once agents start reporting.
How uptime is really computed
Uptime here is not a marketing figure. Every accepted heartbeat (~one per minute per agent) appends a sample. Uptime over a window is samples that landed ÷ samples expected at the heartbeat cadence — but the expected count is capped at the Node's actual age, so a Node registered only two hours ago reads near 100%, not a fraction of a full day. That's why a brand-new Node isn't unfairly penalized.
Alerts
Alerts are derived live from current fleet state every time the page loads — there's no separate alerting job you have to wire up for the basics. A Node raises an alert when:
| Condition | Severity |
|---|---|
| Node is offline (no recent heartbeat) | critical |
| Node heartbeat is stale (degraded) | warning |
| Backups are critical | critical |
| Backup health is degraded | warning |
| Disk pressure at/over 85% | critical |
| Disk pressure at/over 65% | warning |
The alerts table lists Severity (with a pulsing dot for unacknowledged criticals), the alert title, its Source (the Node or monitor name), when it opened, and whether it's open or acknowledged. Alerts raised by DNS monitors surface here too. The card header shows the open / total counts.
Thresholds today. Disk pressure warns at 65% and goes critical at 85%; a Node is degraded after 3 minutes without a heartbeat and offline after 15. These are the built-in thresholds — the alert set reflects reality on every render rather than a hand-tuned rules engine (a persisted ack/resolve lifecycle is on the roadmap).
Per-server posture
The bottom table is one row per Node — the honest recoverability scorecard:
| Column | What it shows |
|---|---|
| Server | Node name, provider · region. |
| Status | online / degraded / offline / provisioning / migrating. |
| Resources | CPU · MEM · DSK, each colored (muted, warn at 75%, danger at 90%). |
| Backup health | healthy / warning / critical, with a pulsing dot when critical. |
| Uptime (24h) | Real 24h uptime %, with the 7-day figure beneath it. |
| Last drill | When recoverability was last verified by a restore-drill. |
The header shows a running healthy / total count. Where a Node has no heartbeat history yet, the uptime cell falls back to current reachability and says so on hover rather than inventing a percentage.
Why the restore-drill column matters. "Backup succeeded" and "backup restores" are different claims. The Last drill column is the only one that proves recoverability — a Node whose drill is blank or stale is a Node you haven't confirmed you can bring back. Set up a monthly drill in Getting started.
Related
- Fleet & Nodes — the per-Node detail behind these numbers, plus disk cleanup.
- DNS Tools — the DNS/email monitors that also raise alerts here.
- Fleet Copilot — ask the AI to triage these signals and propose fixes.
- Backups & recovery — restore-drills in depth.