Inventory checks and ordered recovery

Fleet capture now records two source observations and refuses publication when container, volume, or watched configuration metadata differs. Fleet restore records phase transitions and can verify configured sidecars, an application endpoint, and destination HTTPS. These controls are local implementation, not a completed production recovery drill.

Capture evidence

system/inventory-before.json and system/inventory-after.json contain component names and SHA-256 summaries. Container configuration, image, mounts, restart count and running timestamps, Docker volume metadata, and watched configuration file bytes are compared. New inventory records do not contain raw environment values. Existing backup payloads and inspect files can contain secrets and must retain the backup's access controls.

Fleet watches the brain's source/.env, /etc/restic, /etc/wireguard, and /opt/corereflex-pgbouncer. Symlinks and special files in watched trees fail observation. system/capture-consistency.json records two-observation-inventory-only and names changes. HOSTSSH_CAPTURE_DEADLINE_SECONDS defaults to 1800 (maximum 86400). A failed comparison stops before catalog publication; retry explicitly after stabilizing the source.

Observations bracket the staged capture work before catalog publication. Live external volume paths are subsequently hashed and streamed; they are not filesystem snapshots. Pack verification checks received bytes against the catalog, while each database adapter records its own consistency scope. Neither an unchanged inventory nor matching hashes establish a transaction spanning multiple databases and files. Changes that occur and revert between observations can be missed. The 60-second observation context bounds Docker commands and is checked between file reads; it does not forcibly interrupt every filesystem read or the entire Bash capture operation.

Operator-authored verification plan

Supply a local JSON file using fleet-restore.sh --recovery-plan /path/recovery.json:

{
  "schema": 1,
  "timeoutSeconds": 1800,
  "nodes": [
    {"id":"prerequisites","kind":"phase","dependsOn":[]},
    {"id":"database","kind":"phase","dependsOn":["prerequisites"]},
    {"id":"volumes","kind":"phase","dependsOn":["database"]},
    {"id":"cache","kind":"sidecar","container":"recovery-cache","dependsOn":["volumes"]},
    {"id":"app","kind":"application","url":"http://app.example.com:8080/health","dialIP":"192.0.2.10","dependsOn":["cache"]},
    {"id":"route","kind":"route","url":"https://app.example.com/health","dialIP":"192.0.2.10","dependsOn":["app"]}
  ]
}

Replace the documentation addresses with the isolated destination. The plan is operator input; it is never executed as commands. All nodes are required. Every higher phase must depend, directly or transitively, on all lower phases. Unknown fields, cycles, missing prerequisites, unsupported kinds and invalid targets are rejected before restore mutations.

Sidecars must already be running with Docker health status healthy. The runner observes these services; it does not provision or start application/sidecar containers. Application and route probes require HTTP 2xx, refuse redirects, and connect to dialIP while preserving the URL Host header and TLS hostname verification. Route probes require HTTPS with system certificate trust. Credentials, query strings and fragments are prohibited in URLs. Use a dedicated health endpoint. Each probe has a five-second limit, bounded by the attempt deadline (1–7200 seconds).

Journal and failure handling

Each attempt retains /var/restore/attempt.XXXXXXXX/recovery-state.json. Core phases move pending → running → applied; probes move pending → running → passed. A failed phase blocks later work and records the first failed node. An existing journal cannot be reset, and replaying an applied phase or probe is refused. Interrupted running states require inspection and a new isolated attempt; this runner does not resume partially applied mutations automatically.

Without a plan, only prerequisites/database/volumes are tracked and ready remains false. With sidecar, application and HTTPS coverage, passing all checks sets ready: true with evidence configured-health-checks-passed; data-fingerprints-unverified. This is the configured probe result, not authorization to cut over or a guarantee of full recoverability. No DNS changes occur. The deadline is enforced at phase boundaries and during probes; existing Bash restore operations are not all forcibly terminated at that deadline.

Restore refuses an existing selected database container, selected named volumes, and captured MinIO/PgBouncer destination paths during preflight. Existing systemd units with identical readable contents are preserved (including their symlinks); differing contents are refused. New systemd copies are no-clobber and verified afterward. It no longer deletes the database container. This is an absence check, not an atomic reservation against another administrator. Use a fresh isolated target with exclusive administration. Failures retain the attempted destination and input artifact for inspection; there is no automatic destructive rollback. The optional brain rewrite and WireGuard activation remain explicitly destructive/replacement operations and are outside the default isolation guarantee.

Legacy global-role SQL reconciliation remains unchanged and separately unverified. Complete a live drill with database fingerprints, volume digests, service checks, and a reviewed rollback window before claiming application recovery or performing cutover.