Recovery commands
Step-by-step use of hostssh recovery-pg, recovery-order, pack, and verify. What each command proves, and what it leaves unproved.
Recovery commands
These four commands are how you prove a backup before you trust it. They run on the Node. They do not cut DNS, and they do not mark an application healthy by themselves.
Source: agent/internal/cli/cli.go dispatches them. recovery-pg is agent/internal/cli/recovery_pg.go. recovery-order is agent/internal/cli/recovery_order.go. pack is agent/internal/cli/pack.go. verify is agent/internal/cli/cli.go (cmdVerify).
Recovery commands are not license-gated. capture, restore, pack, recovery-pg, recovery-order, and verify keep working when the license is expired, so you can always get data out. agent/internal/cli/cli.go gatedCommands is the list that does require a license.
Read this before you run anything
- A checksum is not a restore. A restore that exits 0 is not proof the app answers.
recovery-pg restoreprintsapplicationHealth: unverified. You still have to request the app yourself.recovery-order describesayscutover: never-automatic. Nothing in this page flips production traffic.- A mixed pack keeps the weakest consistency label that applies to any piece of it.
The four consistency labels, strongest first (recovery_order.go):
| Label | Meaning in plain language |
|---|---|
native-database-snapshot | The database engine froze a consistent snapshot. |
paused-filesystem | The filesystem was paused while the bytes were copied. |
immutable-image | The bytes came from an image that was not changing. |
best-effort-live-file | Files were copied from a live tree. A write during the copy can tear a file. |
The order to follow
Do this on a disposable copy first. Production data stays where it is until an operator has a passing receipt.
1. Say what "good" means
Write a recovery plan file before you capture. recovery-order validate checks that file. The plan is yours. The tool does not invent one.
hostssh recovery-order validate --plan /var/lib/hostssh/recovery-plan.json
Exit 0 and a JSON object means the plan loaded and validated (engine.ValidateRecoveryPlan). Exit 2 means the flags were wrong. Exit 1 means the plan was rejected. The error text is on stderr.
2. See which Postgres databases are on the box
hostssh recovery-pg describe
This prints the adapter's own description as JSON. It does not touch a database.
To look at one container:
hostssh recovery-pg discover --container <name> --select <database>
--user defaults to app. --database defaults to app. --host empty means a local socket. 127.0.0.1 is the other accepted host (recovery_pg.go).
Repeat --select once per database you intend to include. A discovery you did not select is not part of the capture.
3. Preflight the database
hostssh recovery-pg check --container <name> --database <name>
When you already have a dump file and want to know if a restore into that target is allowed:
hostssh recovery-pg check --container <name> --database <name> --file /path/to.dump
A passing preflight prints {"preflight":"passed"}. It does not restore.
If the destination image is already on the box:
hostssh recovery-pg check-image --image <image-ref> --file /path/to.dump
4. Capture the logical dump
hostssh recovery-pg capture --container <name> --database <name> --file /var/backups/app.dump
The command waits up to 30 minutes (context.WithTimeout in recovery_pg.go). Success prints a JSON receipt. Failure prints the error and exits 1.
5. Record what else was on disk
A database dump is one artifact. Named volumes and config files are another. pack write builds an integrity catalog for the trees you name.
hostssh pack write \
--out /var/backups/integrity.json \
--format 2 \
--root /var/lib/hostssh/stage=/stage \
--root-live /var/lib/docker/volumes/app/_data=/volumes/app
--out is required. --format is 1 (legacy) or 2 (durable recovery). The default is 1, so pass --format 2 when you want the current catalog.
--root abs=rel is a curated tree. A symlink in a curated tree fails the write. That is intentional.
--root-live abs=rel is a real system tree (a mounted Docker volume, for example). Sockets, fifos, and symlinks there are skipped instead of failing the catalog. Use --root-live only for trees that really are live system trees (pack.go).
hostssh pack capabilities
prints recovery-catalog-v2 and exits 0.
hostssh pack verify --catalog /var/backups/integrity.json --base /var/restore
--catalog and --base are both required. --base is the restore root that contains the paths named in the catalog. --allow-legacy accepts a schema 1 catalog without publication proof. Without that flag, a legacy catalog can still have its hashes checked, and the command prints LEGACY_PACK: hashes checked; publication unverified when the schema is the legacy one (cmdPackVerify).
6. Watch the live tree while you capture
recovery-order can take two inventories and refuse a capture whose tree changed underneath it.
hostssh recovery-order snapshot --out /tmp/before.json --watch /etc/hostssh --watch /var/lib/app
hostssh recovery-order snapshot --out /tmp/after.json --watch /etc/hostssh --watch /var/lib/app
hostssh recovery-order compare --before /tmp/before.json --after /tmp/after.json --max-age 1800 --out /tmp/compare.json
--max-age defaults to 1800 seconds. If the inventory changed, the command prints INVENTORY_CHANGED and exits 1. Do not publish that capture as stable.
--label repeats. The command reports the weakest label in the set:
hostssh recovery-order consistency --label native-database-snapshot --label best-effort-live-file
That example reports best-effort-live-file.
7. Restore into an empty destination
hostssh recovery-order isolate --container <dest> --volume-map /var/backups/volumes.json --destination-path /var/lib/app
unoccupied: true means the destination paths were free. If they were not, the command exits 1 and you stop.
hostssh recovery-pg restore --container <name> --database <name> --file /var/backups/app.dump
Add --clean only when you mean to replace objects already in that database. The JSON result is:
{"verification":"restore-exit-and-connection","applicationHealth":"unverified"}
Connection success is the database accepting a session. It is not the application.
8. Ask the application
verify checks one container: it is running, and if you pass --image the running image matches that ref.
hostssh verify <app> --image <image-ref>
You get pretty-printed JSON. Exit 0 means OK. Exit 1 means the container was not confirmed. Exit 2 means you forgot the app name (cmdVerify).
verify does not HTTP-probe your product. If the recovery plan names a URL, request that URL yourself and write down the status code. A restore drill that never requests the app has not finished.
recovery-pg verify is a different command. It only checks that the database connection still works and prints verification: connection-only.
9. Keep a journal if the restore has phases
hostssh recovery-order init --state /var/lib/hostssh/recovery.json --plan /var/lib/hostssh/recovery-plan.json
hostssh recovery-order phase --state /var/lib/hostssh/recovery.json --node database --action begin
hostssh recovery-order phase --state /var/lib/hostssh/recovery.json --node database --action complete
hostssh recovery-order checks --state /var/lib/hostssh/recovery.json
--action is begin, complete, or fail. describe lists the phase names the tool knows:
hostssh recovery-order describe
The phases are prerequisites, database, volumes, sidecar, application, route. Consistency mode is two-observation-inventory-only. Cutover is never-automatic.
Exit codes
| Code | Meaning |
|---|---|
| 0 | The command did what it claimed. |
| 1 | The check failed, or the tool hit an error. Read stderr. |
| 2 | Usage. A required flag or subcommand was missing. |
recovery-pg subcommands: describe, discover, check, check-image, inspect, capture, restore, verify.
inspect reads a dump file. --database-only is valid only with inspect, and then it prints the verified database name and nothing else.
recovery-order subcommands: describe, consistency, snapshot, compare, validate, init, phase, checks, isolate.
What to do when a step fails
| You see | What it means | What you do |
|---|---|---|
INVENTORY_CHANGED | A watched path changed between the two snapshots. | Capture again. Do not ship the first pair. |
CONSISTENCY: unknown label | --label is not one of the four names above. | Fix the label. Do not invent a fifth. |
CONSISTENCY: missing label | consistency was run with no --label. | Pass every label that applies to the pack. |
applicationHealth: unverified | The dump was restored and a connection worked. | Request the app. Record the answer. |
verify exit 1 | The container is missing, unhealthy, or on the wrong image. | Leave traffic where it is. |
unoccupied: false or isolate exit 1 | The destination already has data in a path you named. | Pick an empty destination, or stop. |
The full-server image commands (capture, restore, drill) are in Backups and recovery. This page is the Postgres and catalog layer those drills call.