Recovery commands

Step-by-step use of hostssh recovery-pg, recovery-order, pack, and verify. What each command proves, and what it leaves unproved.

Recovery commands

These four commands are how you prove a backup before you trust it. They run on the Node. They do not cut DNS, and they do not mark an application healthy by themselves.

Source: agent/internal/cli/cli.go dispatches them. recovery-pg is agent/internal/cli/recovery_pg.go. recovery-order is agent/internal/cli/recovery_order.go. pack is agent/internal/cli/pack.go. verify is agent/internal/cli/cli.go (cmdVerify).

Recovery commands are not license-gated. capture, restore, pack, recovery-pg, recovery-order, and verify keep working when the license is expired, so you can always get data out. agent/internal/cli/cli.go gatedCommands is the list that does require a license.


Read this before you run anything

  1. A checksum is not a restore. A restore that exits 0 is not proof the app answers.
  2. recovery-pg restore prints applicationHealth: unverified. You still have to request the app yourself.
  3. recovery-order describe says cutover: never-automatic. Nothing in this page flips production traffic.
  4. A mixed pack keeps the weakest consistency label that applies to any piece of it.

The four consistency labels, strongest first (recovery_order.go):

LabelMeaning in plain language
native-database-snapshotThe database engine froze a consistent snapshot.
paused-filesystemThe filesystem was paused while the bytes were copied.
immutable-imageThe bytes came from an image that was not changing.
best-effort-live-fileFiles were copied from a live tree. A write during the copy can tear a file.

The order to follow

Do this on a disposable copy first. Production data stays where it is until an operator has a passing receipt.

1. Say what "good" means

Write a recovery plan file before you capture. recovery-order validate checks that file. The plan is yours. The tool does not invent one.

hostssh recovery-order validate --plan /var/lib/hostssh/recovery-plan.json

Exit 0 and a JSON object means the plan loaded and validated (engine.ValidateRecoveryPlan). Exit 2 means the flags were wrong. Exit 1 means the plan was rejected. The error text is on stderr.

2. See which Postgres databases are on the box

hostssh recovery-pg describe

This prints the adapter's own description as JSON. It does not touch a database.

To look at one container:

hostssh recovery-pg discover --container <name> --select <database>

--user defaults to app. --database defaults to app. --host empty means a local socket. 127.0.0.1 is the other accepted host (recovery_pg.go).

Repeat --select once per database you intend to include. A discovery you did not select is not part of the capture.

3. Preflight the database

hostssh recovery-pg check --container <name> --database <name>

When you already have a dump file and want to know if a restore into that target is allowed:

hostssh recovery-pg check --container <name> --database <name> --file /path/to.dump

A passing preflight prints {"preflight":"passed"}. It does not restore.

If the destination image is already on the box:

hostssh recovery-pg check-image --image <image-ref> --file /path/to.dump

4. Capture the logical dump

hostssh recovery-pg capture --container <name> --database <name> --file /var/backups/app.dump

The command waits up to 30 minutes (context.WithTimeout in recovery_pg.go). Success prints a JSON receipt. Failure prints the error and exits 1.

5. Record what else was on disk

A database dump is one artifact. Named volumes and config files are another. pack write builds an integrity catalog for the trees you name.

hostssh pack write \
  --out /var/backups/integrity.json \
  --format 2 \
  --root /var/lib/hostssh/stage=/stage \
  --root-live /var/lib/docker/volumes/app/_data=/volumes/app

--out is required. --format is 1 (legacy) or 2 (durable recovery). The default is 1, so pass --format 2 when you want the current catalog.

--root abs=rel is a curated tree. A symlink in a curated tree fails the write. That is intentional.

--root-live abs=rel is a real system tree (a mounted Docker volume, for example). Sockets, fifos, and symlinks there are skipped instead of failing the catalog. Use --root-live only for trees that really are live system trees (pack.go).

hostssh pack capabilities

prints recovery-catalog-v2 and exits 0.

hostssh pack verify --catalog /var/backups/integrity.json --base /var/restore

--catalog and --base are both required. --base is the restore root that contains the paths named in the catalog. --allow-legacy accepts a schema 1 catalog without publication proof. Without that flag, a legacy catalog can still have its hashes checked, and the command prints LEGACY_PACK: hashes checked; publication unverified when the schema is the legacy one (cmdPackVerify).

6. Watch the live tree while you capture

recovery-order can take two inventories and refuse a capture whose tree changed underneath it.

hostssh recovery-order snapshot --out /tmp/before.json --watch /etc/hostssh --watch /var/lib/app
hostssh recovery-order snapshot --out /tmp/after.json --watch /etc/hostssh --watch /var/lib/app
hostssh recovery-order compare --before /tmp/before.json --after /tmp/after.json --max-age 1800 --out /tmp/compare.json

--max-age defaults to 1800 seconds. If the inventory changed, the command prints INVENTORY_CHANGED and exits 1. Do not publish that capture as stable.

--label repeats. The command reports the weakest label in the set:

hostssh recovery-order consistency --label native-database-snapshot --label best-effort-live-file

That example reports best-effort-live-file.

7. Restore into an empty destination

hostssh recovery-order isolate --container <dest> --volume-map /var/backups/volumes.json --destination-path /var/lib/app

unoccupied: true means the destination paths were free. If they were not, the command exits 1 and you stop.

hostssh recovery-pg restore --container <name> --database <name> --file /var/backups/app.dump

Add --clean only when you mean to replace objects already in that database. The JSON result is:

{"verification":"restore-exit-and-connection","applicationHealth":"unverified"}

Connection success is the database accepting a session. It is not the application.

8. Ask the application

verify checks one container: it is running, and if you pass --image the running image matches that ref.

hostssh verify <app> --image <image-ref>

You get pretty-printed JSON. Exit 0 means OK. Exit 1 means the container was not confirmed. Exit 2 means you forgot the app name (cmdVerify).

verify does not HTTP-probe your product. If the recovery plan names a URL, request that URL yourself and write down the status code. A restore drill that never requests the app has not finished.

recovery-pg verify is a different command. It only checks that the database connection still works and prints verification: connection-only.

9. Keep a journal if the restore has phases

hostssh recovery-order init --state /var/lib/hostssh/recovery.json --plan /var/lib/hostssh/recovery-plan.json
hostssh recovery-order phase --state /var/lib/hostssh/recovery.json --node database --action begin
hostssh recovery-order phase --state /var/lib/hostssh/recovery.json --node database --action complete
hostssh recovery-order checks --state /var/lib/hostssh/recovery.json

--action is begin, complete, or fail. describe lists the phase names the tool knows:

hostssh recovery-order describe

The phases are prerequisites, database, volumes, sidecar, application, route. Consistency mode is two-observation-inventory-only. Cutover is never-automatic.


Exit codes

CodeMeaning
0The command did what it claimed.
1The check failed, or the tool hit an error. Read stderr.
2Usage. A required flag or subcommand was missing.

recovery-pg subcommands: describe, discover, check, check-image, inspect, capture, restore, verify.

inspect reads a dump file. --database-only is valid only with inspect, and then it prints the verified database name and nothing else.

recovery-order subcommands: describe, consistency, snapshot, compare, validate, init, phase, checks, isolate.


What to do when a step fails

You seeWhat it meansWhat you do
INVENTORY_CHANGEDA watched path changed between the two snapshots.Capture again. Do not ship the first pair.
CONSISTENCY: unknown label--label is not one of the four names above.Fix the label. Do not invent a fifth.
CONSISTENCY: missing labelconsistency was run with no --label.Pass every label that applies to the pack.
applicationHealth: unverifiedThe dump was restored and a connection worked.Request the app. Record the answer.
verify exit 1The container is missing, unhealthy, or on the wrong image.Leave traffic where it is.
unoccupied: false or isolate exit 1The destination already has data in a path you named.Pick an empty destination, or stop.

The full-server image commands (capture, restore, drill) are in Backups and recovery. This page is the Postgres and catalog layer those drills call.