Disposable local recovery rehearsal

The native rehearsal exercises HostSSH's real Postgres recovery adapter against two real, private PostgreSQL clusters. On 2026-09-05 it passed on PostgreSQL 18.4 (Homebrew). A test-only docker exec shim forwards the adapter's commands to installed PostgreSQL utilities through separate Unix socket directories. SQL, dump creation, archive inspection and transactional restore are real. Docker isolation and volume behavior are not established by this run.

From the repository root:

HOSTSSH_NATIVE_RECOVERY_DRILL=1 node --test scripts/recovery-native-drill.test.mjs

Prerequisites: installed Node, Go, tar and PostgreSQL 18 tools (initdb, pg_ctl, postgres, psql, pg_dump, pg_restore, createdb). The harness installs nothing. It starts new clusters with TCP disabled, private socket directories and a generated fixture password. It does not connect to existing databases. Cleanup stops only clusters under its temporary root; if stopping fails, the root is retained for inspection.

The fixture contains 100 rows with bytea, Unicode, NULL and numeric values, plus a sequence, constraints and a view. It hashes ordered row representations, sequence state, constraint definitions and the view definition before and after restore. It also checks the next generated ID. A conflicting restore and constraint violation must leave the checked destination unchanged; a corrupted archive must be rejected before mutation. A deliberately changed destination row must change the fingerprint. The source fingerprint must remain unchanged.

Three local files exercise binary, text and empty-file archive restoration and digest mismatch detection. This tests those fixture files, not Docker volume semantics, permissions, sparse files, links, large datasets or application-wide consistency. The database is quiescent during capture. Roles/ACLs, extensions, cross-major recovery and application/TLS/cutover are outside this fixture.

Docker rehearsal

A separate harness runs against a local Docker Unix socket with already-present postgres:18-alpine and alpine:latest images:

HOSTSSH_LOCAL_RECOVERY_DRILL=1 \
HOSTSSH_DRILL_DOCKER_HOST=unix:///absolute/path/to/docker.sock \
node --test scripts/recovery-local-drill.test.mjs

It records image IDs, uses --pull=never, gives fixtures unique ownership labels, disables container networking, and limits memory/CPU. Database storage uses temporary memory mounts; named test volumes hold only synthetic files. Cleanup checks the exact ownership label before removing resources. No existing images or volumes are pruned. A killed test process can prevent cleanup; inspect its uniquely labelled fixtures before removing anything.

The first Docker attempt failed with ENOSPC before the first database fixture started. A subsequent run passed all 7 tests on a second existing local Colima profile with available capacity. It used the preinstalled pgvector/pgvector:pg18 image (Postgres 18.6) for both the Postgres and file-utility roles. No images were downloaded or existing resources pruned. To select suitable already-present images, add:

HOSTSSH_DRILL_PG_IMAGE=pgvector/pgvector:pg18
HOSTSSH_DRILL_FILE_IMAGE=pgvector/pgvector:pg18

Pass these as environment assignments on the same invocation above. The file image must supply sh, tar, find, sha256sum, sort, mkdir, printf, and touch. References are inspected and pinned to image IDs before running. The completed rehearsal proved the synthetic database and volume fixtures plus a real Docker health observation; readiness remained false without application/TLS checks. This is not a cross-host or full fleet-script migration rehearsal.

Both scripts skip by default. Run them explicitly; a skipped test is not recovery evidence. Neither script deploys HostSSH, modifies production schema/authentication, or changes DNS.

Application and HTTPS integration

The Docker harness now also runs six scenarios against local HTTP/HTTPS endpoints whose health response compares the restored database with its expected fixture fingerprint. A correct restore returns 200; a changed destination row returns 503 and blocks the route phase. Other cases cover missing Docker service, wrong TLS hostname, untrusted certificate, and an unresponsive endpoint. The complete expanded run passed 13 tests on the same Postgres 18.6 image.

The HTTPS fixture uses OpenSSL to generate a temporary certificate for localhost. A temporary Go build overlay trusts only that certificate for the positive case; the production-trust binary must reject it. Neither the host certificate store nor production agent source is changed. Listeners bind only to loopback, and every health response checks real restored data. The helper is scripts/recovery-app-drill.mjs; it runs inside the existing Docker fixture and uses its already-restored database and source baseline. OpenSSL is an additional installed-tool prerequisite, not a dependency installed by the harness.

Passing the complete plan sets the journal's configured-check readiness to true, while its data-fingerprints-unverified label remains unchanged: the runner does not understand the fixture endpoint's SQL verification. Negative cases retain false readiness and block later phases. This demonstrates the synthetic test application, not an arbitrary customer application, a public certificate/DNS route, full fleet-script migration, or cross-host production recovery.