Stateful services (never-auto-recreate)

A stateful app is a data-bearing service — Postgres, MariaDB, Redis, MinIO. For a normal app, "redeploy" means docker rm -f the old container and start a new one; for a database, that removal is the incident. The stateful class (agent v0.6.0, deploy.Spec.Stateful) makes replacing a data-bearing container an explicit operator decision instead of a redeploy side effect.

Semantics

• Never auto-recreate. If the app's container already exists (any state), a stateful deploy refuses with an instructive error. Nothing is stopped, nothing is removed.
• Explicit replace = --force-recreate. hostssh deploy -name pg -stateful -force-recreate … is the per-action consent (e.g. rolling a Postgres minor upgrade). Named volumes survive the replace; in-flight state does not. On the jobs pipeline the consent is spec.forceRecreate — it is never persisted on the deployment row, so a later routine redeploy can't inherit it.
• Requires a named volume. A "stateful" app with no persistent data is incoherent; the deploy fails fast (before the build) without at least one --volume name:/path.
• Never blue-green. The blue-green candidate would mount the SAME named volume as the live container — two databases writing one data dir is corruption. Stateful deploys always take the direct path, and only when allowed by the rules above.
• First-ever deploy needs no force — there is nothing to protect yet.

Control-plane defaults

Database templates (Postgres, Redis, MariaDB) and any template that declares a dataDir (MinIO) build stateful by default with a per-app named data volume (<app>-data:<dataDir>), so a one-click Postgres both keeps its data across recreates and is protected from redeploy-recreate. Persisted on the deployment row (stateful, warmup_cmd — migration 0031) and carried on every redeploy by deploymentJobSpec, so a control-plane-driven redeploy can never silently drop the protection. An explicit stateful: false on the spec opts out (advanced, at-your-own-risk).

pg_dump → R2 recipes

Volumes make data survive recreates; backups make it survive the box. The recipe below dumps a containerized Postgres and ships the dump to Cloudflare R2 (any S3-compatible store works the same).

One-off logical backup, no client tools on the host (uses the DB container's own pg_dump):

APP=orders-pg                      # the container name
DB=app USER=app                    # from the app's POSTGRES_* env
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
docker exec "$APP" pg_dump -U "$USER" -d "$DB" -Fc \
  | aws s3 cp - "s3://gm-db-backups/$APP/$APP-$STAMP.dump" \
      --endpoint-url "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"

-Fc (custom format) is compressed and restores selectively with pg_restore. Credentials come from the standard AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env (an R2 API token) — keep them in root-only files, never in the app's env.

Nightly, on the node (cron as root):

15 3 * * * APP=orders-pg; docker exec $APP pg_dump -U app -d app -Fc | aws s3 cp - s3://gm-db-backups/$APP/$APP-$(date -u +\%Y\%m\%dT\%H\%M\%SZ).dump --endpoint-url https://<ACCOUNT_ID>.r2.cloudflarestorage.com >> /var/log/hostssh-pgdump.log 2>&1

Restore (into a fresh stateful deploy of the same template):

aws s3 cp "s3://gm-db-backups/$APP/<dump>" - --endpoint-url "https://<ACCOUNT_ID>.r2.cloudflarestorage.com" \
  | docker exec -i "$APP" pg_restore -U app -d app --clean --if-exists

Notes:

• A logical dump is not a substitute for the volume: it's point-in-time. Pair it with the node-level hostssh capture (restic → R2) snapshot the install schedules nightly.
• Verify restores. A backup that has never been restored is a hope, not a backup — restore into a scratch stateful deploy (-name pg-drill) periodically.
• MariaDB: same shape with mariadb-dump; Redis: prefer volume snapshots (BGSAVE + copy dump.rdb); MinIO: mc mirror to a second bucket.

Warm-up gate (shared primitive)

deploy.Spec.WarmupCmd (-warmup, warmupCmd on the job spec) is documented with the GPU pre-warm use case in GPU support — it applies to any app that wants a post-readiness, pre-live in-container warm-up.