HHostSSH

Changelog

What shipped recently and why.

Developer changelog

A running record of notable changes to the HostSSH control plane, newest first. Each entry says what shipped, why, and which files to read. For step-by-step operator runbooks (applying schema, setting roles, deploying), this doc cross-links the AI-operator references under .claude/skills/hostssh/references/ rather than repeating them.

Background context that the entries below assume: the control plane is a Next.js 16 output: 'standalone' app deployed on Coolify (app box 31.220.104.207); Postgres (pgvector, PG18) lives on a separate box (31.220.104.211, container t6cr90jl6qq19vuz7v3qv1fy) reached over a WireGuard/socat bridge at 10.10.0.2:5432; the app connects as the DML-only role hostssh, while schema changes run as hostssh_migrator; deploys happen by git push (Coolify auto-deploy). See ../ARCHITECTURE.md and the skill's database-migrations-roles and control-plane-api references.

2026-07-11 → 2026-07-12 — GPU nodes, launch hardening, first GPU node live

GPU deploy support (e3c6a86) — hostssh deploy -gpu all|<N>|<ids> reserves GPUs via docker --gpus, persisted on the app + job spec; a doctor gpu check (nvidia-smi + container passthrough probe); GPU heartbeat stats; -image alias + trailing -- ARGV. Proven end-to-end on a real Vast.ai RTX 3090 on 2026-07-12 — the 5th fleet node. See GPU support, GPU workloads, and the GPU node runbook. Companion: managed Traefik long-response timeout (≥620s floor, 875a2f3) for long inference/render jobs.

Node telemetry (952404b + 19bb0f8) — a node is now legible in the dashboard without SSHing in. The agent's new internal/inventory package puts the container inventory + software stack (OS/Docker/ NVIDIA driver/CUDA/toolkit) in every heartbeat alongside GPU stats; the control plane persists it (migration 0030, agents.telemetry JSONB, COALESCE-preserved) and the Hardware page renders a collapsible per-node panel. See agent protocol → node-legibility telemetry.

M4 multi-container migrate (bc7a13a, agent v0.5.0) — migrate.RunMulti moves a multi-container app (web+worker+redis) together onto the shared network so they stay name-reachable; ordered, fail-closed, dry-run-safe. hostssh migrate --source-ssh host --containers redis,api,web. See migration engine.

Launch-readiness hardening — a full 5-dimension audit (0 launch blockers) then: header-only cron secrets + setDensity authz (829acb9), the add-admin invite flow on the Identity page (36d0eef, super-admin gated, show-once temp password — lib/auth/admin-actions.ts), and license-scoped webhook + email-key operator APIs closing two cross-tenant IDORs (dfafae4, migration 0029 adds license_id).

2026-06-27 → 2026-07-03 — the Gate A / Coolify-exit build wave

The largest stretch of the project so far (~180 commits). Newest-first summary of what shipped; each item cross-links the deeper doc. The through-line: turn every manual migration step into a product feature (the "we are our own first customer" doctrine of ../../planning/MIGRATION-MASTER-PLAN.md).

Security hardening baseline (G15–G19) — 2026-07-03

Container hardening on by default (no-new-privileges, cap-drop); a dual-engine host firewall (nftables or ufw) with per-source SSH rate limiting; auth brute-force protection on sign-in + TOTP; panel-configurable security settings; and boot-reapply so hardening survives a rebuild. See hardening.md, build-guards.md, and ../user/hardening.md.

License enforcement + backup resilience (G14) — 2026-07-02

A Node won't function without a license (the agent daemon and proxy are gated; recovery never is). Nightly hostssh capture is auto-scheduled by install.sh; scripts/backup-secrets.sh age-encrypts platform keys off-box. See Security model.

WordPress migration v0.3.0 (G13) — 2026-07-02

The hostssh-migrate plugin hardened after a review found 14 defects (2 critical): regex/two-phase token rewrite, restore verification + success page, drift guard, license gate. See ../user/wordpress-migration.md.

Gate A P0 burn-down — 2026-07-01

A cluster of deploy-correctness and recovery features: sealed secret-env delivery end to end; editable App env post-deploy; deploy into a Slot (caps + dockerfile builder + placement↔deploy bridge); node targeting for unslotted deploys; persistent volumes (data, not just names) for stateful apps; webhook deploys clone by SHA; fleet alerts persisted + delivered with an ack lifecycle; auto app-health monitor per routed domain; import apps from Coolify (reviewable batch-create); Cloudflare token held sealed server-side so cutover is a dashboard action; the mTLS client cert presented for hostssh ssh. See Deploy pipeline, Security model, and ../user/.

DNS-01 production TLS (G12) — 2026-07-01

The managed proxy issues certs via ACME DNS-01 (Cloudflare) as well as HTTP-01 — certs issue before cutover (zero-downtime), support wildcards, and work behind orange-cloud. See ../user/domains-and-tls.md.

Local-dev → live-node deploy surfaces — 2026-07-01

Three surfaces on one primitive: git-push (Dokku-style), hostssh push CLI (Fly-style), and a GitHub webhook receiver (Vercel-style). See ../../deploy/README.md.

Attack-surface self-scan — 2026-06-30/07-01

Owner-gated (DNS-TXT) active recon: common-port sweep, exposed-file probe, CT-log subdomains, tech fingerprint — IP-pinned and SSRF-guarded after an adversarial audit. See DNS tools.

Deploy honesty + fleet honesty (G7/G8/G9) — 2026-06-30

Build cache persisted across deploys (G7); rollback on any failed gate so a bad build never takes an app down (G8); agentless nodes read as "—" not a misleading 0% (G9). See Build guards.

Public expose (cloudflared) — 2026-06-29

expose/unexpose job kinds provision a Cloudflare tunnel + *.apps.hostssh.com hostname behind a new access.expose RBAC permission.

Access, cleanup, budgets — 2026-06-27→29

Web-SSH session recording (hash-chained) + active-session tracking + a native reverse stream proxy; disk-pressure cleanup jobs with reclaimed-GB reporting and archive-first safety rails; per-app image retention, build-cache GC, and image-size budgets; local hostssh doctor prerequisite checks; HostPack + BuildKit installer provisioning.

2026-06-26

A single session that hardened the operational spine of the control plane — audit trail, a real migration mechanism, the DB privilege model, and request validation — plus two product-shaped additions (the operations skill and the Slots model).

Append-only audit trail for sensitive operator actions

What. A real audit_events table and a dual-backend access layer now record every privileged operator action: launch-mode changes, impersonation start/stop, simulation set, license issue/revoke, SSH-key issue/revoke, and web-SSH session opens.

Why. God-mode mutations (impersonate a tenant, flip launch mode, mint a license) previously left no durable trail. These are exactly the actions you need to be able to answer "who did what, on whose behalf, from where" about.

How it works. Each 'use server' mutation calls a convenience helper audit(session, action, target, outcome) that derives the actor from session.admin.email, the onBehalfOf label from an active impersonation, the launch mode from the session, and the client IP from x-forwarded-for / x-real-ip. It is best-effort and never throws into the caller — an audit-write failure must not fail the operation it records. The read path (listAuditEvents) tolerates the table not yet existing (42P01 → empty list) so a deploy that lands before the schema is applied degrades to an empty Audit page instead of crashing.

Files.

The baseline audit_events schema was applied and verified live in prod this session.

Boot-time, forward-only migration runner

What. A versioned migration runner that applies pending schema changes automatically on server boot, eliminating the manual "apply the SQL by hand" step that the audit_events schema had needed.

Why. The control plane builds as a Next.js standalone image whose build context is control-plane/web, so control-plane/db/*.sql (the go-live baseline) is not in the image. Post-baseline schema changes therefore can't ship as loose .sql files. Migrations instead live as embedded TypeScript modules in lib/migrations/ so Next traces and bundles them; the .sql baseline stays the one-time provisioning step (scripts/apply-schemas.sh).

How it works. instrumentation.ts's register() fires the runner whenever DATABASE_URL is set. The runner takes a session-level pg_advisory_lock, creates a schema_migrations ledger if absent, plans pending vs. drifted migrations, and applies each pending one. GET /healthz reads the ledger and reports applied/pending/drift honestly.

The runner was hardened through an adversarial review; the resulting safety properties are worth enumerating because each one closes a specific failure mode:

  1. Detached from register(). Next awaits register() before serving any request (including /healthz), so awaiting a slow migration there would block readiness and risk a probe-timeout crash-loop. The run is fired as void (async () => …)(); the server comes up immediately and /healthz reports pending until it converges.
  2. Never throws on failure. This app also serves marketing + sign-in, so a migration (or config) failure must not crash-loop the whole control plane. Failures log [migrate][FATAL] and surface on /healthz instead.
  3. Dedicated single-use connection end()'d in finally. All of BEGIN/DDL/COMMIT and the advisory lock acquire+release are pinned to one max:1 session that is torn down even if the unlock query itself fails — no leaked lock.
  4. lock_timeout = 30s bounds the lock wait. A stuck peer surfaces as a logged error instead of an unbounded hang.
  5. Ledger read happens under the lock. A peer replica may have applied while this one waited, so the recorded set is re-read after acquiring the lock.
  6. Per-migration transaction. Each migration's DDL and its schema_migrations row commit (or roll back) together, with a transactional:false opt-out for DDL that can't run in a transaction (e.g. CREATE INDEX CONCURRENTLY).
  7. INSERT … ON CONFLICT (id) DO NOTHING on the ledger row — defense-in-depth against a race.
  8. Whitespace-stable checksums. checksum() collapses all whitespace before SHA-256, so reindentation or an LF↔CRLF checkout of an already-applied migration does not register as drift; .gitattributes pins eol=lf as a second line of defense. A genuinely changed sql is reported as drift and skipped — never silently re-run (forward-only).
  9. validateMigrations() runs at module load from list.ts: it rejects out-of-order or duplicate ids and migrations that manage their own transaction (BEGIN/COMMIT/…), failing the build/boot loudly rather than corrupting the ledger at runtime.

Files. lib/migrations/core.ts (pure logic — types, checksum, planMigrations, validateMigrations; unit-tested in core.test.ts), runner.ts, list.ts, the first migration 0001-audit-events-indexes.ts (adds composite (actor, at DESC) and (action, at DESC) indexes for the Audit page's two filters), instrumentation.ts, and app/healthz/route.ts.

For the apply/converge runbook and the /healthz check matrix, see database-migrations-roles.

MIGRATIONS_DATABASE_URL and the least-privilege role model

What. The runner now prefers a dedicated MIGRATIONS_DATABASE_URL (falling back to DATABASE_URL when unset), and a new idempotent script codifies a two-role split.

Why. DDL requires ownership of the target table. We do not want the app's internet-facing runtime credential to be able to rewrite (or drop) the schema. The split ("Model B"):

  • hostssh_migrator owns the public schema and every object; the only role that runs DDL; used solely by the runner via MIGRATIONS_DATABASE_URL.
  • hostssh is the app runtime role: SELECT/INSERT/UPDATE/DELETE only, with CREATE ON SCHEMA public revoked. A leak of the app credential can read/write rows but cannot alter or drop structure.

How it works. scripts/setup-db-roles.sql (run once as the superuser gmadmin, safe to re-run): creates both roles without ever resetting an existing password, reassigns ownership of every public table/sequence/view not already migrator-owned (sweeping up the gmadmin-created baseline tables and the schema_migrations table the app role made on first boot), grants the app role DML on all existing objects, and uses ALTER DEFAULT PRIVILEGES so future migration-created tables are auto-DML-accessible — no per-migration GRANT bookkeeping. A final SELECT verifies nothing in public is owned by a non-migrator role. The runner was also taught to name the exact failure: a 42501 ("must be owner of table …") on a migration now annotates the error with a pointer to MIGRATIONS_DATABASE_URL and this script.

Files. scripts/setup-db-roles.sql, lib/migrations/runner.ts (URL precedence + the self-explaining 42501/42P01 hints).

Zod request-body validation on all body-taking routes

What. A single parseBody(req, schema) helper plus zod schemas now validate every API route that accepts a JSON body — all six: license/activate, telemetry/heartbeat, jobs/claim, jobs/[id]/status, jobs/[id]/logs, and chat.

Why. The agent-facing API previously hand-parsed bodies. Centralized schema validation rejects malformed/oversized payloads early (e.g. caps the logs array so a pathological body can't make zod walk millions of entries) and returns concise, field-scoped 400s without leaking internals.

How it works. The schemas are tolerant receivers that mirror the Go agent's wire structs field-by-field, so a legitimate agent request is never wrongly 400'd:

  • Unknown keys (the agent's ts, storage) are stripped — a newer agent adding a field doesn't break an older control plane.
  • Go omitempty fields map to .optional() (never .nullable()), because omitempty omits rather than nulls.
  • Only what a handler genuinely needs is required: fingerprint everywhere it fences/signs, and state (a jobStates enum) on job status.
  • parseBody never throws — it returns {ok:true,data} or {ok:false,res} (a 400 Response).

The jobStates and chatCategories consts are exported from the same file and must stay in sync with the Go jobs.JobState and agent wire structs, or a legit agent will 400.

Files. lib/api/validate.ts, lib/api/schemas.ts, and the six wired routes under app/api/. See the route + schema inventory in control-plane-api.

HostSSH operations skill

What. An AI-operator skill at .claude/skills/hostssh/ — a navigator (SKILL.md) plus five terse runbook references: deploy-and-cd, database-migrations-roles, agent, control-plane-api, and operations-and-process.

Why. Operating the platform (deploys, migrations, the DB, the agent, the API, incidents) involves fleet-specific facts (box IPs, the canonical PG container, the CUJO-filtered Coolify REST API, the role split) that are easy to get wrong. The skill captures them as step-by-step operator runbooks. These docs (under docs/) are the human-readable explanations; the skill is the operator's checklist — they cross-link rather than duplicate.

Slots capacity/mobility model

What. The Slots product model: a typed capacity/placement model, a racks + placements schema (migration 0002), a dual-backend store, and a read-only Slot Board at /slots.

Why. Slots are the unit that makes hosting portable — one Slot is a movable unit of hosting capacity; a Rack is a machine with a configured Slot count; a Placement is a workload occupying a Slot. It's the capacity, billing, and mobility unit in one. The canonical model lives in docs/product/SLOTS.md.

How it works. lib/slots/types.ts defines Rack, Placement, and SlotBoard, plus recommendedSlotCapacity() — capacity derived from a machine's resources and Slot class (burstable = dense/oversubscribed for idle WordPress; reserved = guaranteed 1 vCPU / 2 GB / 25 GB envelopes for production). lib/slots/store.ts is dual-backend: live Postgres when DATABASE_URL is set, otherwise a seeded in-memory backend so the board is real in dev/demo. The board is assembled from racks + placements — empty Slots are derived (capacity − used), not stored as rows. Migration 0002-slots.ts creates racks and placements (these are net-new tables, so the app role can create them once the runner reaches them). The page lives at app/(god)/slots/page.tsx.

This is the prototype slice. Next slices (placement scheduler, WordPress-into-a-Slot via the Migrate plugin, "move Slot" wired to the agent mover, per-Slot metering, the drag-to-move board) are listed in docs/product/SLOTS.md.

Open items

  • Apply the DB role model in prod so migrations 0001/0002 go live. The runner, the role model, and both migrations are merged, but the role split has not yet been applied to the live database, so 0001 and 0002 remain pending in prod (visible on /healthz, no outage). To converge: run scripts/setup-db-roles.sql as gmadmin, set the hostssh_migrator password out-of-band, set MIGRATIONS_DATABASE_URL on the Coolify app, and redeploy — see the "must be owner" fix in database-migrations-roles.