HHostSSH

The deploy pipeline

How source becomes a running, routed container inside the agent — orchestrator, builders, gates, rollback, and the env/secret flow.

The deploy pipeline

This is the internals of hostssh deploy — how the agent turns a source into a running, HTTPS-served container. The user-facing version is docs/user/deploying-apps.md.

Everything lives under agent/internal/:

deploy/     the orchestrator: Deploy() build→run→route→gate→rollback
builder/    dockerfile.go, hostpack.go, image (how source → image)
runtime/    the docker CLI wrapper: Run/RunOnce/State/ContainerIP/…
proxy/      Traefik provisioning (provision.go) + per-app routing labels (proxy.go)
source/     resolving a Git URL / local path into a build context
secrets/    sealing/unsealing env envelopes
slots/      resolving --slot-size into cpu/mem/pids caps

The orchestrator: deploy.Deploy()

agent/internal/deploy/deploy.go holds the Orchestrator and its Deploy(ctx, spec, out) method. The Orchestrator wires the pieces and exposes injectable seams (Builder, Runtime, Proxy, and func fields like Smoke/PrevImage) so the whole pipeline is testable without Docker.

A deploy.Spec carries everything about one deploy: App, Source, Ref, Domain, Port, ReadinessPath, Env, Network, Builder (hostpack|dockerfile|image), Command (worker override), Slot/CPUs/MemoryMB/PidsLimit, Volumes, and the optional browser-smoke fields.

The stages, in order:

  1. Validate — app+source required; slot caps validated (choke-point for CLI/jobs/MCP); volume mounts validated (named-only, no host binds); domain validated against proxy.ValidHost.
  2. Source resolve (source.Resolve) — a Git URL becomes a fresh, agent-owned shallow clone in a temp dir (removed after the deploy); a local path is used in place.
  3. Build — the chosen builder produces an image (see below). Env is passed to the builder as build args.
  4. Proxy ensure — if Domain != "" and ManageProxy, proxy.Ensure provisions/refreshes the managed Traefik container idempotently.
  5. Network — ensure the shared docker network exists.
  6. Migrate (optional) — run a one-off release command in a throwaway container.
  7. Run (runApp) — start the new container with env, labels, caps, volumes, and restart policy. The old container stays up until the new one is proven.
  8. Gate — liveness → HTTP readiness (if ReadinessPath) → optional browser smoke. This is where a broken deploy is caught.
  9. Route — the container carries Traefik labels (proxy.Labels) so the managed proxy picks up the domain and requests a cert.
  10. Tag :good + rollback on failure — on a passed gate the image is tagged good; on a failed gate, rollback re-runs runApp with the previous good image so the app keeps serving.

Design invariant: runApp must be idempotent and must not mutate spec.Env in place — the map is shared with the caller (build/migrate/verify already read it) and rollback re-invokes runApp with the same spec. Copy-on-write any per-run env defaulting.

Builders

agent/internal/builder/:

  • hostpack.go — passes env as native flags to the HostPack/Railpack binary, which detects the stack and builds via BuildKit. Handles build env natively (no --build-arg fiddling). This is the zero-config default. ⚠ Known gap (G23): every --env value still lands on the build command line, readable via ps for the duration of the build. Use --builder dockerfile for secret-bearing builds until the hostpack secret path lands.
  • dockerfile.go — shells out to docker build --tag <image> [--build-arg K=V…] [--secret id=K,src=…] <context>. Build args are inert unless the Dockerfile declares matching ARGs — this is the root of guard G11, whose forward fix materializes an env file into the build context. Only PUBLIC build vars go on the command line (--build-arg); secrets never do — see Build-time env & secrets below (guard G23).
  • image — no build; the agent runs a prebuilt image reference directly.

Which one runs: dockerfile when spec.Builder=="dockerfile" or a Dockerfile exists at the context root; image when spec.Builder=="image"; otherwise hostpack.

Build-time env & secrets

A running docker build's argv is world-readable on the box (ps aux, pgrep -af, /proc/<pid>/cmdline) for the whole build — ~10 minutes for a real Next.js app. So the Dockerfile builder splits req.Env in two (agent/internal/builder/buildvars.go, the single definition — never inline a copy):

ClassRuleHow it reaches the build
Public build varNEXT_PUBLIC_*, VITE_*, REACT_APP_*, NUXT_PUBLIC_*, EXPO_PUBLIC_*, GATSBY_*, or an exact key on the non-secret allowlist (NODE_ENV, NODE_VERSION, NIXPACKS_NODE_VERSION, RAILPACK_NODE_VERSION, PORT, NEXT_TELEMETRY_DISABLED, other version pins…), or named in HOSTSSH_PUBLIC_BUILD_VARS--build-arg K=V (argv)
Secreteverything else — the rule is fail-closed, so an unknown key is a secretthe materialized build-context env file (G11), plus a BuildKit --secret mount if the Dockerfile declares one

Why public vars must stay on argv: Next.js (and Vite, CRA, …) inline NEXT_PUBLIC_*/VITE_* into the client bundle at build time, so ARG NEXT_PUBLIC_APP_URL needs the value on the build command line or the browser bundle ships with it undefined. That is not a leak — those values are served to every visitor by definition.

BuildKit secret mounts. A Dockerfile that declares RUN --mount=type=secret,id=<KEY> gets that secret mounted from a 0600 file — only the path reaches argv, and the file is removed when the build returns. The id must equal the env KEY. A Dockerfile declaring no secret mounts gets no --secret flags and builds exactly as before, so no app is required to change its Dockerfile; and a Dockerfile that does declare one already required BuildKit, so this adds no new requirement.

Caveat — the context env file lands in a layer. The G11 file enters the build context, so it is baked into the layer of whichever stage COPYs it. In the standard multi-stage build that stage is discarded (the runtime stage copies only the built artifact), which is why docker history on a shipped image is clean. A single-stage Dockerfile that COPY . . would bake secrets into the shipped image — use a BuildKit secret mount there.

The runtime wrapper

agent/internal/runtime/runtime.go builds the actual docker run command. Every app launches with:

  • --restart unless-stopped (restarts an exited PID 1),
  • securityArgs — the G15 hardening defaults (--security-opt no-new-privileges, dropped caps),
  • --init (proper PID1 signal/zombie handling),
  • the shared network, port/volume/env/label flags (env sorted for determinism),
  • and slot caps (--cpus, --memory, --pids-limit) when a Slot size is set.

State inspects {{.State.Status}};{{.State.Restarting}}; ContainerIP reads the container's IPv4 (used by the readiness gate — an IPv6-only bind fails the gate, backstopping G1). ContainersByLabel lists containers by a label (hostssh.slot, hostssh.app).

Guard G2 forward-fix (in progress): docker never restarts a container that's alive but not serving. The design is a host-side TCP watchdog (label the port, probe it, restart after N consecutive failures with a rate cap) rather than --health-cmd, because arbitrary app images lack curl/shell and a bad in-container probe would restart-loop the fleet. See docs/dev/build-guards.md.

The managed proxy

agent/internal/proxy/:

  • provision.go — Ensure() idempotently runs the hostssh-proxy Traefik v3 container (ports 80/443, docker socket read-only, an hostssh-acme volume for issued certs). It short-circuits if the proxy is already running unless Force:true. It wires ACME: DNS-01 when HOSTSSH_ACME_DNS_PROVIDER=cloudflare + a token is present (issues before cutover, wildcards, orange-cloud-safe — guard G12), otherwise HTTP-01 as the fallback.
  • proxy.go — Labels(Route{Name,Domain,Port}) stamps the Traefik router/service labels (Host(domain) → :port, websecure, tls.certresolver, health-check path). These are routing health checks (stop routing to a bad backend); they do not restart containers.

Env and secrets flow

Env reaches a deploy from three callers, all converging on deploy.Spec.Env:

  1. CLI (cmdDeploy) — merges --env-file (base) with --env flags (override).
  2. Job queue (executeJob → envForJob) — copies the job's plain Env, then decrypts the sealed SecretEnv envelopes (via secrets.DecryptEnvelope, key HOSTSSH_ENC_KEY) and overlays them, persisting the decrypted values to a local 0600 store for restart survival.
  3. MCP — no env.

The merged map is passed verbatim to the builder (build-time), the migrate/verify one-offs, and the runtime container. Sealed secrets are decrypted only inside the agent — never in the queue, logs, or database. See Security model.

Jobs and the control plane's view

A control-plane deploy becomes a jobs.Spec (agent/internal/jobs/jobs.go) with kind deploy/redeploy, pinned to a Node fingerprint. The agent claims it, maps it to a deploy.Spec, runs the pipeline, and ships every stage line back as job logs (via the logShipper writer). Job kinds the agent executes: deploy, redeploy, stop, remove, db, restore, firewall, prune, expose, unexpose. See Agent protocol for the claim/report contract.

Testing the pipeline

All three packages test without Docker:

  • deploy_test.go — fakeRuntime records the last RunSpec; shrinkHealth/shrinkReadiness shrink the gate windows so tests run in milliseconds; httptest servers stand in for the app.
  • orchestrate_test.go — recording fakes assert the exact stage order (e.g. build,tag,net,run,tag) so a reordering is caught.
  • runtime_test.go — stubs execRun/execCapture and asserts the entire docker run command as a literal (any new default flag must be updated deliberately); TestRunHardensByDefault guards the G15 defaults.
  • builder_test.go — stubs execRun and asserts the exact docker build command.

Run: cd agent && go test ./internal/deploy/ ./internal/runtime/ ./internal/proxy/ ./internal/builder/ -race.