GPU support (agent)

HostSSH runs GPU inference/training workloads by reserving GPUs for a deployed container — the hostssh deploy -gpu flag, translated to docker run --gpus. There is no GPU scheduler and no multi-tenant VRAM sharing: a workload reserves the GPU(s) it asks for and owns them. Shipped e3c6a86 (2026-07-11); proven end-to-end on a real RTX 3090 on 2026-07-12 (see ../../planning/GPU-NODE-RUNBOOK.md). Operator-facing guide: GPU workloads.

Code map

Concern	Where
Normalize -gpu → docker arg, sample stats, doctor report	agent/internal/gpu/gpu.go (NormalizeSpec, Stats, Doctor)
Emit --gpus on the container	agent/internal/runtime/runtime.go — RunSpec.GPU, appended in Run()
Carry the reservation through deploy	agent/internal/deploy/deploy.go — Spec.GPU (validated in Deploy via gpu.NormalizeSpec), Spec.Args
Persist across redeploys	agent/internal/jobs/jobs.go — Spec.GPU (json:"gpu") → wired to deploy.Spec in cli.go
CLI surface	agent/internal/cli/cli.go — -gpu, -image, trailing -- ARGV in cmdDeploy
doctor gpu row	agent/internal/cli/cli.go — checkGPU → gpu.Doctor
Heartbeat stats	agent/internal/telemetry/telemetry.go — Heartbeat.GPUs []GPUSample; populated from gpu.Stats() in the heartbeat loop
Traefik long-response timeout	agent/internal/proxy/provision.go — proxyResponseTimeout() (≥620s floor)

The -gpu value → docker --gpus

gpu.NormalizeSpec validates and normalizes at the deploy choke-point (so CLI, jobs queue, and MCP are all covered by one guard), storing the docker arg on Spec.GPU:

-gpu input	--gpus arg	Meaning
(absent)	(none)	CPU container — unchanged behavior
all	all	every GPU
2	2	a count (docker picks them)
0 / 0,1	device=0,1	specific device indices
GPU-<uuid>,…	device=GPU-<uuid>,…	specific device UUIDs

A count 0, over-64, or a value with unsafe chars is rejected before any container runs.

Explicit command vector (-- ARGV)

cmdDeploy reads fs.Args() (everything after --) into Spec.Args, run verbatim after the image's entrypoint (quoting preserved) — unlike -command, which wraps in sh -c. -image is an alias for -source when -builder image. Together they make the acceptance smoke work:

hostssh deploy -name gpu-smoke -builder image \
  -image nvidia/cuda:12.4.0-base-ubuntu22.04 -gpu all -- bash -c 'nvidia-smi -L'

A command override (-command or Args) marks a non-web workload: it skips the port watchdog label, the Next.js cache volume, the HOSTNAME=0.0.0.0 default, and blue-green (nothing to swap behind a proxy).

doctor gpu row (gpu.Doctor)

• n/a — no nvidia-smi on PATH (a CPU node — NOT a failure; most nodes are CPU boxes).
• ok — nvidia-smi + a docker run --gpus all … nvidia-smi -L probe both pass (GPU reaches containers).
• fail — GPU present on the host, but a container can't get it → install nvidia-container-toolkit + enable Docker's nvidia runtime.

The container probe is skipped unless the probe image is already local (HOSTSSH_GPU_PROBE_IMAGE, default nvidia/cuda:12.4.0-base-ubuntu22.04), so doctor never triggers a large pull. checkGPU gates on lookPath("nvidia-smi") first, so on a CPU node the row is n/a with no exec (keeps the doctor unit tests GPU-free).

Heartbeat GPU stats

On a GPU node the heartbeat carries GPUs: []GPUSample{ Name, MemTotalMB, MemUsedMB, UtilPct }, sampled via nvidia-smi --query-gpu=name,memory.total,memory.used,utilization.gpu --format=csv,noheader,nounits (gpu.Stats). Nil on a CPU node — the heartbeat is unchanged there. The control plane surfaces GPU capacity + load from this.

Long-response proxy timeout (companion)

Long inference/render jobs run past Traefik v3's default read (60s) / idle (180s) timeouts. The managed proxy's TLS entrypoint now sets respondingTimeouts.readTimeout/idleTimeout floored at 620s (default 900s, HOSTSSH_PROXY_RESPONSE_TIMEOUT), writeTimeout=0 (875a2f3). Same ≥620s floor as the cloudflared tunnel path (internal/expose/ingress.go).

Node prerequisites & join

A GPU node needs the NVIDIA driver + nvidia-container-toolkit (Docker nvidia runtime). Vast.ai's KVM VM image ships the driver + an nvidia runtime but not the toolkit — install it, or --gpus fails with "could not select device driver". The agent also needs HOSTSSH_LICENSE_PUBKEY in its env to verify the license token at activation. Full turnkey procedure + the two live gotchas: ../../planning/GPU-NODE-RUNBOOK.md.

Model pre-warm (-warmup)

GPU model pre-warm is a platform primitive (agent v0.6.0): hostssh deploy … -warmup 'curl -sf localhost:9700/warmup' runs the command inside the new container (docker exec sh -c) after liveness/readiness and before the deploy is declared live — on a blue-green swap, before the swap — so the first real request never pays the cold weight-load. A non-zero exit fails the gate exactly like a failed readiness probe (the candidate is discarded / the direct deploy rolls back). It rides the deployment spec (warmupCmd, migration 0031), so control-plane redeploys keep it. See deploy.Spec.WarmupCmd in agent/internal/deploy/deploy.go.

Out of scope

No GPU scheduler / VRAM sharing. Building from source on a GPU node still needs the hostpack/buildkit toolchain — use -builder dockerfile or a prebuilt image meanwhile.