GPU workloads

HostSSH can run GPU inference and training workloads on any node that has an NVIDIA GPU and the container toolkit. A GPU deploy is a normal deploy with one extra flag — -gpu — that reserves the GPU(s) for the container, the same way --slot-size reserves CPU and memory.

Proven on hardware (2026-07-12): a rented Vast.ai RTX 3090 joined a live fleet and ran a -gpu all deploy end to end. For the turnkey join-a-rented-box procedure, see the GPU node runbook.

Mental model: a GPU node is just a node with a GPU. You install the agent the same way, add the NVIDIA container prerequisites, and deploy with -gpu. There is no GPU scheduler and no multi-tenant VRAM sharing — one workload owns the GPU(s) it reserves.

---

1. Reserve a GPU on deploy

Add -gpu to hostssh deploy. It accepts three forms:

Flag	Meaning	Docker equivalent
-gpu all	Every GPU on the node	--gpus all
-gpu 2	A count — any 2 GPUs	--gpus 2
-gpu 0,1	Specific device indices (or GPU-<uuid>s)	--gpus device=0,1

Absent -gpu, the container gets no GPU — behavior is unchanged from a CPU deploy.

The reservation is stored on the app spec, so every redeploy keeps it — you don't re-pass -gpu on a dashboard-driven redeploy.

Smoke test

hostssh deploy -name gpu-smoke -builder image \
  -image nvidia/cuda:12.4.0-base-ubuntu22.04 -gpu all -- bash -c 'nvidia-smi -L'

• -image is shorthand for -source when running a prebuilt image.
• Everything after -- is the container command, run verbatim (quoting preserved).

nvidia-smi -L lists the GPUs the container can see, then exits — so this container is a one-shot: it runs, prints the GPU list to its logs, and stops (the deploy will report it exited, which is expected for a one-shot). Check what it saw:

docker logs gpu-smoke      # → "GPU 0: NVIDIA … (UUID: GPU-…)"

For a real inference service, deploy a long-running command instead (a server process), and it stays up and gets HTTPS routing like any other app.

---

2. Node prerequisites

A GPU node needs, on the host:

1. The NVIDIA driver (nvidia-smi works on the host).
2. nvidia-container-toolkit and Docker's nvidia runtime enabled, so the GPU passes into a container.

Verify both at once:

hostssh doctor

The gpu row reports:

Status	Meaning
n/a	No GPU on this node — it's a CPU box (not a failure; most nodes are CPU).
ok	GPU + driver/CUDA present and docker run --gpus all passes the GPU into a container.
fail	GPU present on the host, but a container can't get it — install nvidia-container-toolkit and enable the nvidia runtime.

The doctor's container probe runs docker run --rm --gpus all <image> nvidia-smi -L. It uses nvidia/cuda:12.4.0-base-ubuntu22.04 by default (override with HOSTSSH_GPU_PROBE_IMAGE) and is skipped unless that image is already local, so doctor never triggers a large pull — run the smoke deploy above once and the image is present.

Once a node has a working GPU, its heartbeats carry GPU stats — model, VRAM total, VRAM used, and utilization per GPU — so GPU capacity and load are visible in the control plane.

---

3. Models volume (weights)

Model weights are large (50–200 GB) and you don't want to re-download them on every redeploy. Use a named volume — the same persistent-volume mechanism every app uses:

hostssh deploy -name llm -builder image -image my/inference:latest \
  -gpu all -port 8000 -volume models:/models

The models volume survives the docker rm -f that each redeploy performs, so the weights are downloaded once and reused across redeploys and image updates. No special GPU handling — a named volume persists across GPU redeploys exactly like any other.

---

4. Exposing a rented GPU box

A rented GPU box usually has no public route. Expose a node-local port through a Cloudflare Tunnel:

• hostssh doctor shows reachability (public vs private/CGNAT) and whether cloudflared is installed with HOSTSSH_CF_TUNNEL + an active service. Full join path: tunnel-first.
• The tunnel's ingress must map the public hostname to http://localhost:<port>, with a per-route timeout of at least 620 seconds — long video/render jobs run well past cloudflared's short defaults. HostSSH generates the ingress config with the origin timeouts set generously and enforces a 620 s floor (default 900 s; raise it with HOSTSSH_CF_ROUTE_TIMEOUT).

Edge caveat: Cloudflare's edge imposes a ~100 s HTTP proxy timeout on non-Enterprise plans. The tunnel config removes the origin-side limits so cloudflared never truncates the stream, but a job that runs past ~100 s must stream its response (bytes flowing keep the connection alive) or use WebSockets, which cloudflared passes through without the HTTP timeout. That is the supported pattern for long GPU jobs on a free/pro plan.

---

5. Onboarding a rented GPU box (rental types)

The agent installs on any Ubuntu box — the same join flow as any node. What matters for GPU rentals is whether Docker can use the GPU:

Rental type	GPU-in-Docker works?	Notes
Full VM / bare-metal (Vast.ai "VM" offers, RunPod bare-metal/secure-cloud with host access)	✅ Yes	You control the host: install the driver + nvidia-container-toolkit and the agent runs Docker with --gpus normally. Preferred.
Nested-container rental (Vast.ai/RunPod default container instances)	⚠️ Only with Docker-in-Docker	You get a container, not a host. Running the agent's Docker inside it needs DinD with GPU passthrough (--gpus forwarded into the nested daemon). Possible but fiddly — confirm the provider allows a privileged nested daemon with GPU access.
Managed serverless GPU (RunPod serverless endpoints)	❌ No	No host Docker to join — these aren't HostSSH nodes.

Recommendation: rent a full VM or bare-metal GPU box. Then it's a normal node: install the driver + nvidia-container-toolkit, run the agent join (the .223 join is the template), confirm hostssh doctor shows ok gpu, and deploy with -gpu.

---

Out of scope

There is no GPU scheduler and no multi-tenant VRAM sharing. A workload reserves the GPU(s) it asks for and owns them. Building from source on a GPU node still needs the hostpack/buildkit toolchain (a separate gap) — meanwhile use -builder dockerfile or a prebuilt image (-builder image).