Environment variables & secrets

An app's configuration reaches its container as environment variables. HostSSH splits those into two custody classes and never lets a secret value sit in plaintext where it doesn't belong:

• Non-secret vars (PORT, NODE_ENV, a public base URL) — stored in plaintext, carried to the agent in the deploy job, injected with docker run -e.
• Secret vars (DATABASE_URL, API keys, a generated master key) — never stored in plaintext in Postgres and never carried as a value in the job queue. They reach the container through the agent's on-host secret store, injected via a 0600 --env-file.

This doc is the as-built map of how a secret value gets from the dashboard to a running container without ever being persisted in the clear. The user-facing guide is Environment variables & secrets; the regression catalog entry is G12 in planning/BUILD-GUARDS.md.

The custody rule (one function)

Every write of a secret value goes through one rule — storedEnvValue() in lib/platform/store.ts:

Kind	secret	generate	Stored in deployment_env.value
Non-secret	false	—	plaintext
Generated secret	true	true	NULL — the agent mints it on the Node
User-entered secret	true	false, has value	AES-256-GCM ciphertext
Declared, not yet filled	true	false, blank	NULL

Ciphertext is produced by encryptSecret() in lib/addons/crypto/crypto.ts (a versioned v1:<iv>:<tag>:<ct> envelope) under HOSTSSH_ENC_KEY — a 32-byte key held only on the control plane, never in the database. A Postgres dump therefore yields NULLs and ciphertext, never a usable secret. This is the same at-rest custody the Connections and webhook features use; it is fail-closed (a missing key throws rather than silently storing plaintext).

Why this passes the secret-scan gate. CI runs gitleaks over the repository (not the database), so custody = "no plaintext in Postgres" + "no secret literals committed." Tests use fake, low-entropy values and generated (never committed) material.

The three delivery paths

All three converge on the agent's on-host secret store (agent/internal/secrets) — /etc/hostssh/secrets/<app>.json, 0600 — and one injection point.

1. Generate on the Node (generate: true)

For template secrets like MEILI_MASTER_KEY or POSTGRES_PASSWORD. The control plane cannot custody a secret it never persists, so it doesn't generate one — it sends only the key in the job (JobSpec.generateSecrets, keys never values). The agent's secrets.EnsureGenerated(app, keys) mints a strong random value for any key not already present, persists it, and injects it. Because it persists, the value is stable across redeploys — a rotating master key would break the running app, so this is a guarded invariant (TestDeployGeneratedSecretStableAcrossRedeploys).

This closes the original bug: a generate:true template secret used to be generated in the control plane, then dropped to NULL on persist and filtered out of the job — generated then dropped twice, so it never reached the container.

2. Sealed at rest, delivered with the claim (user-entered)

A value an operator types in the App Settings editor is sealed with encryptSecret() and stored as ciphertext. It is never put in the jobs table. Instead, when the agent claims a deploy job, the claim route (app/api/v1/jobs/claim/route.ts) calls getDeploymentSecrets(deploymentId), which decrypts the user-entered secrets in-process and attaches them to the claim response (job.secrets) over pinned TLS. The delivery is fenced by the claim itself — claimJob only returns a job for this license + host. The agent injects them but does not persist them (the control plane stays custody-of-record). A value that fails to decrypt (missing/rotated key) is skipped with a server-side warning rather than stalling the whole queue.

3. Link a managed database by name

A managed database writes its DATABASE_URL (and POSTGRES_*) into its own on-host store under the database's name. An app deployed separately from its DB lists that name in secretSources (persisted on deployments.secret_sources, migration 0019). The agent's secrets.LoadMerged([app, ...sources]) folds those stores in, so the app gets its DB credentials without the operator copy-pasting a connection string.

Injection (the agent side)

In agent/internal/deploy/deploy.go, before the release-phase migration runs, Deploy builds the merged secret map:

EnsureGenerated(app, GenerateSecrets)        // mint + persist generate:true keys
merged = LoadMerged([app] + SecretSources)   // app store + linked DB stores
merged ← DeliverSecrets                        // claim-delivered user secrets win
envFile, inline = WriteEnvFile(merged)         // split single-line vs multi-line

• Single-line secrets are written to a 0600 env-file (secrets.WriteEnvFile) inside the store dir and passed to docker run --env-file, so values never appear in argv or docker inspect.
• Multi-line secrets (PEM keys, JSON blobs) can't live in a docker env-file, so they're returned as inline and folded into the -e env instead — which carries newlines and is not echoed to the deploy log (docker doesn't print its own argv).
• The same env-file is passed to the release-phase migrate and drift-verify one-offs (runtime.RunSpec.EnvFile), so a migration reaches the DB with the app's real creds.
• The env-file is removed once every container that references it is created.

Non-secret env stays in JobSpec.env and is injected with -e as before.

Build time (the other custody boundary)

Everything above is about the runtime container. A build has its own argv, and it leaked: until 2026-07-16 the Dockerfile builder expanded the whole app env into docker build --build-arg K=V, publishing live secrets to any local ps reader for the duration of the build (guard G23 — that is where the incident detail lives).

The rule now, defined once in agent/internal/builder/buildvars.go and fail-closed (an unknown key is a secret):

• Public build vars — NEXT_PUBLIC_*, VITE_*, REACT_APP_*, NUXT_PUBLIC_*, EXPO_PUBLIC_*, GATSBY_*, a short exact allowlist of non-secret pins (NODE_ENV, NODE_VERSION, NIXPACKS_NODE_VERSION, RAILPACK_NODE_VERSION, PORT, …), or a key named in HOSTSSH_PUBLIC_BUILD_VARS — go on argv as --build-arg. They must: a bundler inlines them into the client bundle at build time, so ARG NEXT_PUBLIC_APP_URL needs the value there. They are served to every visitor anyway — argv is not the marginal risk.
• Secrets — everything else — never touch argv. They reach the build through the materialized build-context env file (G11) and, for each id the Dockerfile actually declares (RUN --mount=type=secret,id=<KEY>), a BuildKit --secret id=<KEY>,src=<0600 file> mount whose file is removed when the build returns.

Two caveats worth knowing:

• The G11 context env file lands in a layer — safe in the standard multi-stage build (the runtime stage copies only the built artifact, so the shipped image's history and Config.Env stay clean), but a single-stage Dockerfile that COPY . . would bake secrets into the shipped image. Use a secret mount there.
• The hostpack builder still puts --env values on argv (G23's open half). Prefer --builder dockerfile for secret-bearing builds; hostpack warns at build time.

Editing env (the App Settings editor)

The deploy detail page mounts components/platform/app-env-card.tsx, which saves via updateAppSettingsAction(id, { env, secretSources }). Two custody details:

• Masking. getDeployment never returns secret material — secret values read back as '' (the UI shows •••••••• (secret) / •••••••• (generated)).
• Blank-secret preservation. Because the editor can't read an existing secret back, a save leaves that field blank. updateDeploymentEnv treats a blank, non-generate secret whose key already exists as keep the stored ciphertext — editing one var never silently wipes the others.

Saving persists the desired settings; the operator redeploys to apply them.

Code map

Concern	File
Custody rule + masking + delivery read	control-plane/web/lib/platform/store.ts (storedEnvValue, getDeployment, getDeploymentSecrets, updateDeploymentEnv, updateDeploymentSecretSources)
At-rest seal	control-plane/web/lib/addons/crypto/crypto.ts (encryptSecret/decryptSecret, HOSTSSH_ENC_KEY)
Job spec + validation	control-plane/web/lib/platform/jobs.ts (generateSecrets, secretSources)
Dispatch → job	control-plane/web/lib/platform/actions.ts (dispatch, updateAppSettingsAction)
Claim delivery	control-plane/web/app/api/v1/jobs/claim/route.ts
Editor UI	control-plane/web/components/platform/app-env-card.tsx
Schema	control-plane/db/schema-platform.sql; migrations 0018-deployment-env-generate, 0019-deployment-secret-sources
Agent store	agent/internal/secrets/secrets.go (EnsureGenerated, LoadMerged, WriteEnvFile, Generate)
Agent inject	agent/internal/deploy/deploy.go (secret prep + env-file), agent/internal/runtime/runtime.go (--env-file)
Agent job spec	agent/internal/jobs/jobs.go (GenerateSecrets, SecretSources, Job.Secrets)

Tests / guards

Go: agent/internal/secrets/secrets_test.go (generation stability, env-file 0600 + content, multi-line inline split, LoadMerged), runtime_test.go (--env-file ordering), deploy_test.go (generate + inject, redeploy stability, delivered-not-persisted, linked store, multi-line inline). TS: lib/platform/secret-custody.test.ts (the custody rule), lib/platform/app-settings.test.ts (linking, masking, blank-secret preservation), lib/platform/jobs.test.ts (generateSecrets + secretSources validation). See G12 in planning/BUILD-GUARDS.md.

Paid provisioning boundary

Cloud compute provisioning for Hostinger, Hetzner, DigitalOcean, and Vultr uses the same AES-256-GCM store, but fails closed more strictly: the Connections wizard refuses a provider token unless Postgres, connections, connection_credentials, deks, and a valid HOSTSSH_ENC_KEY are all present. The token is verified with a read-only provider request before it is sealed. Provider API origins are deployment configuration, never browser input.

Paid creation remains disabled unless every guard is set:

HOSTSSH_PROVISIONING_LIVE=1
HOSTSSH_PROVISIONING_MAX_MONTHLY_USD=<positive amount>
PROVISIONING_WORKER_SECRET=<dedicated random bearer>
PROVISIONING_ENROLL_SECRET=<at least 32 characters>
HOSTSSH_PROVISIONING_PUBLIC_URL=<public HTTPS control plane>
<PROVIDER>_PLAN_MAP_JSON={...}
<PROVIDER>_REGION_MAP_JSON={...}

Hostinger additionally requires HOSTINGER_VPS_TEMPLATE_ID; Hetzner and DigitalOcean require their *_IMAGE; Vultr requires VULTR_OS_ID. Optional SSH key arrays and backup flags can be stored with the Connection or supplied by the server environment. Safe reads and exact-ID deletes use HOSTSSH_PROVIDER_API_TIMEOUT_MS and the bounded HOSTSSH_PROVIDER_READ_RETRIES; billable creates are never retried automatically.

Tokens normally come from verified Connections. HOSTINGER_API_TOKEN, HETZNER_API_TOKEN, DIGITALOCEAN_API_TOKEN, and VULTR_API_KEY are supported only as labelled legacy server-environment fallbacks. The worker secret is not a license key: ordinary agents and customers cannot trigger a billable operation.

Inference artifact boundary

The Deploy catalog resolves the first-party AI Gateway image from one server-only release setting:

HOSTSSH_AI_GATEWAY_IMAGE=<registry/image>@sha256:<64-hex-digest>

The catalog stays visibly blocked when this is absent or uses a floating tag, and the server repeats that validation before any deployment record is written. This setting identifies an artifact; it is not injected into the gateway container.

The gateway's GATEWAY_TOKEN is a different credential. Mint it on the Inference page, where its plaintext is shown once, then paste it into the template's secret field. It follows the user-entered sealed-secret path above. Do not mark it generate:true: Node-generated values intentionally cannot be read back by a remote API client.