Add-on — Outbound Webhooks

Customers subscribe an endpoint to fleet events (backup.done, deploy.finished, …); the platform delivers signed HTTP POSTs with at-least-once durability and automatic reliability backoff. Absorbed from altohost, but the in-process retry loop, Redis circuit breaker, and Prisma store were all left behind — this version is durable and dependency-light.

• id / entitlement: webhooks — granted at team+ (see addon-entitlements.ts).
• Code: lib/addons/webhooks/. Import from @/lib/addons/webhooks.
• Schema: control-plane/db/schema-webhooks.sql + migration 0013-webhooks.
• Reuses: the ssrf guard (safeFetch / assertFetchableUrl) and the crypto guard (secret sealing). See Add-on system.

Flow

import { emitEvent, runWebhookQueueOnce } from '@/lib/addons/webhooks'

// Producer (e.g. a deploy finished): fan out, never blocks on HTTP.
await emitEvent(tenantId, { event: 'deploy.finished', data: { deployId } })

// Worker tick (cron/loop): drain the durable queue.
await runWebhookQueueOnce()

emitEvent(tenantId, { event, data }) (emit.ts) resolves every enabled endpoint for the tenant whose eventTypes contains the exact event name or '*', and enqueues one delivery row per endpoint. It returns { event, enqueued, deliveryIds } and does not block on the HTTP call — delivery is the worker's job.

Signing (Stripe-style)

sign.ts. A receiver verifies the raw body against the endpoint secret shown once at registration.

Detail	Value
Header	X-HostSSH-Signature
Format	t=<epoch_seconds>,v1=<hmac_hex>
HMAC	SHA-256 over the string <timestamp_seconds>.<raw_body>
Replay window	300 s default (toleranceSec) — reject too-old / too-new timestamps
Compare	timingSafeEqual on hex buffers (length-checked first)
Secret format	whsec_<48 hex chars> (24 random bytes) via generateWebhookSecret()

Functions: generateWebhookSecret(), signPayload(tsSec, body, secret), buildSignatureHeader(tsSec, body, secret), verifySignature(header, body, secret, { toleranceSec?, nowSec? }).

Circuit breaker (pure, no Redis)

circuit.ts — a pure state machine over recent delivery outcomes, so it needs no external store. States: closed → open → half_open. DEFAULT_CIRCUIT:

Field	Value	Meaning
windowSize	100	outcomes considered for reliability
openThreshold	0.3	open below a 30% success ratio
minAttempts	10	need ≥10 recent outcomes before opening
cooldownMs	300000	5 min before an open circuit surfaces as half_open

computeReliability(outcomes) (empty window ⇒ 1.0), effectiveState(stored, now) (an open past cooldown reads as half_open), shouldDeliver(stored, now) (true for closed/half_open), transitionAfterOutcome(stored, outcomes, success, now). A half_open success closes the circuit; a half_open failure re-opens it.

The durable queue (the delivery row is the job)

store.ts. Dual backend — postgres.js when DATABASE_URL is set, otherwise an in-memory store with the same API (tests + dev). Endpoint secrets are sealed at rest with the crypto add-on (encryptSecret) and unsealed only at delivery time.

• Claim — claimDueDeliveries(limit, lockedBy) selects status='pending' AND run_after<=now() ORDER BY run_after FOR UPDATE SKIP LOCKED, flips them to delivering + locked_by/locked_at. Multiple workers can run concurrently without collisions.
• Complete — completeDelivery(id, statusCode) → delivered, attempts++ (terminal).
• Fail — failDelivery(id, error, statusCode) → attempts++; if attempts >= maxAttempts set failed (terminal), else back to pending with run_after = now + backoff.
• Defer — deferDelivery(id, runAfterMs) re-queues without spending an attempt (used when the circuit is open).
• Backoff — nextBackoffMs(attempts, baseMs=30000, capMs=3_600_000) = min(cap, base * 2^(attempts-1)) → 30 s, 60 s, 120 s … capped at 1 h. Default maxAttempts = 5.
• recentOutcomes(endpointId, limit) — the boolean success history the circuit reads (terminal deliveries only).

Delivery lifecycle statuses: pending → delivering → delivered | failed.

The worker tick

runWebhookQueueOnce(opts?) (worker.ts). Defaults: batchSize 10, timeoutMs 10000 per delivery, workerId webhooks@<host>:<pid>. Per claimed delivery:

1. Endpoint gone/disabled → fail.
2. Circuit gate (shouldDeliver) — if open, defer (re-queue, no attempt spent, counted as blocked).
3. Unseal the signing secret (missing → fail).
4. Build the body { id, event, data, time_ms }, sign it, POST via the ssrf add-on's safeFetch (so a redirect/DNS-rebind to a private host is blocked mid-delivery).
5. 2xx → complete; non-2xx or network/timeout → fail (retry or terminal).
6. Recompute the circuit state from the 100-outcome window and persist if it changed.

Headers on every POST: content-type: application/json, x-hostssh-signature, x-hostssh-event, x-hostssh-delivery, x-hostssh-attempt, user-agent: HostSSH-Webhooks/1.0. Returns { claimed, delivered, deferred, failed, blocked }.

HTTP routes

Operator routes under /api/v1/webhooks/* are guarded by the platform license-key bearer via agentGate (rate-limited); the SSRF check runs at registration.

Method · Path	Auth	Request	Response
POST /api/v1/webhooks/endpoints	license bearer (agentGate)	{ url, eventTypes: string[], tenantId?, brandId? } — eventTypes non-empty; use ["*"] for all	201 { id, url, eventTypes, secret } — the secret is shown once
GET /api/v1/webhooks/endpoints?tenantId=	license bearer	—	{ endpoints } (no secrets)
DELETE /api/v1/webhooks/endpoints/{id}	license bearer	—	204 (cascades queued deliveries); 404 if unknown
POST /api/v1/webhooks/run?batch=<n>	license bearer	—	the tick summary { claimed, delivered, deferred, failed, blocked }

The registration route rejects SSRF targets up front via assertFetchableUrl(url) (resolve-and-pin), so a private/metadata endpoint URL is a 422 at create time. endpoints/route.ts · run/route.ts.

Related — inbound GitHub push webhook

Distinct from the outbound add-on: POST /api/v1/git/webhook (route) is the "push to your repo → it ships" path on the deploy queue. It uses GitHub's own signing scheme (not the add-on's): it looks the hook up by (repo, branch) first (unknown repo ⇒ cheap 202), then verifies GitHub's X-Hub-Signature-256 HMAC over the raw body against the hook's secret, then enqueues a license-scoped deploy job pinned to the hook's node. Branch-delete pushes (after = 40 zeros) are ignored. Hooks are managed by operators at POST/GET /api/v1/git/hooks (settings.manage RBAC), which returns the secret + the payload URL to paste into GitHub once.

Honest status notes

• The deploy producer is already wired: an agent job-status callback fires emitDeployFinished → emitEvent('deploy.finished') (app/api/v1/jobs/[id]/status/route.ts
  • lib/platform/fleet-events.ts). What remains is the backup.done producer call-site and a scheduled runWebhookQueueOnce tick (see the add-on's README).
• Operator routes are gated by the license-key bearer (agentGate) until better-auth dashboard sessions land; there is no per-endpoint entitlement re-check in the route beyond that bearer.
• Inbound webhook relay (altohost's webhook-relay.ts / inbound-webhooks.ts) was deliberately not harvested.