Email & Deliverability add-on

Architecture of the self-hosted, white-label email engine — engine → native Postgres queue → send worker → API surface — with the /v1/email/* endpoints, DKIM/SPF/DMARC handling, and where real outbound is gated.

Email & Deliverability add-on

A native, 100%-ours email capability absorbed from usermails' tech and rebranded: own-MTA SMTP transport, in-app RFC 6376 DKIM signing, VERP bounce routing, a native Postgres send queue, and an hs_-key send API. No third-party sender — there is no SES/SendGrid/Mailgun/Resend SDK anywhere in the path, and no vendor identity is hardcoded (From/bounce/DKIM domains are always caller-supplied per-tenant brand).

Dependencies deliberately dropped on absorb: nodemailer, bullmq, ioredis, Prisma, Hono, smtp-server, Cloudflare DoH. The transport is pure-TS, the queue is the backbone's own Postgres pattern, persistence is postgres.js, and the API is Next.js route handlers.

Where it lives

LayerPath
Barrel (exports)control-plane/web/lib/addons/email/index.ts
Manifestcontrol-plane/web/lib/addons/email/manifest.ts
Types + validationcontrol-plane/web/lib/addons/email/types.ts
Provider seamcontrol-plane/web/lib/addons/email/provider.ts
MIME buildercontrol-plane/web/lib/addons/email/mime.ts
DKIM keygen/records/sign/verifycontrol-plane/web/lib/addons/email/dkim.ts
VERP bounce encode/decodecontrol-plane/web/lib/addons/email/verp.ts
SMTP transportcontrol-plane/web/lib/addons/email/smtp.ts
Queue helpers (pure)control-plane/web/lib/addons/email/records.ts
Persistence + queuecontrol-plane/web/lib/addons/email/store.ts
API keyscontrol-plane/web/lib/addons/email/keys.ts
Send workercontrol-plane/web/lib/addons/email/worker.ts
API routescontrol-plane/web/app/api/v1/email/*
Operator pagecontrol-plane/web/app/(god)/email/page.tsx
Schemacontrol-plane/db/schema-email.sql (migration 0012-email)

The manifest (manifest.ts) is status: 'active', entitlement: 'email', and declares env: ['HOSTSSH_ENC_KEY', 'HOSTSSH_SMTP_URL'].

The pipeline

POST /v1/email/send  ──►  enqueueSend()  ──►  email_jobs (Postgres queue)
   (auth, validate,        (email row +        │
    verified-domain,        send job, one       │  runSendQueueOnce() (worker)
    suppression,            transaction)        ▼
    ssrf-guard)                          claim (SKIP LOCKED)
                                          → drop suppressed recipients
                                          → unseal + attach DKIM
                                          → MailProvider.send()   ── default: SmtpTransport ──► your MTA
                                          → lifecycle: sent / deferred+backoff / dead-letter

Four seams keep it honest and testable:

  1. MailProvider (provider.ts) — the one extension point every backend implements. The shipped default is our own SmtpTransport (smtp.ts); the worker takes an injectable provider so the whole tick is unit-tested with no network.
  2. The native queue (store.ts) — the backbone's Postgres job pattern (claim via FOR UPDATE SKIP LOCKED, exponential backoff, dead-letter), not Redis/BullMQ.
  3. Dual backend — every store function runs against postgres.js when DATABASE_URL is set, else an in-memory backend (mirrors the rest of the control plane).
  4. DKIM at rest — private keys are sealed by the crypto core add-on (encryptSecret/decryptSecret) and only unsealed in memory at send time.

Engine (typesmime/dkim/verp/smtp)

  • types.tsNormalizedMessage is the provider-agnostic shape the send API normalizes into. validateMessage() enforces MAX_RECIPIENTS (1000), MAX_ATTACHMENTS (25), and at-least-one-body; the MAX_MESSAGE_BYTES (25 MiB) cap is enforced separately in mime.ts's buildMime() on the assembled raw message. addressOnly / domainOf are the address helpers.
  • mime.ts — RFC 2822/MIME builder (header-injection-safe; boundary =_hs_; emits Date + Message-ID).
  • dkim.ts — pure node:crypto RFC 6376 relaxed/relaxed signer + verifier (usermails delegated signing to nodemailer; this is net-new ours). Supports rsa-sha256 (RSA-2048, default) and ed25519-sha256 (RFC 8463). generateDkimKeypair self-checks that the public key derives from the private key. domainDnsRecords() produces the DKIM/SPF/DMARC records a customer publishes — see below.
  • verp.tsbounceReturnPath(emailId, domain)bounce+<emailId>@<domain> on a domain you control; parseBounceRecipient() decodes the id back out of a DSN's envelope recipient (no body parsing). DMARC alignment is preserved because the From: header keeps the customer domain and is DKIM-signed with that domain's key.
  • smtp.ts — pure-TS SMTP transport (EHLO/STARTTLS/AUTH/MAIL/RCPT/DATA) with IPv4-force, retry/backoff, and a circuit breaker. This is the default MailProvider.

DKIM / SPF / DMARC handling

Record generation is in dkim.tsdomainDnsRecords(domain, publicKey, opts):

RecordNameValue
DKIM<selector>._domainkey.<domain>v=DKIM1; k=<rsa|ed25519>; p=<pubkey>
SPF<domain>v=spf1 include:<spfInclude> ~all (or bare v=spf1 ~all)
DMARC_dmarc.<domain>v=DMARC1; p=none; rua=mailto:<dmarcRua> (or v=DMARC1; p=none;)

Everything is caller-supplied (selector, spfInclude, dmarcRua, algorithm) — nothing is hardcoded to a vendor. Default selector is hs.

Record verification and health live in the dns-integrity add-on, which the email routes call:

  • verifyDomainAuth(domain, { selector, dkimPublicKey, spfInclude }) (lib/addons/dns-integrity/verify.ts) — checks live DNS via the system resolver (no third-party lookup). Per-check VerifyStatus is verified | failed | pending; overall is verified only when DKIM and SPF pass. DKIM failed = record present but key mismatch; SPF failed = multiple v=spf1 records or a missing include. DMARC is advisory — present → verified, else pending; it never fails onboarding.
  • analyzeMailHealth(domain, { dkimSelectors }) (lib/addons/dns-integrity/health.ts) — read-only 0–100 deliverability score over MX / SPF (RFC 7208 10-lookup budget) / DMARC / DKIM, plus a risks catalog.

API surface (/v1/email/*)

Two auth models coexist:

  • Customer/tenant routesAuthorization: Bearer hs_live_…, resolved by authenticateApiKey() to an ApiKeyPrincipal { keyId, tenantId, brandId }. Every query is tenant-scoped by that principal.
  • Operator routes (/keys, /keys/{id}, /alerts/run) — guarded by the platform credential (agentGate / a cron secret), pending a dashboard-session gate once better-auth lands. The key store is auth-agnostic, so that swap is local to the route.

All routes are runtime = 'nodejs', dynamic = 'force-dynamic', and use the shared errors / errorToResponse / json / parseBody helpers from @/lib/addons.

POST /v1/email/domains — register a sending domain

Auth: hs_ key. Generates a DKIM keypair (private key sealed via createDomain), persists the domain (status: pending), and returns the records to publish.

Request:

{ "domain": "mail.example.com", "selector": "hs",
  "algorithm": "rsa-sha256", "spfInclude": "spf.example.net",
  "bounceDomain": "…", "dmarcRua": "dmarc@example.com" }

domain is required and regex-validated; algorithm must be rsa-sha256 or ed25519-sha256; the rest are optional. Response 201:

{ "id": "emd_…", "domain": "mail.example.com", "status": "pending",
  "records": [ { "type": "TXT", "name": "…", "value": "…", "purpose": "dkim|spf|dmarc" } ] }

GET /v1/email/domains — list this tenant's domains

Auth: hs_ key. Returns { domains: EmailDomain[] } scoped to the principal's tenant. Key material is never included (the public EmailDomain view carries dkimPublicKey only).

POST /v1/email/domains/{id}/verify — confirm DNS is live

Auth: hs_ key (must own the domain, else 404). Calls verifyDomainAuth and, when overall === 'verified', flips the domain to verified via setDomainStatus. Read-only against DNS; never auto-fails a pending domain. Response:

{ "id": "emd_…", "status": "verified|pending|failed",
  "verification": { "dkim": "…", "spf": "…", "dmarc": "…", "overall": "…", "details": [ … ] } }

GET /v1/email/domains/{id}/health — deliverability score

Auth: hs_ key (must own the domain, else 404). Calls analyzeMailHealth(domain, { dkimSelectors: [domain.selector] }). Response: { id, health: { score, mx, spf, dmarc, dkim, senders, risks } }.

POST /v1/email/send — the send API (Resend-shaped)

Auth: hs_ key. Order of operations (app/api/v1/email/send/route.ts):

  1. IP rate-limit (300/min) before any DB work, then per-key rate-limit (120/min) after auth — both via lib/ratelimit429 with Retry-After.
  2. Validatefrom, subject, and one of html/text required; at least one recipient across to/cc/bcc. Both camelCase and snake_case (reply_to, content_type) accepted.
  3. Attachments — each needs a filename; body is base64 content or a public http(s) URL in path (guarded by assertPublicUrl from the ssrf add-on). A server-local path is rejected (LFI/SSRF).
  4. Verified-domain gatedomainOf(from) must match a status === 'verified' domain on the tenant, else 403.
  5. Suppression — rejected only if every recipient is suppressed (validation error). Per-recipient suppression happens later, in the worker.
  6. validateMessage structural check, then enqueueSend (transactional email row
    • send job).

Success: 202 { "id": "em_…", "status": "queued" }.

POST|GET /v1/email/keys and DELETE /v1/email/keys/{id} — operator key management

agentGate-guarded. POST mints a key and returns the plaintext token once (createApiKey stores only the sha256; keyPrefix is the non-secret first 12 chars). GET ?tenantId= lists the public view (no hashes). DELETE revokes (idempotent; 404 if no active key). Tokens are hs_live_<base64url>; authenticateApiKey short-circuits anything not starting hs_, matches by sha256, rejects revoked keys, and stamps last_used_at.

POST|GET /v1/email/alerts/run — platform notification scheduler

Not a customer route: a cron entrypoint that queues license-expiry/dunning mail and drains the queue once (runPlatformEmailNotifications). Authorized by EMAIL_ALERTS_CRON_SECRET, falling back to MONITORS_CRON_SECRET (constant-time compare; fail-closed when unset).

The send worker

runSendQueueOnce(opts) (worker.ts) is one drain tick — wire it to any scheduler (mirrors the monitors cron). Per tick it:

  1. claimDueJobs(batch, workerId, undefined, 'send') — claims only send-kind jobs.
  2. For each job: dropSuppressed filters suppressed recipients per-tenant; if none remain, the email is marked failed (all recipients suppressed) and the job completes.
  3. If the job carries a domainId, getDomainSigningKey unseals the DKIM key in memory and attaches { domainName, keySelector, privateKey, algorithm } to the message.
  4. provider.send(msg) delivers; on success → email sent (+ providerMessageId), job succeeded, a sent event recorded.
  5. On error → failJob applies exponential backoff; a still-retryable job re-queues the email as queued (deferred event); an exhausted job dead-letters and marks the email failed. A single poisoned job never stalls the tick.

The default provider is resolveProvider() building a SmtpTransport from HOSTSSH_SMTP_URL (and optional HOSTSSH_BOUNCE_DOMAIN); a fake provider is injected in tests.

Where real outbound is gated

Live delivery is not enabled by default. The engine, queue, API, and worker all ship and are unit-tested, but a queued message only leaves the building when two things are true:

  1. HOSTSSH_SMTP_URL is set to a real MTA — resolveProvider() throws without it ("HOSTSSH_SMTP_URL is not set — cannot deliver mail (set it, or pass a provider)."), so a worker tick can't deliver.
  2. The Go agent MTA sidecar (the box with port-25 egress + PTR) is stood up to pull and deliver send jobs — this is a remaining harvest phase (see planning/HARVEST-BACKLOG.md, phase 4b), not yet wired.

Sending live mail is an irreversible, outward-facing action, so the trigger + sidecar are wired deliberately, not by default. Until then, messages persist as queued and the operator page's SMTP badge reads unset. The operator page also gates on connections.manage and surfaces the HOSTSSH_PLATFORM_EMAIL_FROM / HOSTSSH_PLATFORM_EMAIL_DOMAIN_ID platform-channel wiring.

Data model

schema-email.sql (migration 0012-email): email_domains (DKIM private key sealed at rest), email_api_keys (sha256 hash + prefix), emails (message log), email_jobs (the native queue), email_events, email_suppressions. The records.ts state machine (nextBackoffMs, isValidJobTransition) is pure and unit-tested without a database.

Testing

Every module has a network-free *.test.ts alongside it: MIME header-injection + RFC 2047

  • multipart; DKIM sign→verify roundtrip (RSA + Ed25519) + tamper detection + known empty-body hash; VERP symmetry; an in-process loopback SMTP proving build→sign→VERP→deliver; the queue state machine; API-key mint/authenticate/revoke; and the worker tick via a fake provider. Route tests cover domains, keys, and send.

Extending

Add a delivery backend by implementing MailProvider (provider.ts) and passing it as opts.provider to runSendQueueOnce (or setting it as the default). Connected-account backends (Gmail/Graph) implement the same seam — the send API never changes. The barrel (index.ts) is the single import surface for the whole add-on.

  • DNS Tools — the interactive SuperTool + scheduled monitors that share the SPF/DMARC/DKIM/blacklist toolkit this add-on's verification builds on.
  • Sending email — the task-oriented walkthrough of this same flow.