Email & Deliverability add-on
Architecture of the self-hosted, white-label email engine — engine → native Postgres queue → send worker → API surface — with the /v1/email/* endpoints, DKIM/SPF/DMARC handling, and where real outbound is gated.
Email & Deliverability add-on
A native, 100%-ours email capability absorbed from usermails' tech and rebranded:
own-MTA SMTP transport, in-app RFC 6376 DKIM signing, VERP bounce routing, a native
Postgres send queue, and an hs_-key send API. No third-party sender — there is no
SES/SendGrid/Mailgun/Resend SDK anywhere in the path, and no vendor identity is hardcoded
(From/bounce/DKIM domains are always caller-supplied per-tenant brand).
Dependencies deliberately dropped on absorb: nodemailer, bullmq, ioredis, Prisma, Hono, smtp-server, Cloudflare DoH. The transport is pure-TS, the queue is the backbone's own Postgres pattern, persistence is postgres.js, and the API is Next.js route handlers.
Where it lives
| Layer | Path |
|---|---|
| Barrel (exports) | control-plane/web/lib/addons/email/index.ts |
| Manifest | control-plane/web/lib/addons/email/manifest.ts |
| Types + validation | control-plane/web/lib/addons/email/types.ts |
| Provider seam | control-plane/web/lib/addons/email/provider.ts |
| MIME builder | control-plane/web/lib/addons/email/mime.ts |
| DKIM keygen/records/sign/verify | control-plane/web/lib/addons/email/dkim.ts |
| VERP bounce encode/decode | control-plane/web/lib/addons/email/verp.ts |
| SMTP transport | control-plane/web/lib/addons/email/smtp.ts |
| Queue helpers (pure) | control-plane/web/lib/addons/email/records.ts |
| Persistence + queue | control-plane/web/lib/addons/email/store.ts |
| API keys | control-plane/web/lib/addons/email/keys.ts |
| Send worker | control-plane/web/lib/addons/email/worker.ts |
| API routes | control-plane/web/app/api/v1/email/* |
| Operator page | control-plane/web/app/(god)/email/page.tsx |
| Schema | control-plane/db/schema-email.sql (migration 0012-email) |
The manifest (manifest.ts) is status: 'active', entitlement: 'email', and declares
env: ['HOSTSSH_ENC_KEY', 'HOSTSSH_SMTP_URL'].
The pipeline
POST /v1/email/send ──► enqueueSend() ──► email_jobs (Postgres queue)
(auth, validate, (email row + │
verified-domain, send job, one │ runSendQueueOnce() (worker)
suppression, transaction) ▼
ssrf-guard) claim (SKIP LOCKED)
→ drop suppressed recipients
→ unseal + attach DKIM
→ MailProvider.send() ── default: SmtpTransport ──► your MTA
→ lifecycle: sent / deferred+backoff / dead-letter
Four seams keep it honest and testable:
MailProvider(provider.ts) — the one extension point every backend implements. The shipped default is our ownSmtpTransport(smtp.ts); the worker takes an injectable provider so the whole tick is unit-tested with no network.- The native queue (
store.ts) — the backbone's Postgres job pattern (claim viaFOR UPDATE SKIP LOCKED, exponential backoff, dead-letter), not Redis/BullMQ. - Dual backend — every store function runs against postgres.js when
DATABASE_URLis set, else an in-memory backend (mirrors the rest of the control plane). - DKIM at rest — private keys are sealed by the
cryptocore add-on (encryptSecret/decryptSecret) and only unsealed in memory at send time.
Engine (types → mime/dkim/verp/smtp)
types.ts—NormalizedMessageis the provider-agnostic shape the send API normalizes into.validateMessage()enforcesMAX_RECIPIENTS(1000),MAX_ATTACHMENTS(25), and at-least-one-body; theMAX_MESSAGE_BYTES(25 MiB) cap is enforced separately inmime.ts'sbuildMime()on the assembled raw message.addressOnly/domainOfare the address helpers.mime.ts— RFC 2822/MIME builder (header-injection-safe; boundary=_hs_; emitsDate+Message-ID).dkim.ts— purenode:cryptoRFC 6376 relaxed/relaxed signer + verifier (usermails delegated signing to nodemailer; this is net-new ours). Supportsrsa-sha256(RSA-2048, default) anded25519-sha256(RFC 8463).generateDkimKeypairself-checks that the public key derives from the private key.domainDnsRecords()produces the DKIM/SPF/DMARC records a customer publishes — see below.verp.ts—bounceReturnPath(emailId, domain)→bounce+<emailId>@<domain>on a domain you control;parseBounceRecipient()decodes the id back out of a DSN's envelope recipient (no body parsing). DMARC alignment is preserved because theFrom:header keeps the customer domain and is DKIM-signed with that domain's key.smtp.ts— pure-TS SMTP transport (EHLO/STARTTLS/AUTH/MAIL/RCPT/DATA) with IPv4-force, retry/backoff, and a circuit breaker. This is the defaultMailProvider.
DKIM / SPF / DMARC handling
Record generation is in dkim.ts → domainDnsRecords(domain, publicKey, opts):
| Record | Name | Value |
|---|---|---|
| DKIM | <selector>._domainkey.<domain> | v=DKIM1; k=<rsa|ed25519>; p=<pubkey> |
| SPF | <domain> | v=spf1 include:<spfInclude> ~all (or bare v=spf1 ~all) |
| DMARC | _dmarc.<domain> | v=DMARC1; p=none; rua=mailto:<dmarcRua> (or v=DMARC1; p=none;) |
Everything is caller-supplied (selector, spfInclude, dmarcRua, algorithm) — nothing
is hardcoded to a vendor. Default selector is hs.
Record verification and health live in the dns-integrity add-on, which the email
routes call:
verifyDomainAuth(domain, { selector, dkimPublicKey, spfInclude })(lib/addons/dns-integrity/verify.ts) — checks live DNS via the system resolver (no third-party lookup). Per-checkVerifyStatusisverified | failed | pending;overallisverifiedonly when DKIM and SPF pass. DKIMfailed= record present but key mismatch; SPFfailed= multiplev=spf1records or a missing include. DMARC is advisory — present →verified, elsepending; it never fails onboarding.analyzeMailHealth(domain, { dkimSelectors })(lib/addons/dns-integrity/health.ts) — read-only 0–100 deliverability score over MX / SPF (RFC 7208 10-lookup budget) / DMARC / DKIM, plus ariskscatalog.
API surface (/v1/email/*)
Two auth models coexist:
- Customer/tenant routes —
Authorization: Bearer hs_live_…, resolved byauthenticateApiKey()to anApiKeyPrincipal { keyId, tenantId, brandId }. Every query is tenant-scoped by that principal. - Operator routes (
/keys,/keys/{id},/alerts/run) — guarded by the platform credential (agentGate/ a cron secret), pending a dashboard-session gate once better-auth lands. The key store is auth-agnostic, so that swap is local to the route.
All routes are runtime = 'nodejs', dynamic = 'force-dynamic', and use the shared
errors / errorToResponse / json / parseBody helpers from @/lib/addons.
POST /v1/email/domains — register a sending domain
Auth: hs_ key. Generates a DKIM keypair (private key sealed via createDomain), persists
the domain (status: pending), and returns the records to publish.
Request:
{ "domain": "mail.example.com", "selector": "hs",
"algorithm": "rsa-sha256", "spfInclude": "spf.example.net",
"bounceDomain": "…", "dmarcRua": "dmarc@example.com" }
domain is required and regex-validated; algorithm must be rsa-sha256 or
ed25519-sha256; the rest are optional. Response 201:
{ "id": "emd_…", "domain": "mail.example.com", "status": "pending",
"records": [ { "type": "TXT", "name": "…", "value": "…", "purpose": "dkim|spf|dmarc" } ] }
GET /v1/email/domains — list this tenant's domains
Auth: hs_ key. Returns { domains: EmailDomain[] } scoped to the principal's tenant. Key
material is never included (the public EmailDomain view carries dkimPublicKey only).
POST /v1/email/domains/{id}/verify — confirm DNS is live
Auth: hs_ key (must own the domain, else 404). Calls verifyDomainAuth and, when
overall === 'verified', flips the domain to verified via setDomainStatus. Read-only
against DNS; never auto-fails a pending domain. Response:
{ "id": "emd_…", "status": "verified|pending|failed",
"verification": { "dkim": "…", "spf": "…", "dmarc": "…", "overall": "…", "details": [ … ] } }
GET /v1/email/domains/{id}/health — deliverability score
Auth: hs_ key (must own the domain, else 404). Calls
analyzeMailHealth(domain, { dkimSelectors: [domain.selector] }). Response:
{ id, health: { score, mx, spf, dmarc, dkim, senders, risks } }.
POST /v1/email/send — the send API (Resend-shaped)
Auth: hs_ key. Order of operations (app/api/v1/email/send/route.ts):
- IP rate-limit (300/min) before any DB work, then per-key rate-limit (120/min)
after auth — both via
lib/ratelimit→429withRetry-After. - Validate —
from,subject, and one ofhtml/textrequired; at least one recipient acrossto/cc/bcc. Both camelCase and snake_case (reply_to,content_type) accepted. - Attachments — each needs a
filename; body is base64contentor a publichttp(s)URL inpath(guarded byassertPublicUrlfrom thessrfadd-on). A server-local path is rejected (LFI/SSRF). - Verified-domain gate —
domainOf(from)must match astatus === 'verified'domain on the tenant, else403. - Suppression — rejected only if every recipient is suppressed (
validationerror). Per-recipient suppression happens later, in the worker. validateMessagestructural check, thenenqueueSend(transactional email row- send job).
Success: 202 { "id": "em_…", "status": "queued" }.
POST|GET /v1/email/keys and DELETE /v1/email/keys/{id} — operator key management
agentGate-guarded. POST mints a key and returns the plaintext token once
(createApiKey stores only the sha256; keyPrefix is the non-secret first 12 chars).
GET ?tenantId= lists the public view (no hashes). DELETE revokes (idempotent; 404 if
no active key). Tokens are hs_live_<base64url>; authenticateApiKey short-circuits
anything not starting hs_, matches by sha256, rejects revoked keys, and stamps
last_used_at.
POST|GET /v1/email/alerts/run — platform notification scheduler
Not a customer route: a cron entrypoint that queues license-expiry/dunning mail and drains
the queue once (runPlatformEmailNotifications). Authorized by
EMAIL_ALERTS_CRON_SECRET, falling back to MONITORS_CRON_SECRET (constant-time compare;
fail-closed when unset).
The send worker
runSendQueueOnce(opts) (worker.ts) is one drain tick — wire it to any scheduler
(mirrors the monitors cron). Per tick it:
claimDueJobs(batch, workerId, undefined, 'send')— claims onlysend-kind jobs.- For each job:
dropSuppressedfilters suppressed recipients per-tenant; if none remain, the email is markedfailed(all recipients suppressed) and the job completes. - If the job carries a
domainId,getDomainSigningKeyunseals the DKIM key in memory and attaches{ domainName, keySelector, privateKey, algorithm }to the message. provider.send(msg)delivers; on success → emailsent(+providerMessageId), jobsucceeded, asentevent recorded.- On error →
failJobapplies exponential backoff; a still-retryable job re-queues the email asqueued(deferredevent); an exhausted job dead-letters and marks the emailfailed. A single poisoned job never stalls the tick.
The default provider is resolveProvider() building a SmtpTransport from
HOSTSSH_SMTP_URL (and optional HOSTSSH_BOUNCE_DOMAIN); a fake provider is injected in
tests.
Where real outbound is gated
Live delivery is not enabled by default. The engine, queue, API, and worker all ship and are unit-tested, but a queued message only leaves the building when two things are true:
HOSTSSH_SMTP_URLis set to a real MTA —resolveProvider()throws without it ("HOSTSSH_SMTP_URL is not set — cannot deliver mail (set it, or pass a provider)."), so a worker tick can't deliver.- The Go agent MTA sidecar (the box with port-25 egress + PTR) is stood up to pull and
deliver send jobs — this is a remaining harvest phase (see
planning/HARVEST-BACKLOG.md, phase 4b), not yet wired.
Sending live mail is an irreversible, outward-facing action, so the trigger + sidecar are
wired deliberately, not by default. Until then, messages persist as queued and the
operator page's SMTP badge reads unset. The operator page also gates on
connections.manage and surfaces the HOSTSSH_PLATFORM_EMAIL_FROM /
HOSTSSH_PLATFORM_EMAIL_DOMAIN_ID platform-channel wiring.
Data model
schema-email.sql (migration 0012-email): email_domains (DKIM private key sealed at
rest), email_api_keys (sha256 hash + prefix), emails (message log), email_jobs
(the native queue), email_events, email_suppressions. The records.ts state machine
(nextBackoffMs, isValidJobTransition) is pure and unit-tested without a database.
Testing
Every module has a network-free *.test.ts alongside it: MIME header-injection + RFC 2047
- multipart; DKIM sign→verify roundtrip (RSA + Ed25519) + tamper detection + known
empty-body hash; VERP symmetry; an in-process loopback SMTP proving
build→sign→VERP→deliver; the queue state machine; API-key mint/authenticate/revoke; and the
worker tick via a fake provider. Route tests cover
domains,keys, andsend.
Extending
Add a delivery backend by implementing MailProvider (provider.ts) and passing it as
opts.provider to runSendQueueOnce (or setting it as the default). Connected-account
backends (Gmail/Graph) implement the same seam — the send API never changes. The barrel
(index.ts) is the single import surface for the whole add-on.
Related
- DNS Tools — the interactive SuperTool + scheduled monitors that share the SPF/DMARC/DKIM/blacklist toolkit this add-on's verification builds on.
- Sending email — the task-oriented walkthrough of this same flow.