Add-on — Scraper & Browser

Two related add-ons that share the same extraction contract. The scraper runs deterministic, DOM-free extraction over raw HTML; the browser add-on renders JS pages in a real browser and lights up the DOM-typed strategies the scraper leaves as a seam. Both keep the heavy runtime out of the control-plane bundle — the scraper never needs one; the browser add-on reaches Playwright over HTTP through a services/browser-worker service.

	Scraper	Browser
id / entitlement	scraper (business+)	browser (business+)
Origin	pyrosync	external — patterns from MIT browser-use, code is ours
Code	lib/addons/scraper/	lib/addons/browser/
Reuses	ssrf safeFetch	ssrf assertFetchableUrl + the whole scraper add-on

Scraper

scrape(url, rules, options?) (scrape.ts) = SSRF-safe fetch → extract. Every fetch rides the ssrf add-on's safeFetch (resolve-and-pin + manual redirect re-validation), so a tenant can never aim it at internal services. The body is capped at DEFAULT_MAX_BYTES = 5 MiB (overridable via maxBytes).

import { scrape } from '@/lib/addons/scraper'

const result = await scrape('https://shop.example/p/1', [
  { fieldName: 'price', type: 'regex', selector: '\\$[0-9.]+', transform: 'number' },
  { fieldName: 'product', type: 'jsonld', selector: 'Product' },
])
// → { url, finalUrl, status, data: { price: 19.99, product: [...] }, fetchedAt }

Rules & strategies

An ExtractionRule (types.ts): { fieldName, type, selector, attribute?, transform?, transformOptions?, isRequired?, defaultValue?, multiple?, order? }.

Strategy type	Status
regex, jsonld	Real — run on raw HTML, no DOM (extractByRegex, extractJsonLd in strategies.ts). The regex loop has a zero-width-match guard.
css, xpath, text, attribute	Delegated to an injected DomExtractor — a browser/agent worker with a real DOM.
custom_js, llm	Not supported — eval is dropped for safety; LLM extraction waits for the AI-gateway slice.

The DomExtractor interface (extractCss/extractXpath/extractText/extractAttribute) has a throwing default noDomExtractor — a DOM-typed rule with no extractor injected raises DomExtractorUnavailableError. The ExtractionEngine sorts rules by order, and isolates each rule's failure: one bad selector can't sink the page — an optional rule is skipped, a required rule falls back to defaultValue/null, and a failed transform keeps the untransformed value.

Transforms (transforms.ts) are pure: trim, lowercase, uppercase, number, date, regex_replace, template. Non-string values (arrays from jsonld/multiple) pass through untransformed.

ScrapeResult = { url, finalUrl, status, data, fetchedAt }.

Browser

The browser add-on mirrors the email add-on's posture — a pure core ships now, the heavy runtime is a gated binding. Playwright + Chromium never enter the control-plane bundle; they live in services/browser-worker (an OCI image) reached over HTTP through the BrowserDriver seam. Every piece here is deterministic and unit-tested with a FakeDriver + scriptedPlanner (no network, no browser, no API key).

Env var	Purpose
HOSTSSH_BROWSER_WORKER_URL	the worker base URL. Unset → the driver throws (capability dormant).
HOSTSSH_BROWSER_WORKER_TOKEN	bearer token for the worker.
ANTHROPIC_API_KEY	enables the autonomous agent's planner. Unset → the planner throws.

The indexed-DOM snapshot (the harvested core)

snapshot.ts is pure TS: raw DOM (RawNode[]) → a PageSnapshot of index-ordered interactive IndexedElements plus a compact textDigest (capped at MAX_TEXT_DIGEST = 2000 chars). serializeForPlanner(snapshot) renders it as the "DOM-as-text" the planner reads:

URL: <url>
TITLE: <title>

INTERACTIVE ELEMENTS:
  [<index>] <<role>/<type>> (editable) "<accessible name>"
  ...

PAGE TEXT:
<textDigest>

The driver seam

driver.ts. BrowserDriver = goto/observe/act/content/extract/close. The throwing default noBrowserDriver raises BrowserDriverUnavailableError until a real driver is injected. RemoteDriver is an HTTP client to the worker — it POSTs to /goto, /observe, /act, /content, /extract, /close, sends Authorization: Bearer <token> when configured, and times out at 45 s per request. driverFromEnv() returns a RemoteDriver when HOSTSSH_BROWSER_WORKER_URL is set, else noBrowserDriver.

Rendered scrape

browserScrape(url, rules, options?) (scrape.ts) re-validates the URL (SSRF), renders the page (JS executed), then resolves css/xpath/text/attribute in the live DOM via driver.extract() while reusing the scraper add-on's regex/jsonld + transforms. It always closes the session and returns the scraper's ScrapeResult. This fulfils the scraper's DomExtractor promise.

The observe→plan→act agent

runBrowserAgent(task, options?) (agent.ts) loops: observe (snapshot) → plan (a Planner decides one action) → act. The default planner is createAnthropicPlanner() (model claude-opus-4-8 via the existing @anthropic-ai/sdk, gated on ANTHROPIC_API_KEY); scriptedPlanner drives tests. Guards: DEFAULT_MAX_STEPS = 15, a no-progress guard that stops after DEFAULT_NO_PROGRESS = 3 identical (url, action) repeats, an AbortSignal check each step, and SSRF re-validation on every navigation.

Actions (BrowserAction): goto, click, type (with optional submit), press, scroll, wait, back, extract. AgentResult = { goal, success, answer?, steps, stop, error? } where stop ∈ done | max_steps | no_progress | error.

Rendered-page monitor

runBrowserMonitor(target, params?, options?) (monitor.ts) renders the page and returns a SuperTool ToolResult — this is the browser monitor type wired into lib/dns/monitors (see DNS Tools). Params: selector (must resolve to ≥1 element), text (must appear, case-insensitive), noConsoleErrors ('true' fails on any console error), maxLoadMs (warn near budget / fail over). Always closes the session.

HTTP routes

Both routes are license-bearer gated (agentGate) and requireAddon(key, 'browser') entitlement-gated. They return 503 (not 500) when the runtime isn't configured, so callers can tell "not enabled" from "failed".

Method · Path	Auth	Request	Response
POST /api/v1/browser/scrape	license bearer + browser add-on	{ url, rules[1..100], waitUntil?, timeoutMs?≤120000 }	ScrapeResult; 503 if BrowserDriverUnavailableError
POST /api/v1/browser/agent	license bearer + browser add-on (tight rate limit)	{ goal (≤2000), startUrl, maxSteps?≤50 }	AgentResult; 503 when the runtime/planner is unconfigured

browser/scrape/route.ts · browser/agent/route.ts. The scraper add-on has no dedicated /v1/scraper route of its own — it's consumed in-process and through the browser scrape route.

Honest status notes

• The browser runtime is dormant unless configured. With HOSTSSH_BROWSER_WORKER_URL unset, browserScrape throws BrowserDriverUnavailableError and the runBrowserAgent loop returns stop: 'error'; the two HTTP routes surface these as 503. runBrowserMonitor instead catches the error and returns a failed ToolResult (ok: false) — it is consumed by the monitor runner (lib/dns/monitors/runner.ts), not exposed as an HTTP route. The services/browser-worker image is the gated Playwright runtime.
• The agent needs ANTHROPIC_API_KEY. Without it the default planner throws.
• The scraper's real DomExtractor is the browser add-on (Slice 2). The repair engine (confidence / self-healing / drift-detection) and AI-assisted rule building are later slices; see planning/HARVEST-BACKLOG.md.
• No upstream (browser-use / Raccoon) code was copied and no upstream name appears in any shipping identifier.