Show HN: Draco – A single-binary, self-hostable Firecrawl alternative in Rust

A fast, stealth, native-Rust web scraper — a lighter replacement to Firecrawl / Browserbase. Point it astatine a URL and get cleanable Markdown + metadata back, using a browser-faithful TLS/JA4 fingerprint to scope pages that artifact mean clients. No Node, nary headless-Chrome fleet, nary per-request browser boot.

The fastest measurement to instal Draco connected Linux aliases macOS is via the instal script:

(This will automatically observe your OS/architecture, download the latest binary, and adhd it to your ~/.zshrc, ~/.bashrc, aliases ~/.config/fish/config.fish)

Then effort scraping a page:

For a modular HTML page that's a azygous fingerprinted fetch + parse — typically ~300 ms, nary browser — and the Markdown pipeline mirrors Firecrawl's (deterministic main-content extraction + a Turndown/GFM-equivalent converter), implemented natively successful Rust. Client-rendered SPAs (whose contented only appears after JavaScript runs) are handled too, via render-then-Markdown — no headless browser fleet.

• markdown — the page's main contented arsenic cleanable Markdown: headings, links (absolutized), lists, blockquotes, fenced codification blocks (with language), and GFM tables. Boilerplate (nav / header / speech / footer / ads) is stripped, scripts and styles ne'er leak, base64 images are elided.
• metadata — title, description, language, canonical, favicon, every og:* / twitter:* / article:* tag, positive sourceURL, statusCode, contentType.
• trace + timing — precisely which steps ran and wherever the milliseconds went.

Build prerequisites (for wreq's BoringSSL + bindgen; and V8, only for the optional json mode): cmake, a C/C++ compiler, clang/libclang, perl, pkg-config.

• Debian/Ubuntu: apt instal build-essential cmake clang libclang-dev perl pkg-config
• Fedora: dnf instal gcc gcc-c++ cmake clang clang-devel llvm-devel perl pkgconf
• macOS: Xcode Command Line Tools + brew instal cmake

Exit codes: 0 occurrence · 1 correction · 2 unsupported · 3 needs_browser.

Beyond Markdown, Draco tin extract the structured information an SPA loads from its own API — a powerfulness characteristic for data-driven sites. It escalates done the cheapest tier that yields data:

1. Static embedded state — __NEXT_DATA__, JSON-LD, window.__NUXT__.
2. Next.js build-id replay — fetch /_next/data/<buildId>/…​.json directly.
3. Runtime interception — footwear an in-process V8 isolate (restored from a build-time DOM-engine snapshot successful single-digit milliseconds, JIT on), fto the page's JS hydrate, intercept the fetch/XHR it fires for its data, rank the intercepts, and replay the victor pinch the stealth client. The isolate is a discovery oracle, not a renderer — information requests are answered pinch a synthetic stub, and page JS has nary big bindings: it cannot execute I/O.

Flags: --format <markdown|html|raw-html|links|json|endpoints|both> (repeatable; default markdown; some = markdown+json), --json, --extract <JSONPATH>, --no-main-content, --wait-for <ms>, --tier-max <0|1|2>, --proxy, --delay <ms>, --timeout <ms>, --capture-window-ms <ms>, --ignore-robots, --allow-unsafe-replay, --runtime-log, --pretty. (--no-jail / --strict-sandbox are still accepted for compatibility and are inert — spot Security model.)

Debugging a page that hydrates to nothing? --runtime-log (also connected discover, and arsenic runtimeLog connected the daemon/MCP) surfaces the isolate's page-side diagnostics — swallowed exceptions, console.error lines, each brokered fetch ([raze.fetch] METHOD URL → STATUS (bytes, live|stub)), grounded chunk/module loads, and why/when the seizure model closed — each stamped [+ms] from capture start, truthful the trace sounds arsenic a timeline: browser-devtools visibility, nary browser.

Some pages render their content only aft JavaScript runs — the fetched HTML is a bladed ammunition (an quiet <div id="root">). Draco handles these automatically: when the first parse finds almost nary contented and Tier 2 is permitted (the default), it hydrates the ammunition successful the aforesaid in-process V8 isolate, serializes the live DOM, splices the shell's existent <head> (title / Open Graph / canonical) onto the hydrated <body>, and re-runs the nonstop aforesaid contented motor complete it. You get clean Markdown from a client-rendered page pinch nary headless browser — the trace shows a runtime.render measurement and source_tier: runtime_interception.

This besides covers skeleton screens: a page that ships tons of chrome but whose content rails are still Loading… is detected arsenic an incomplete render (regardless of length) and escalated the aforesaid way. Loading… placeholder lines are always stripped from the output, truthful that sound ne'er reaches you moreover if the render pass is capped (--tier-max 1) aliases can't amended the page.

Pure-CSR SPAs: unrecorded data, safely. Some SPAs vessel nary embedded authorities astatine each — the contented exists only down the JSON APIs the page calls aft it hydrates. For precisely this escalation (and only it), the isolate's fetch agent switches to Render mode: the page's safe information requests — GET/HEAD, and read-style POST/PUT (GraphQL / JSON-RPC-shaped) — are fetched live done the same stealth customer and shared cooky jar, and the page sees the real status/headers/body, including non-2xx, truthful a model router runs its native success/error paths. State-changing requests enactment stubbed unless --allow-unsafe-replay; streaming endpoints and analytics beacons are never fetched live. observe and the JSON tier support the record-and-stub Observe mode. The seizure model closes arsenic soon arsenic the page's content activity settles — analytics/session-replay beacons are recorded but cannot pin the model unfastened — with a difficult ceiling arsenic backstop.

External scripts & ES modules are handled too. The isolate runs a page's external <script src> and <script type="module"> (with import / dynamic import()), not conscionable inline scripts. Script subresources are fetched on demand and concurrently — chunk loads instrumentality retired connected the isolate's arena loop like a browser's web stack — done the pooled stealth customer and a process-global immutable chunk cache (512 MiB RAM + 2 GiB disk), truthful a hashed SPA chunk is fetched erstwhile crossed scrapes. Page JS itself still performs zero I/O: every byte is brokered by the engine's ops.

A bladed ammunition that can't beryllium improved (hydration adds nothing, aliases the isolate is unavailable) falls backmost to the fixed ammunition — ne'er a crash, ne'er a regression.

Run Draco arsenic a persistent HTTP daemon pinch a Firecrawl-compatible REST API — the process stays lukewarm (no per-scrape binary spawn), and existing Firecrawl clients tin constituent astatine it unchanged:

• formats: "markdown" (default) and/or "json" (the gradual JSON-API extraction, nether data.json). Formats Draco doesn't nutrient yet (html, rawHtml, links, screenshot) are rejected pinch a clear 400.
• Unknown Firecrawl fields (onlyMainContent, waitFor, …) are accepted and ignored; failures usage the { "success": false, "error": … } envelope (502 upstream/network, 422 unsupported target, 400 bad request).
• Draco extensions per request: tierMax, captureWindowMs, noJail, allowUnsafeReplay, ignoreRobots, proxy — positive timeout (Firecrawl's). Server-wide defaults travel from the draco service flags.
• Every consequence carries a draco entity (sourceTier, timing, trace) — the aforesaid honorable execution study arsenic the CLI envelope.
• GET /health → { "status": "ok", "version": … }.

Concurrency is bounded (--max-concurrency, default 8); excess requests queue. Warm-process SPA hydration answers successful ~150 sclerosis end-to-end connected the section benchmark fixture (fetch → hydrate → serialize → Markdown).

Isolate concurrency. Tier 2 scrapes tally successful fresh, in-process V8 isolates — 1 per job, restored from the build-time DOM snapshot in single-digit milliseconds, truthful location is nary per-request browser footwear to amortize and ne'er immoderate cross-scrape state, cookie, aliases DOM bleed. --isolate-pool-size bounds really galore isolates tally concurrently (default 0 = car ≈ CPU count); excess Tier 2 activity queues. (--isolate-max-jobs is accepted for compatibility and inert — location are nary long-lived workers to recycle.)

Beyond scraping, the daemon speaks 2 much Firecrawl endpoints:

• POST /v1/map — accelerated tract URL discovery: merges /sitemap.xml (sitemap indexes followed 1 level) pinch the page's ain links; same-host filtered (includeSubdomains opt-in), deduped, search-filtered, limit-capped.

  curl -X POST localhost:3002/v1/map -H 'content-type: application/json' \ -d '{"url": "https://docs.example.com", "search": "guide"}' # → { "success": true, "links": [ … ] }

• POST /v1/crawl — async crawl jobs: a bounded same-host BFS (limit default 10, headdress 100; maxDepth default 2; includePaths/excludePaths path filters) wherever each page runs the afloat extraction ladder — crawled SPAs hydrate for illustration azygous scrapes. Frontier links are harvested from each page's Markdown (already absolutized; JS-injected links included erstwhile the render escalation ran). Poll GET /v1/crawl/{id} for { status, total, completed, data: [ per-page results ] }; DELETE cancels. Jobs are in-memory and stock the daemon's concurrency budget. Status is paginated (?skip=&limit=, adjacent erstwhile much remains); GET /v1/crawl/{id}/errors lists per-page failures.

• POST /v1/batch/scrape — scrape a database of URLs arsenic 1 async job. Scrape options are flat astatine the apical level (formats, onlyMainContent, includeTags/excludeTags, headers, waitFor, …), applied to each URL; ignoreInvalidURLs drops non-http(s) URLs into an invalidURLs database instead of failing the request. URLs tally successful parallel, bounded by --max-concurrency.

  curl -X POST localhost:3002/v1/batch/scrape -H 'content-type: application/json' \ -d '{"urls": ["https://a.example", "https://b.example"], "formats": ["markdown"]}' # → { "success": true, "id": "7", "url": "/v1/batch/scrape/7" }

  Poll GET /v1/batch/scrape/{id} (paginated ?skip=&limit=, adjacent erstwhile more remains) for { status, total, completed, creditsUsed, expiresAt, next, information }; GET /v1/batch/scrape/{id}/errors lists per-URL failures; DELETE cancels.

• Webhooks — crawl and batch requests judge a webhook (a bare URL string or { url, headers, metadata, events }). The occupation fires started, page (with the scraped document), completed, and grounded events — payload { success, type, id, data, metadata }, type prefixed by occupation kind (crawl.page, batch_scrape.completed). Delivery is fire-and-forget pinch a 10s deadline and +1/+5/+15min retries; the endpoint is ne'er robots-gated.

  curl -X POST localhost:3002/v1/crawl -H 'content-type: application/json' \ -d '{"url": "https://site.example", "webhook": "https://my.app/hook"}'

Client-rendered pages load their contented from their ain JSON APIs. Draco's Tier 2 isolate already intercepts each fetch/XHR to prime a replay victor — discovery surfaces the full classed catalog truthful you tin spot (and replay) the APIs down a SPA:

Each endpoint carries a people (higher = much apt the existent information API) and a replayable emblem (clears the viability barroom and is replay-safe — 1 eligibility rule feeds some the catalog and the replay engine). Ranked best-first; the analytics beacons and fixed assets benignant to the bottom. On /v1/scrape, formats: ["endpoints"] returns the catalog nether data.endpoints and composes with markdown/json.

Search the web done Draco's own stealth HTTP stack — nary outer search API, and no rendering (SERP results are server-rendered HTML; booting the isolate for them would beryllium axenic waste). Several engines are queried in parallel and merged by reciprocal-rank consensus, truthful an motor that captcha-walls, geo-blocks, aliases rots is simply a normal partial nonaccomplishment the survivors absorb — the petition only fails if every motor does. (This is the SearXNG model — utilized arsenic a selector/behavior reference, not a dependency aliases a wholesale parser port.)

• Engine set: DuckDuckGo (HTML endpoint), Bing, Brave, Baidu, ZapMeta, Yandex — behind a swappable SearchEngine trait, each parser fixture-tested truthful parser drift needs nary unrecorded search. Google/Mojeek-class blocks are expected and tolerated.
• Request (Firecrawl-shaped; chartless fields accepted-and-ignored): query (required), limit (default 5, 1–100), tbs, location, timeout (default 60000), scrapeOptions, positive Draco proxy/ignoreRobots. Response is simply a flat data[] of { title, description, url }; full motor nonaccomplishment → 502, partial → statement of survivors.
• scrapeOptions.formats runs each consequence URL done the scrape ladder and merges the Document fields (markdown/html/rawHtml/links/json/ metadata) onto the deed — aforesaid FormatSet, bounded concurrency.

Drive a page for illustration a devtools console, nary browser: unfastened a stateful session, then tally JS successful page scope, publication the returned worth + console, click/type via selectors, and navigate — pinch cookies persisted for the full session (a Set-Cookie connected 1 page rides to the next, truthful multi-page and login flows work). It's the DOM-only analog of Firecrawl's browser actions: querying contented and clicking a nexus needs a DOM + JS runtime + a cookie-carrying web stack, all of which Draco already has — not a renderer. The isolate keeps its no-host- bindings containment; exec runs arbitrary page JS but its only I/O is the fetches the motor brokers.

• exec — the move is an async usability assemblage (may await, returns a value); the worth is serialized nether a fund (DOM nodes/functions described, over-budget → a truncation descriptor unless full/maxBytes). navigate — fetch the adjacent archive (cookie-aware) and re-hydrate successful place; "click a link by selector" is exec("return el.href") → navigate(href). scrape — Markdown/HTML/links of the unrecorded DOM. DELETE closes; idle sessions are reaped.
• Sessions are held in-memory connected the daemon, concurrency-capped and idle-reaped. The interact aboveground needs Tier 2 (V8); a thin service build omits it.

Draco's scraping is disposable arsenic Model Context Protocol tools for agent clients (Claude Desktop/Code, editors, orchestrators):

The aforesaid server is bound connected the daemon astatine POST /mcp (minimal Streamable-HTTP subset: single-message POST → azygous JSON response, 202 for notifications). Three tools, each annotated read-only:

• draco_scrape (url, formats: ["markdown"|"json"|"endpoints"], tierMax, captureWindowMs, timeout, ignoreRobots) — scrape to Markdown/JSON.
• draco_discover (url, tierMax, captureWindowMs, timeout, ignoreRobots, allowUnsafeReplay) — the classed API-endpoint catalog + the replayed winner, for agents that want a page's information API.
• draco_search (query, limit, tbs, location, timeout, formats) — parallel multi-engine web hunt merged by reciprocal-rank consensus; with formats it besides scrapes each consequence and merges the contented onto the hit.
• draco_interact_open/exec/navigate/scrape/close — a stateful page session an supplier drives crossed calls (open → tally JS / click / navigate → read → close), cookies persisted for the session. Advertised only connected the daemon (POST /mcp), wherever sessions are held; requires Tier 2.

Tool-level failures travel backmost arsenic isError results the exemplary tin respond to; protocol misuse is simply a due JSON-RPC error.

• default (tier2, serve) — everything: the V8 isolate for --format json runtime interception / render-then-Markdown, positive the draco service daemon.
• serve — the persistent HTTP daemon (axum). Independent of tier2: --no-default-features --features service exposes the aforesaid REST API pinch the ladder capped astatine the fixed tiers.
• --no-default-features — a thin build pinch no V8/axum linked. Markdown scraping and static/build-id JSON extraction still work; runtime interception reports unsupported. Smaller binary, faster build.

Markdown scraping of a fixed page executes nary page JavaScript. Tier 2 (runtime interception / render-then-Markdown) does, truthful containment matters. Draco's containment is the V8 isolate itself: the discourse has no host-capability bindings — the only ops exposed to page JS grounds an intercepted request, load a book chunk, log a diagnostic, sleep, and resoluteness URLs. There is nary network, filesystem, aliases process access; the only I/O page JS tin cause is the fetches the motor explicitly brokers (script subresources always; information requests only in Render mode, nether the mutation-safety policy). This is the aforesaid people of isolation Puppeteer/Playwright/jsdom trust on, useful identically connected macOS and Linux, and needs zero configuration. JIT is connected (--single-threaded, truthful V8 spawns nary inheritance threads); the achieved posture shows successful the trace arsenic a runtime.sandbox measurement (isolate: in-process v8 (no big bindings)).

The OS process jailhouse of earlier releases (fork + userns/netns air-gap + seccomp + Landlock) was retired successful v0.14: its per-chunk blocking IPC was the engine's throughput ceiling, and for hosted deployments the information perimeter belongs to the infrastructure furniture (stateless, ephemeral, unidirectional workers). --no-jail and --strict-sandbox are accepted for CLI compatibility and are inert.

Draco does not conclusion JS situation walls (Cloudflare/DataDome/…); a genuine interstitial (blocking position + existent situation page) short-circuits to needs_browser. A normal 200 down a CDN is ne'er treated arsenic a challenge.

Both are first-class — Tier 2 is the aforesaid in-process isolate connected both, with identical behaviour and zero platform-specific configuration.

On a memory-/disk-constrained container (CI containers, the ~4 GiB build sandbox), run the gates done the guarded wrapper alternatively — it pins single-job builds and refuses to commencement erstwhile disk is debased alternatively than filling it and losing the session:

See docs/SANDBOX.md for the afloat runbook (and the commit-before-build norm that makes a mislaid sandbox costs nothing).

Draco is for nationalist data, properties you operate, and APIs you're permitted to use. Defaults are polite (robots.txt respected, per-host complaint limiting, bounded retries). JA4/TLS emulation is for compatibility, not to conclusion authentication or access controls. You are responsible for compliance pinch target sites' Terms of Service and applicable law.

MIT OR Apache-2.0.