███████╗██╗██████╗ ███████╗ ██████╗██████╗  █████╗ ██╗    ██╗██╗
██╔════╝██║██╔══██╗██╔════╝██╔════╝██╔══██╗██╔══██╗██║    ██║██║
█████╗  ██║██████╔╝█████╗  ██║     ██████╔╝███████║██║ █╗ ██║██║
██╔══╝  ██║██╔══██╗██╔══╝  ██║     ██╔══██╗██╔══██║██║███╗██║██║
██║     ██║██║  ██║███████╗╚██████╗██║  ██║██║  ██║╚███╔███╔╝███████╗
╚═╝     ╚═╝╚═╝  ╚═╝╚══════╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝ ╚══╝╚══╝ ╚══════╝

Firecrawl Integration Guide

Focus: Turning any URL — or a web search — into clean, LLM-ready markdown or schema-validated JSON, with proxies, anti-bot, and JS rendering handled for you. The managed alternative to hand-rolled Playwright/Puppeteer for RAG and agents.

Verification note (2026-06-11): Every endpoint path, SDK method name, credit cost, price, and rate-limit number below was live-verified against docs.firecrawl.dev. Firecrawl is on the v2 API; the SDK package names changed from the older @mendable/firecrawl-js to plain firecrawl (Node) and firecrawl-py (Python). The one item that shifts most often is pricing — the per-plan dollar figures reflect the annual-billing effective monthly rate shown on the pricing page; monthly-billed prices are higher. Re-check firecrawl.dev/pricing before quoting numbers to a client.

Overview

Firecrawl is a web data API for AI. You hand it a URL and it returns the page as clean markdown, raw/processed HTML, a screenshot, a link list, or structured JSON — having already solved the parts that make DIY scraping painful: rotating proxies, anti-bot challenges, JavaScript/SPA rendering, PDFs, and dynamic content. It exposes five core primitives plus interactive browser control, all behind one API key.

For codeAmani, Firecrawl is the "web → context" lever: it's how you feed external pages into a RAG pipeline (pairs with the pinecone / pgvector guides), how an agent reads a page it was asked about, and how you pull structured facts (prices, listings, docs) off sites that have no API.

When Firecrawl wins — and when raw Playwright/Cheerio wins

Rule of thumb: Firecrawl converts the web into LLM input; Playwright/Cheerio give you a browser/parser you operate yourself. Firecrawl even offers a Browser Sandbox (managed Playwright-over-CDP) when you do need raw browser control without running the infra.

Primary use cases

• RAG ingestion — crawl a docs site or map→scrape selected pages into markdown, chunk, embed, store in Pinecone/pgvector.
• Agent web-browsing — give an agent scrape/search so it can read live pages; available as an MCP server and AI-SDK tools.
• Structured data extraction — extract (or the json format) turns a schema + prompt into validated JSON from one or many URLs.

Official Documentation

Setup

Get an API key at firecrawl.dev/app/api-keys (keys are prefixed fc-). Both SDKs read FIRECRAWL_API_KEY from the environment automatically, or you can pass it explicitly.

Node / TypeScript

npm install firecrawl

import { Firecrawl } from "firecrawl";

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

// Single page → clean markdown
const doc = await firecrawl.scrape("https://firecrawl.dev", {
  formats: ["markdown"],
  onlyMainContent: true,
});
console.log(doc.markdown);

Python

pip install firecrawl-py

from firecrawl import Firecrawl  # AsyncFirecrawl is also exported

firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")  # or omit to read FIRECRAWL_API_KEY

doc = firecrawl.scrape("https://firecrawl.dev", formats=["markdown"], only_main_content=True)
print(doc.markdown)

Node uses camelCase option keys (onlyMainContent, scrapeOptions); Python uses snake_case (only_main_content, scrape_options). The REST API itself is camelCase.

Core Endpoints

All endpoints live under https://api.firecrawl.dev/v2/ and authenticate with Authorization: Bearer fc-....

/scrape — single URL → markdown / HTML / JSON / screenshot / links

const doc = await firecrawl.scrape("https://example.com/article", {
  formats: ["markdown", "html", "links", "screenshot"],
  onlyMainContent: true,          // strip nav/footer/ads — fewer tokens
  includeTags: ["article", "main"],
  excludeTags: ["nav", "footer", ".ad"],
  maxAge: 600000,                 // serve from cache if scraped < 10 min ago
});

Response (SDKs return the data object directly; cURL wraps it in { success, data }):

{
  "markdown": "# Article title\n\nClean body text…",
  "html": "<!DOCTYPE html>…",
  "links": ["https://example.com/next", "…"],
  "screenshot": "https://…/screenshot.png",
  "metadata": {
    "title": "Article title",
    "sourceURL": "https://example.com/article",
    "statusCode": 200,
    "scrapeId": "019eb884-…"
  }
}

Output formats: markdown, html, rawHtml, links, screenshot, summary, json, changeTracking, branding. Use onlyMainContent: true plus includeTags/excludeTags to cut boilerplate (and token count) before it ever reaches your LLM.

Actions (dynamic pages — click, scroll, wait, input)

Pass an actions array to drive a real browser before extraction. v2 action shape (verified):

const doc = await firecrawl.scrape("https://example.com", {
  actions: [
    { type: "wait", milliseconds: 1000 },
    { type: "click", selector: "#accept" },
    { type: "scroll", direction: "down" },
    { type: "click", selector: "#q" },
    { type: "write", text: "firecrawl" },   // text input
    { type: "press", key: "Enter" },
    { type: "wait", milliseconds: 2000 },
    { type: "screenshot" },
  ],
  formats: ["markdown"],
});

Action types: wait (milliseconds), click (selector), scroll (direction), write (text), press (key), screenshot. For a persistent interactive session, use interact(scrapeId, …) / stopInteraction(scrapeId) against a prior scrape's metadata.scrapeId, or the Browser Sandbox (firecrawl.browser(...) → CDP URL for full Playwright).

/map — fast sitemap / URL discovery

const res = await firecrawl.map("https://docs.firecrawl.dev", {
  search: "webhook",   // optional: rank URLs by relevance
  limit: 100,
});
console.log(res.links); // ["https://docs.firecrawl.dev/webhooks/overview", …]

map is the cheap reconnaissance step: discover the URL list first, decide what's worth scraping, then crawl/scrape only those — instead of crawling blind.

/crawl — recursive site crawl (async job + status polling)

Crawl-and-wait (handles the job + pagination for you — recommended):

const job = await firecrawl.crawl("https://docs.firecrawl.dev", {
  limit: 100,                       // default is 10,000 — always set a limit
  includePaths: ["^/features/.*"],  // regex on pathname
  excludePaths: ["^/blog/.*"],
  maxDiscoveryDepth: 3,
  sitemap: "include",               // "include" | "skip" | "only"
  scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
});
console.log(job.status, job.data.length); // "completed", N docs

Start-and-poll (long crawls / custom polling):

const { id } = await firecrawl.startCrawl("https://docs.firecrawl.dev", { limit: 500 });
const status = await firecrawl.getCrawlStatus(id);
// status.status ∈ "scraping" | "completed" | "failed"; status.completed / status.total
// status.data = pages scraped so far; cancel with firecrawl.cancelCrawl(id)

Python mirrors this: firecrawl.crawl(url, limit=…, scrape_options=ScrapeOptions(...)), firecrawl.start_crawl(...), firecrawl.get_crawl_status(job.id).

Crawl gotchas (verified): default limit is 10,000 and the endpoint pre-checks that your credit balance covers it — set a real limit or you'll hit 402 Payment Required. By default crawl only follows children of the start path; use crawlEntireDomain, allowSubdomains, or allowExternalLinks to widen. Job results are retrievable via the API for 24 hours; after that, use the activity logs. data holds pages Firecrawl successfully scraped (even if the site returned 404) — fetch hard failures via the Get Crawl Errors endpoint (GET /crawl/{id}/errors).

/search — web search → scraped results

const results = await firecrawl.search("best dash cams 2026", {
  limit: 5,
  sources: ["web", "news", "images"],
  tbs: "qdr:m",                              // time filter: past month
  scrapeOptions: { formats: ["markdown"] },  // scrape each result inline
});
// results.web[] = { url, title, description, position, (markdown if scraped) }

One call searches the web and returns full page content for each hit — no separate scrape loop.

/extract — LLM structured extraction (schema → JSON)

const res = await firecrawl.extract({
  urls: ["https://example-forum.com/topic/123"],
  prompt: "Extract all user comments from this thread.",
  schema: {
    type: "object",
    properties: {
      comments: {
        type: "array",
        items: {
          type: "object",
          properties: { author: { type: "string" }, comment_text: { type: "string" } },
          required: ["author", "comment_text"],
        },
      },
    },
    required: ["comments"],
  },
});
console.log(res.data);

from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: str

data = firecrawl.extract(
    urls=["https://shop.example.com/item/42"],
    prompt="Extract the product name and price.",
    schema=Product,                 # a Pydantic model or a raw JSON Schema
    enable_web_search=True,         # optionally enrich from related pages
)
print(data.data)

• Accepts a JSON Schema or a Pydantic/Zod model; omit the schema for prompt-only freeform extraction.
• Wildcard URLs (https://docs.example.com/*) extract across a whole section.
• Async variant: startExtract(...) / start_extract(...) → getExtractStatus(id) / get_extract_status(id).
• For agentic, multi-step extraction add agent: { model: "FIRE-1" }.
• Alternative: for a single page you can skip /extract and request the json format on scrape with a { type: "json", schema } format object — cheaper and synchronous.

Caching, proxies/stealth, PDFs & dynamic content

• Caching — maxAge (ms) serves a recent cached scrape instead of re-fetching; set storeInCache / maxAge: 0 to force fresh. Big cost saver on re-runs.
• Proxies / stealth — proxy: "basic" | "stealth" | "auto". Stealth/enhanced proxy beats stubborn anti-bot but adds 4 credits/page.
• PDFs — handled natively; PDF parsing costs 1 credit per PDF page.
• Dynamic content — JS rendering is on by default; add waitFor (ms) or actions for slow SPAs.

Developer Resources

Official SDKs

• Node / TypeScript — npm install firecrawl → import { Firecrawl } from "firecrawl" (minimal snippet above).
• Python — pip install firecrawl-py → from firecrawl import Firecrawl (sync) or AsyncFirecrawl (async). Also official Go and Rust SDKs, a CLI (npx firecrawl-cli), and community SDKs.

Framework integrations

• LangChain — FireCrawlLoader (document loader) in langchain-community.
• LlamaIndex — FireCrawlWebReader.
• Vercel AI SDK — firecrawl-aisdk exports ready-made tools (search, scrape, map, crawl, batchScrape, agent, interact) plus poll/status/cancel — drop straight into a tool-calling agent.
• Also: OpenAI, CrewAI, Dify, n8n, Zapier, and more.

MCP server (yes — first-class)

Firecrawl ships an official MCP server so Claude, Cursor, Windsurf, and VS Code can call scrape/search/crawl/etc. directly. Connect via the hosted endpoint or run it locally — see docs.firecrawl.dev/mcp-server. (This guide's research was done through that exact MCP server.)

Self-hosting / open source

Firecrawl is open source (github.com/firecrawl/firecrawl, AGPL-licensed) and self-hostable — run the API on your own infra for data-residency or cost control. The hosted cloud adds managed proxies, anti-bot, scale, and higher reliability; the self-host build asks you to bring your own proxy/anti-bot. Decision guide: docs.firecrawl.dev/contributing/open-source-or-cloud.

Rate limits & concurrency (verified 2026-06-11)

Two independent limits; exceeding either returns 429:

API rate limits (requests/min, current plans):

Concurrent browsers (parallel jobs ceiling): Free 2, Hobby 5, Standard 50, Growth 100, Scale 150+. Jobs beyond the ceiling queue (and queue time counts against the request timeout; queued jobs expire after 48h). Check live headroom with the Queue Status endpoint.

Firecrawl's own guidance: rate limits exist mainly to prevent abuse — your real bottleneck is concurrent browsers, so size the plan by concurrency, not req/min.

Retry / backoff

The SDKs auto-retry and handle async polling. For your own loops, treat 429 and 5xx as retryable with exponential backoff + jitter; for 429, prefer reducing concurrency over hammering. 402 means out of credits (raise limit awareness or enable auto-recharge), not a transient error.

Webhooks for async crawl completion (verified)

Attach a webhook object to a crawl to get pushed events instead of polling:

// POST https://api.firecrawl.dev/v2/crawl
{
  "url": "https://docs.firecrawl.dev",
  "limit": 100,
  "webhook": {
    "url": "https://your-domain.com/api/webhooks/firecrawl",
    "metadata": { "tenant": "acme" },
    "events": ["started", "page", "completed"]
  }
}

Event types: crawl.started, crawl.page, crawl.completed, crawl.failed. Every request carries an X-Firecrawl-Signature header (sha256=…) — an HMAC-SHA256 of the raw body using your webhook secret (from the dashboard Advanced tab). Verify it with a timing-safe compare before processing — see codeAmani notes below.

Pricing (verify live before quoting)

These change — re-check firecrawl.dev/pricing. Dollar figures below are the annual-billing effective monthly rate shown on the pricing page on 2026-06-11; monthly-billed is higher. Verified live.

Credit-per-action model (verified)

So a plain scrape or crawl page is cheap (1 credit); turning on structured JSON or stealth proxy roughly 5×'s the per-page cost. Budget accordingly.

Overage behavior

No pure pay-as-you-go. Auto-recharge can auto-purchase additional credit packs when you dip below a threshold (larger packs = better rate). Credits do not roll over month-to-month; credit packs have their own billing periods. Downgrades take effect at the next renewal.

Usage Monitoring

• Dashboard — credit usage, activity logs (firecrawl.dev/app/logs), and per-key usage live in the app.
• Programmatic credit check — the API exposes a credit usage / token usage endpoint (GET /v2/team/credit-usage) so you can read remaining credits before launching a big crawl. Every scrape response also reports metadata.creditsUsed and concurrencyLimited.
• Job status — poll getCrawlStatus(id) / getExtractStatus(id) (completed / total / creditsUsed), or use Queue Status for live concurrency headroom.
• Avoiding surprise overages — set conservative limits, set an auto-recharge cap, prefer webhooks over tight polling loops, and gate large crawls behind a credit-usage pre-check in code.

codeAmani Notes

• Secrets server-side only. FIRECRAWL_API_KEY lives in .env.local (and Vercel env vars) and is used only from API routes / server actions / edge functions — never shipped to the browser. Treat the key like any other server secret; it's already covered by the repo's gitleaks pre-push gate.

• Verify webhook signatures (don't skip). Per CLAUDE.md's webhook rule, validate the X-Firecrawl-Signature HMAC before trusting any crawl callback:

  TypeScript Copy

  // app/api/webhooks/firecrawl/route.ts
  import crypto from "node:crypto";
  import { NextRequest } from "next/server";
  
  export async function POST(req: NextRequest) {
    const raw = await req.text(); // raw body — verify BEFORE JSON.parse
    const sig = req.headers.get("x-firecrawl-signature") ?? "";
    const expected =
      "sha256=" +
      crypto
        .createHmac("sha256", process.env.FIRECRAWL_WEBHOOK_SECRET!)
        .update(raw)
        .digest("hex");
  
    // timing-safe compare; bail if lengths differ
    const ok =
      sig.length === expected.length &&
      crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected));
    if (!ok) return new Response("invalid signature", { status: 401 });
  
    const event = JSON.parse(raw);
    // … handle crawl.page / crawl.completed
    return new Response("ok");
  }

• Cost-aware patterns for low-bandwidth / African-market use:

  • map before crawl — discover URLs cheaply, scope to exactly the pages you need, then scrape only those. Blind crawls burn credits and pull pages your users on 2G/3G never asked for.
  • Cache aggressively — set a generous maxAge so re-ingestion of slow-moving docs serves from cache (1 credit saved per page, and faster).
  • onlyMainContent: true + includeTags/excludeTags — strip nav/ads/footer so the LLM sees only the article. Fewer tokens = lower Anthropic/OpenAI bills and faster responses for bandwidth-constrained users.
  • Avoid the 5× multipliers unless needed — JSON mode and stealth proxy each add 4 credits/page. Prefer plain markdown scrape + a cheap local parse, or the single-page json format over a multi-URL /extract, when you can.
  • Cap the blast radius — always pass an explicit limit (the 10,000 default plus the credit pre-check can 402 a whole crawl); set an auto-recharge ceiling so a runaway job can't drain the account.

• TypeScript-first, per CLAUDE.md. Wrap Firecrawl in a lib/firecrawl.ts with named exports and typed helpers, async/await, and structured logging (no console.log in production):

  TypeScript Copy

  // lib/firecrawl.ts
  import { Firecrawl } from "firecrawl";
  
  const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
  
  export async function scrapeToMarkdown(url: string): Promise<string> {
    const doc = await firecrawl.scrape(url, {
      formats: ["markdown"],
      onlyMainContent: true,
      maxAge: 600_000, // 10-min cache window
    });
    return doc.markdown ?? "";
  }
  
  export async function mapSite(url: string, search?: string) {
    const res = await firecrawl.map(url, { search, limit: 100 });
    return res.links;
  }

• AI routing fit. Firecrawl is retrieval, not generation — it's the ingestion edge of a RAG stack. Pattern: map/crawl → markdown → chunk → embed → Pinecone / pgvector → answer with Anthropic Claude. It complements, not replaces, the AI providers in the routing table.

Troubleshooting

Verification Status (2026-06-11)

Live-verified against docs.firecrawl.dev (via Firecrawl's own scrape API + Context7):

• ✅ Endpoints & signatures — /scrape, /crawl (+ startCrawl/getCrawlStatus/cancelCrawl), /map, /search, /extract (+ startExtract/getExtractStatus), actions shape, webhooks.
• ✅ SDK packages & methods — Node firecrawl / Python firecrawl-py, v2 method names confirmed from the live SDK pages.
• ✅ Credit model — per-action costs and the +4 JSON/stealth and PDF-per-page multipliers, from the live pricing + crawl pages.
• ✅ Rate limits & concurrency — req/min table and concurrent-browser ceilings from the live rate-limits page.
• ⚠️ Pricing dollar figures — verified live, but they reflect annual-billing effective monthly rates and Firecrawl changes pricing periodically; re-confirm at firecrawl.dev/pricing before client-facing quotes.
• ⚠️ GET /v2/team/credit-usage path — the credit-usage/token-usage endpoint exists and is documented under the API reference; confirm the exact path/casing in the API reference for your SDK version.