Firecrawl

Firecrawl is a web scraping and crawling API built specifically for developers building AI applications. Created by the team at Mendable AI (YC W22), it solves a problem every LLM developer hits: the web contains most of human knowledge, but raw HTML is noisy, unstructured, and token-expensive to feed to a language model. Firecrawl converts any URL into clean Markdown, structured JSON, or screenshots in a single API call, handling JavaScript rendering, dynamic content, and document formats automatically. The open-source repository at github.com/mendableai/firecrawl has accumulated 113,000 GitHub stars, and over 80,000 companies use the hosted API as of April 2026.

The core API offers five endpoints: /scrape for single-page extraction, /crawl for recursive site crawling, /search for web search that returns full-page content rather than just result snippets, /extract for schema-driven or prompt-based structured data extraction, and /interact for AI-driven page automation. A newer /parse endpoint converts PDFs, Word documents, and spreadsheets into structured data for AI ingestion. SDKs are available for Python, Node.js, Go, Rust, Java, and Elixir. Native integrations exist for LangChain, LlamaIndex, and CrewAI, making it easy to drop into existing AI pipelines. Over 400,000 Model Context Protocol (MCP) servers have been installed using Firecrawl.

What Firecrawl actually does in April 2026

The hosted API runs on Mendable's Fire-Engine, a proprietary scraping infrastructure that auto-detects whether JavaScript rendering is required and applies ML-based extraction without requiring developers to write CSS selectors or XPath. P95 latency is 3.4 seconds across millions of pages. Coverage claims reach 96% of the web, including heavily JavaScript-dependent single-page applications.

The /extract endpoint is what separates Firecrawl from traditional scrapers. Pass a JSON schema and a URL, and the API returns structured data matching that schema. Pass just a plain-English prompt ("extract the author, publication date, and main claims from this article") and the API figures out the structure itself. This was introduced during Launch Week I in August 2024 as part of the v1 release, and it remains the most-cited reason developers choose Firecrawl over writing their own scraping layer.

The /crawl endpoint accepts depth limits, domain filters, include/exclude patterns, and webhook callbacks for async jobs, making it practical for building full knowledge bases from documentation sites. The v2 update (August 2025) added semantic crawling, where you describe a site in plain English and Firecrawl determines the relevant pages to follow, rather than crawling everything indiscriminately.

Firecrawl is available as a hosted API with a free tier (500 one-time credits) and a self-hostable open-source core under AGPL-3.0. The SDKs and certain UI components are MIT-licensed. Current version is v2.9.0, released April 10, 2026.

Where Firecrawl sits versus Browserbase and Apify

Browserbase is cloud headless-browser infrastructure. You get managed Chromium instances, write Playwright or Puppeteer automation scripts against them, and receive raw HTML back. The difference is fundamental: Browserbase gives you a controlled browser; Firecrawl gives you the extracted data. If your use case is complex authenticated multi-step flows with full Chrome DevTools Protocol access, session persistence, and custom automation logic, Browserbase is the right choice. If your use case is extracting content from a site and feeding it to an LLM, Firecrawl removes the entire parsing layer. Browserbase bills on a multi-dimensional model (browser-hours + proxy gigabytes + API calls), making cost prediction difficult at scale. Firecrawl bills 1 credit per standard page scrape. Teams pairing Firecrawl with n8n for workflow automation typically find the credit model easier to budget.

Apify is a scraping platform built around "Actors": self-contained cloud containers, each a program for scraping a specific site or task type. The marketplace has 10,000+ community and official Actors covering everything from LinkedIn to Google Shopping. Apify uses compute-unit billing (1 GB-hour of RAM), which makes costs unpredictable when JavaScript rendering is involved. Firecrawl's auto-detection handles JS rendering transparently under the same per-page credit model. The trade-off: Apify wins when you need a pre-built, maintained scraper for dozens of specific sites and don't want to write extraction logic. Firecrawl wins when you want a clean, unified API for arbitrary URLs with LLM-ready output and predictable pricing. Alex Reibman of AgentOps documented a representative result on X in 2025:

"Moved our internal agent's web scraping tool from Apify to Firecrawl because it benchmarked 50x faster with AgentOps." -- alexreibman, X, 2025

What the API workflow reality looks like

Most production Firecrawl integrations fall into three patterns. The first is the RAG knowledge-base build: crawl a documentation site or set of URLs, get back Markdown for each page, chunk and embed, store in a vector database. Because Firecrawl preserves heading structure in its Markdown output, the chunks are semantically coherent rather than arbitrary HTML fragments. Developers pairing Firecrawl with LlamaIndex or LangChain typically reduce their embedding preprocessing code to near zero. The AnythingLLM community has published several integration guides using Firecrawl as the ingestion layer.

The second pattern is real-time agent research: an agent calls /search to retrieve full-page content for web results (not just snippets), then calls /extract with a schema to pull structured signals. This powers competitive intelligence tools, lead enrichment pipelines, price monitoring, and deep research agents. The v2 semantic crawling feature fits naturally here, letting agents describe in natural language what content they want rather than specifying URLs.

The third pattern is document ingestion: finance, legal, and compliance teams pipe PDFs and Word documents through /parse to get structured JSON for downstream AI processing, bypassing the fragile PDF-parsing ecosystem. Firecrawl's /parse endpoint handles multi-column layouts, embedded tables, and footnotes better than naive PDF extraction libraries, which is the specific reason it gained traction in document-heavy enterprise workflows after the feature launched.

A fourth emerging pattern is competitive monitoring: teams set up recurring crawl jobs against competitor pricing pages, product feature lists, and job boards, feeding the extracted Markdown directly into a summarization step. The crawl webhook (introduced in v1, August 2024) fires when a job completes, making it straightforward to wire into a notification or analysis pipeline without polling.

The rough edges developers hit most often: credit costs multiply up to 9x per page when combining AI extraction with Enhanced Mode, and AI extraction runs on a separate token-based billing system. A developer on the Standard plan ($83/month for 100k credits) who also uses structured extraction is actually paying closer to $170/month once the extraction tier is added. This surprises developers who see "$83/month" and assume it covers all features. A user on Hacker News captured the frustration concisely:

"Firecrawl is egregiously expensive." -- nextworddev, Hacker News, 2025

Failed requests also consume credits. On sites with inconsistent availability or aggressive anti-bot measures, developers report burning 20-30% of credits on failures.

Who Firecrawl is built for

Firecrawl targets developers building AI-native applications who need web data as an input. The sweet spot is teams that want a maintained, reliable scraping layer without building and operating one themselves: startups building research assistants, agent frameworks, RAG-powered knowledge bases, and competitive intelligence tools. The LangChain, LlamaIndex, CrewAI, and MCP integrations mean it slots into existing AI stacks with minimal code. The open-source core means teams with strict data residency requirements can self-host, though with meaningful capability trade-offs (see below).

The Series A investor list reveals the use case better than any marketing copy: Shopify's CEO, Postman's CEO, and Mux's founder all backed the round alongside Nexus Venture Partners and Y Combinator. These are operators who run platforms where third-party developers need reliable, programmatic web access. Firecrawl fits the same profile: it is infrastructure for builders, not an end-user product.

Firecrawl is also a strong fit for developers evaluating it alongside Browserbase for agent automation needs. When the goal is data extraction rather than complex browser session management, Firecrawl's single-endpoint simplicity wins on development time. Developers building autonomous research agents often chain Firecrawl's /search and /extract endpoints directly, feeding the output to Claude, GPT-4o, or an open-weights model without any intermediate parsing step.

The 500-credit free tier is generous enough to prototype a real workflow, including a full crawl of a documentation site and embedding pipeline. The Hobby plan at $16/month (3,000 credits) covers personal projects and small applications. Teams doing production-scale crawling at 100,000+ pages monthly will be on the Standard plan ($83/month) or above, and should budget for extraction tier costs on top of that if structured output is part of their workflow.

What Firecrawl is not

Firecrawl is a tool for developers. There is no no-code interface, no visual workflow builder, no dashboard for non-technical users. Content strategists, marketers, and operations teams cannot use it directly without engineering support.

It is not a solution for scraping heavily protected sites at high reliability. Independent benchmarks show Firecrawl achieves roughly 33% success on Cloudflare-protected and WAF-guarded sites, ranking poorly compared to proxied services like Bright Data or Zyte. Amazon product pages, LinkedIn profiles, and sites with sophisticated fingerprinting are known failure cases. The proprietary Fire-Engine anti-bot system is cloud-only; self-hosted deployments have no anti-bot capabilities and must supply their own proxy infrastructure.

Self-hosting carries a specific warning: the gap between the cloud API and the self-hosted version has widened with each release as features migrate to cloud-only. The community maintains a fork called firecrawl-simple specifically to address this. If anti-bot bypass is a hard requirement for self-hosted deployment, Firecrawl is not the right choice.

Finally, it is not cost-competitive for very large-scale operations. At 10 million+ pages per month, the per-credit pricing makes custom infrastructure more economical. The developer who paid $190/month and found the experience "expensive and felt half baked" (Hacker News, 2024-2025) eventually replaced it with 2,700 lines of custom Elixir code. That is a reasonable inflection point for teams with engineering capacity to maintain their own stack. For everyone else, the alternative to Firecrawl is not usually custom code: it is a different managed service like Bright Data or Zyte for protected-site coverage, or Apify for multi-site Actor-based workflows. The correct question is not "Firecrawl or custom code" but "Firecrawl or which other managed layer," and for clean LLM-output at moderate scale with a simple API, Firecrawl wins that comparison consistently.