Firecrawl

Use one API surface for search results, individual pages, and full-site crawling

Firecrawl provides Search, Scrape, Crawl, Map, and Batch Scrape endpoints for retrieving search-result content, converting individual URLs, crawling a site, discovering URLs, and processing large URL sets asynchronously. Outputs include LLM-friendly Markdown as well as HTML, screenshots, and structured JSON, and the README also documents parsing for web-hosted PDFs, DOCX files, and other media. This makes it useful as a shared ingestion layer for RAG, competitive research, and knowledge collection instead of rebuilding extraction logic for each workflow.

Sources: [1]

Interact with pages before extraction instead of stopping at static scraping

The Interact flow can click, scroll, type, wait, and press keys on a page before data is extracted. Firecrawl also exposes an Agent API that can take a natural-language request without requiring the caller to know the target URLs in advance, then search, navigate, and gather the requested information. MCP, CLI, and agent Skill integrations make the same web-access layer available to tools such as Claude Code and OpenCode.

Sources: [1]

For teams that want web acquisition to be shared infrastructure for RAG, agents, and automated research

Firecrawl fits applications that need to ingest web pages into an LLM, monitor competitor sites, turn user-provided URLs into clean context, or give agents controlled access to live web information. It is particularly relevant when a team would otherwise maintain site-specific Playwright scripts, proxy handling, retries, JavaScript rendering waits, and extraction cleanup on its own. The hosted service reduces that operational burden, while self-hosting is available when infrastructure and data handling need to remain under your control.

Sources: [1][2]

Self-hosting means operating a crawling stack, not just running a single container

The official self-host stack runs the Firecrawl API and workers alongside Playwright, Redis, RabbitMQ, NuQ PostgreSQL, and related queue components. That is operationally different from the hosted experience where proxying and orchestration are handled for you. The project's SELF_HOST.md explicitly treats the checked-in Compose setup as a starting point rather than a production architecture and calls out authentication, TLS termination, network policy, persistence, backups, monitoring, scaling, and rollback as operator responsibilities. The baseline API starts unauthenticated, so it should not simply be exposed to the public internet as-is.

Firecrawl is licensed under GNU AGPL v3. If you modify it and make that modified service available to users over a network, AGPL source-availability obligations may apply, so SaaS embedding and redistribution need license review. Separately, the fact that Firecrawl can retrieve a page does not determine whether you are permitted to crawl or reuse its content; site terms, robots policies, copyright, privacy, and responsible request rates remain separate operational and legal considerations.

Sources: [2][3]