Crawl4AI

Extract LLM-friendly Markdown and structured data with full browser rendering when needed

Crawl4AI generates clean Markdown that preserves headings, tables, code, and citation hints, with Fit Markdown and BM25 filtering to reduce noise. It also supports LLM-driven extraction and schema-based structured JSON extraction with CSS or XPath selectors.

For dynamic pages it can execute JavaScript, handle lazy loading and infinite scroll, inspect iframes, capture screenshots, and control sessions, proxies, cookies, and user scripts.

Sources: [2]

Use the same crawler as a Python library, CLI, or Docker/FastAPI service

The project can run as a Python package or CLI, while its Docker image exposes a FastAPI server with JWT authentication for API workflows. Since v0.9.0, the Docker API has moved to secure-by-default behavior with stronger authentication and binding defaults.

Sources: [2][3]

For RAG ingestion, research agents, site-to-Markdown pipelines, and scheduled web collection

It fits workflows that want to clean and structure web content before search, embedding, or agent processing instead of passing raw HTML directly to an LLM. It is particularly useful when one toolchain needs to cover both dynamic browser-based pages and repetitive pages suited to schema extraction.

v0.9.3 is a security release; prioritize upgrades for older Docker/API and PDF-processing deployments

Version 0.9.3 fixes five coordinated-disclosure advisories: arbitrary file write, SSRF, and denial of service in the PDF processing path, plus two XSS issues in the Docker Playground. Because the release is focused on security rather than new features, older internet-facing or PDF-processing deployments should not remain pinned indefinitely.

Sources: [2][3]

Full-browser crawling requires explicit resource, scope, and untrusted-input boundaries

Async browser pools, deep crawling, JavaScript execution, sessions, and proxies can process large sites, but they also increase CPU, memory, and network consumption. Set page limits, concurrency, and caching deliberately rather than allowing an open-ended crawl.

When crawled pages feed RAG or agents, remember that external web content is input you do not control. Keep extracted content in a data boundary rather than mixing it with trusted system instructions.

Sources: [2]