DEDUPEDWhat is the best open-source repo for scraping websites into LLM-ready data?
The best open-source AI repos for web scraping for LLMs and RAG right now are Firecrawl, Crawl4AI and Browser Use. Firecrawl is the most starred option and returns clean markdown or JSON through an API; Crawl4AI is fully self-hosted; browser-use helps when a site needs clicks and logins before the data appears.
Comparison
| Repo | What it does | Stars | Start here |
|---|---|---|---|
| Firecrawl | Firecrawl is an API that searches, scrapes, crawls and interacts with the web and returns clean markdown, structured JSON or screenshots for AI agents. | 195,071 | Turn any web page into clean markdown for an LLM |
| Crawl4AI | Crawl4AI is an open-source web crawler and scraper that turns websites into clean, LLM-ready Markdown for RAG, agents and data pipelines, self-hosted or via a hosted API and MCP server. | n/a | Turn any web page into clean Markdown for RAG |
| Browser Use | browser-use is an open-source Python library for AI browser agents that navigate websites with any LLM, plus an optional cloud with stealth browsers and a hosted agent API. | n/a | Automate a web task with a Python browser agent |
When to pick which
Firecrawl
Firecrawl is an API that searches, scrapes, crawls and interacts with the web and returns clean markdown, structured JSON or screenshots for AI agents. It handles JavaScript-heavy pages, proxies and rate limits, parses PDFs and DOCX, can click and type before extracting, and plugs into agents as a skill or an MCP server. It is open source and also offered as a hosted service with an API key.
Best for: Developers building AI agents, RAG pipelines or research tools that need clean page content instead of raw HTML.
Needs: A Firecrawl API key from the hosted service, or a self-hosted instance; Python, Node.js, or curl; or an MCP client such as Claude Code or Cursor
Start here: Turn any web page into clean markdown for an LLM
- Get an API key at Firecrawl.dev
- Python: from firecrawl import Firecrawl, create app = Firecrawl(api_key="fc-YOUR_API_KEY"), then result = app.scrape('firecrawl.dev')
- Or from the CLI: firecrawl https://firecrawl.dev --only-main-content
- Ask for other formats such as structured JSON or screenshots when you need them
Crawl4AI
Crawl4AI is an open-source web crawler and scraper that turns websites into clean, LLM-ready Markdown for RAG, agents and data pipelines, self-hosted or via a hosted API and MCP server.
Best for: Developers building RAG pipelines, agents or datasets from web content.
Needs: Python with pip; Playwright Chromium (installed by crawl4ai-setup)
Start here: Turn any web page into clean Markdown for RAG
- Install with pip install -U crawl4ai
- Run crawl4ai-setup once to install the browser, then check with crawl4ai-doctor
- From the shell, crwl https://news.ycombinator.com -o markdown prints a page as Markdown
- In Python, open AsyncWebCrawler() as crawler, await crawler.arun(url=...) and save result.markdown
Browser Use
browser-use is an open-source Python library for AI browser agents that navigate websites with any LLM, plus an optional cloud with stealth browsers and a hosted agent API.
Best for: Developers automating web tasks such as form filling, lookups and bookings with AI agents.
Needs: Python 3.12 with uv; An LLM API key such as OpenAI, or a Browser Use key
Start here: Automate a web task with a Python browser agent
- With uv installed, start a project with uv init --python 3.12 if needed
- Add the library: uv add browser-use
- Put OPENAI_API_KEY in .env (or BROWSER_USE_API_KEY to use the BU2 model)
- In agent.py create Agent(task="...", llm=ChatOpenAI(...)), call await agent.run() and print history.final_result()
Use cases for web scraping for LLMs and RAG
- Turn any web page into clean markdown for an LLM
- Give Claude Code or Cursor live web search and scraping
- Crawl a whole docs site into a knowledge base
- Turn any web page into clean Markdown for RAG
- Give Claude Code a web scraping tool over MCP
- Scrape pages with one curl call, hosted or on your own server
- Automate a web task with a Python browser agent
- Give your coding agent a browser skill
- Run browser agents on cloud stealth browsers
Questions
What is the best open-source repo for scraping websites into LLM-ready data?
The best open-source AI repos for web scraping for LLMs and RAG right now are Firecrawl, Crawl4AI and Browser Use. Firecrawl is the most starred option and returns clean markdown or JSON through an API; Crawl4AI is fully self-hosted; browser-use helps when a site needs clicks and logins before the data appears.
Which repo for web scraping for LLMs and RAG is easiest to start with?
Firecrawl: its starter use case "Turn any web page into clean markdown for an LLM" is rated Beginner and takes about 10 minutes.
How do I get started with Firecrawl?
Get an API key at Firecrawl.dev; Python: from firecrawl import Firecrawl, create app = Firecrawl(api_key="fc-YOUR_API_KEY"), then result = app.scrape('firecrawl.dev'); Or from the CLI: firecrawl https://firecrawl.dev --only-main-content. Full steps and a copy-paste agent prompt: https://dedupedai.com/use-cases/firecrawl-firecrawl/turn-any-url-into-clean-markdown/
Are these web scraping for LLMs and RAG repos free?
They are open-source projects on GitHub (check each license; some, like fair-code projects, limit commercial use). Some also sell a hosted version or need a paid model API key; the requirements for each repo are listed on this page.
How is this list of web scraping for LLMs and RAG repos chosen?
Deduped tracks the hottest AI repos on GitHub and writes step-by-step use cases from each README. For this page we picked the repos that do this job, ordered by fit and then GitHub stars. Repos we only inferred from a short README, or that we do not index, are left out. The page is rebuilt with the site; stars are as of the last build in 2026.
More answers
- Best open-source AI repos for running LLMs locally
- Best open-source AI repos for AI coding agents
- Best open-source AI repos for skills and workflows for coding agents
- Best open-source AI repos for browser automation with AI agents
- Best open-source AI repos for RAG and chatting with your documents
- Best open-source AI repos for long-term memory for AI agents
- Best open-source AI repos for voice, speech-to-text and text-to-speech
- Best open-source AI repos for AI video editing and generation
- Best open-source AI repos for AI image generation and editing
- Best open-source AI repos for UI design with AI coding agents
- Best open-source AI repos for MCP servers
- Best open-source AI repos for 3D modeling and CAD with AI
- Best open-source AI repos for research with AI agents
- Best open-source AI repos for running several coding agents in parallel
- Best open-source AI repos for LLM gateways and model routers
- Best open-source AI repos for AI workflow automation
Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them free