DEDUPEDWhat do you want to do?
3 use cases
-
Turn any web page into clean Markdown for RAG
Install the crawler and get LLM-ready Markdown from a URL in a few lines of Python.
- REPO
unclecode/crawl4ai - Data and analytics
- Intermediate
View prompt
Using unclecode/crawl4ai, crawl <url> with AsyncWebCrawler and save the Markdown. Confirm navigation menus and scripts are not in the output.
- REPO
-
Give Claude Code a web scraping tool over MCP
Add the hosted Crawl4AI MCP server so your agent can scrape pages to Markdown.
View prompt
With the crawl4ai MCP server connected, scrape <url> to Markdown and summarize it. Confirm the summary only uses content from the page.
-
Scrape a page with one curl call
Call the hosted scrape endpoint and pipe the Markdown through jq.
View prompt
Call the Crawl4AI hosted /scrape endpoint for <url> and save the markdown field to page.md. Confirm the file is non-empty.
Quick start with Crawl4AI
- Check you have what it needs: Python with pip; Playwright Chromium (installed by crawl4ai-setup).
- Open unclecode/crawl4ai on GitHub and read the README and license.
- Install with
pip install -U crawl4ai - Run
crawl4ai-setupand check withcrawl4ai-doctor - Use AsyncWebCrawler and crawler.arun(url=...)
- Save result.markdown
These steps come from our top use case, Turn any web page into clean Markdown for RAG, which also has a copy-paste agent prompt.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for unclecode/crawl4ai
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeCrawl4AI alternatives · Best AI repos for Data and analytics
About this repo
- Who it is forDevelopers building RAG pipelines, agents or datasets from web content.
- Why it is hot86k stars
- LanguagePython
- Sourcegithub.com/unclecode/crawl4ai · Back to Hot repos
Requirements
- Python with pip
- Playwright Chromium (installed by crawl4ai-setup)
Add this badge to your README
Maintain Crawl4AI? Show readers where to find these use cases. 
Markdown
[](https://dedupedai.com/use-cases/unclecode-crawl4ai/?utm_source=github&utm_medium=badge&utm_campaign=unclecode-crawl4ai)
HTML
<a href="https://dedupedai.com/use-cases/unclecode-crawl4ai/?utm_source=github&utm_medium=badge&utm_campaign=unclecode-crawl4ai"><img src="https://dedupedai.com/badge/unclecode/crawl4ai.svg" alt="Use cases on Deduped" height="20"></a>
A static image, no tracking script. About the badge
Questions about Crawl4AI
What is Crawl4AI?
Crawl4AI is an open-source web crawler and scraper that turns websites into clean, LLM-ready Markdown for RAG, agents and data pipelines, self-hosted or via a hosted API and MCP server. It is an open-source project on GitHub: https://github.com/unclecode/crawl4ai.
Who is Crawl4AI for?
Developers building RAG pipelines, agents or datasets from web content.
What can I do with Crawl4AI?
3 use cases so far, for example: Turn any web page into clean Markdown for RAG; Give Claude Code a web scraping tool over MCP; Scrape a page with one curl call. Each has steps and a copy-paste agent prompt.
What do I need to run Crawl4AI?
Python with pip; Playwright Chromium (installed by crawl4ai-setup).
What are alternatives to Crawl4AI?
Other hot AI repos in Data and analytics: DSH Usage, Schemagate, Asobeast, Coreclaw API Directory, Open Index.
Is Crawl4AI safe to run?
Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.