# AI use cases for Crawl4AI (unclecode/crawl4ai)

Repo: [unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) | Use cases: 3 | Page: https://dedupedai.com/use-cases/unclecode-crawl4ai/

Crawl4AI is an open-source web crawler and scraper that turns websites into clean, LLM-ready Markdown for RAG, agents and data pipelines, self-hosted or via a hosted API and MCP server.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Turn any web page into clean Markdown for RAG: https://dedupedai.com/use-cases/unclecode-crawl4ai/turn-website-into-markdown-for-rag/
- Give Claude Code a web scraping tool over MCP: https://dedupedai.com/use-cases/unclecode-crawl4ai/give-claude-code-web-scraping-tool/
- Scrape a page with one curl call: https://dedupedai.com/use-cases/unclecode-crawl4ai/scrape-page-via-hosted-api/

## Crawl4AI tutorial: turn any web page into Markdown for RAG

Repo: [unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) | Difficulty: Intermediate | Time: 15 minutes | Basis: From the README

Install the crawler and get LLM-ready Markdown from a URL in a few lines of Python.

**Who it is for:** A developer feeding documentation sites into a RAG index.

**The problem:** Raw HTML is noisy and wastes tokens.

**The outcome:** Clean Markdown per page ready to chunk and embed.

### Steps

1. Install with `pip install -U crawl4ai`
2. Run `crawl4ai-setup` and check with `crawl4ai-doctor`
3. Use AsyncWebCrawler and crawler.arun(url=...)
4. Save result.markdown

### Prompt to paste into your agent

```text
Using unclecode/crawl4ai, crawl <url> with AsyncWebCrawler and save the Markdown. Confirm navigation menus and scripts are not in the output.
```

### Caveat

Respect each site's terms and robots rules when crawling.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/unclecode-crawl4ai/turn-website-into-markdown-for-rag/

## Crawl4AI MCP server: give Claude Code a web scraping tool

Repo: [unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README

Add the hosted Crawl4AI MCP server so your agent can scrape pages to Markdown.

**Who it is for:** A developer whose agent needs to read web pages during research.

**The problem:** Agents' built-in fetch returns messy HTML.

**The outcome:** Scrape, extract and search tools available to the agent.

### Steps

1. Get a Crawl4AI key
2. Run `claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp` with the Authorization header
3. Ask the agent to scrape a page
4. Use the Markdown in your task

### Prompt to paste into your agent

```text
With the crawl4ai MCP server connected, scrape <url> to Markdown and summarize it. Confirm the summary only uses content from the page.
```

### Caveat

The hosted API is optional; the open-source crawler runs free on your machine.

Risk, credentials: The CRAWL4AI_KEY bearer token is billed for hosted requests.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/unclecode-crawl4ai/give-claude-code-web-scraping-tool/

## How to scrape a page with one curl call with Crawl4AI

Repo: [unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README

Call the hosted scrape endpoint and pipe the Markdown through jq.

**Who it is for:** A developer scripting quick page captures in a shell pipeline.

**The problem:** Running a browser locally for one page is heavy.

**The outcome:** Markdown of the page from a single HTTP request.

### Steps

1. Get a key and export CRAWL4AI_KEY
2. POST the URL to https://api.crawl4ai.com/scrape
3. Pipe the response to jq to read the markdown field
4. Save it to a file

### Prompt to paste into your agent

```text
Call the Crawl4AI hosted /scrape endpoint for <url> and save the markdown field to page.md. Confirm the file is non-empty.
```

### Caveat

Self-hosting avoids per-request costs.

Risk, credentials: Keep CRAWL4AI_KEY out of shared scripts.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/unclecode-crawl4ai/scrape-page-via-hosted-api/