DEDUPED Hot AI repos, turned into use cases Get the Monday 5

Crawl4AI tutorial: turn any web page into Markdown for RAG

Install the crawler and get LLM-ready Markdown from a URL in a few lines of Python.

Who it is forA developer feeding documentation sites into a RAG index.

The problemRaw HTML is noisy and wastes tokens.

The outcomeClean Markdown per page ready to chunk and embed.

What you need first

  • Python with pip
  • Playwright Chromium (installed by crawl4ai-setup)

Steps

  1. Install with pip install -U crawl4ai
  2. Run crawl4ai-setup and check with crawl4ai-doctor
  3. Use AsyncWebCrawler and crawler.arun(url=...)
  4. Save result.markdown

Prompt for your agent

Using unclecode/crawl4ai, crawl <url> with AsyncWebCrawler and save the Markdown. Confirm navigation menus and scripts are not in the output.

Caveat. Respect each site's terms and robots rules when crawling.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for unclecode/crawl4ai and "Turn any web page into clean Markdown for RAG"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer