DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to crawl a whole docs site into a knowledge base with Firecrawl

Map every URL on a site, then crawl it to markdown in one job for RAG or offline reading.

Short answer: Discover pages first with the Map endpoint, optionally with search to rank URLs by relevance; then start a crawl with a POST to the v2 crawl endpoint with the site URL, a page limit and markdown format; Poll the returned job ID until the crawl finishes. All steps.

Who it is forSomeone building a chatbot or search over a product's documentation.

The problemCopying a docs site page by page is slow and misses pages.

The outcomeMarkdown for every page of the site, ready to chunk and index.

What you need first

  • A Firecrawl API key from the hosted service, or a self-hosted instance
  • Python, Node.js, or curl; or an MCP client such as Claude Code or Cursor

Steps

  1. Discover pages first with the Map endpoint, optionally with search to rank URLs by relevance
  2. Start a crawl with a POST to the v2 crawl endpoint with the site URL, a page limit and markdown format
  3. Poll the returned job ID until the crawl finishes
  4. Chunk and index the markdown in your vector store

Prompt for your agent

Use firecrawl/firecrawl to map <docs url>, then crawl up to <n> pages as markdown, save each page to ./kb/<slug>.md, and give me a count of pages and any that failed.

Caveat. Crawls use credits on the hosted plan; set a page limit before you start.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for firecrawl/firecrawl and "Crawl a whole docs site into a knowledge base"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer