DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to turn any web page into clean markdown for an LLM with Firecrawl

Scrape a URL and get LLM-ready markdown, HTML, a screenshot or JSON instead of raw HTML.

Short answer: Get an API key at Firecrawl.dev; then Python: from firecrawl import Firecrawl, create app = Firecrawl(api_key="fc-YOUR_API_KEY"), then result = app.scrape('firecrawl.dev'); Or from the CLI: firecrawl https://firecrawl.dev --only-main-content. All steps.

Credentials. Keep the fc- API key in an environment variable, not in code you commit.

Who it is forA developer feeding web pages to a model who is tired of cleaning boilerplate and broken JavaScript pages.

The problemRaw HTML wastes tokens and JS-heavy pages often come back empty.

The outcomeClean markdown of the page's main content, ready to drop into a prompt or a vector store.

What you need first

  • A Firecrawl API key from the hosted service, or a self-hosted instance
  • Python, Node.js, or curl; or an MCP client such as Claude Code or Cursor

Steps

  1. Get an API key at Firecrawl.dev
  2. Python: from firecrawl import Firecrawl, create app = Firecrawl(api_key="fc-YOUR_API_KEY"), then result = app.scrape('firecrawl.dev')
  3. Or from the CLI: firecrawl https://firecrawl.dev --only-main-content
  4. Ask for other formats such as structured JSON or screenshots when you need them

Prompt for your agent

Using firecrawl/firecrawl with my API key from an environment variable, scrape <url> to markdown with only the main content, and save it to <file>. Report the token-sized length before and after.

Caveat. Respect each site's terms and robots rules when scraping.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for firecrawl/firecrawl and "Turn any web page into clean markdown for an LLM"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer