DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to scrape a page with one curl call with Crawl4AI

Call the hosted scrape endpoint and pipe the Markdown through jq.

Credentials. Keep CRAWL4AI_KEY out of shared scripts.

Who it is forA developer scripting quick page captures in a shell pipeline.

The problemRunning a browser locally for one page is heavy.

The outcomeMarkdown of the page from a single HTTP request.

What you need first

  • Python with pip
  • Playwright Chromium (installed by crawl4ai-setup)

Steps

  1. Get a key and export CRAWL4AI_KEY
  2. POST the URL to https://api.crawl4ai.com/scrape
  3. Pipe the response to jq to read the markdown field
  4. Save it to a file

Prompt for your agent

Call the Crawl4AI hosted /scrape endpoint for <url> and save the markdown field to page.md. Confirm the file is non-empty.

Caveat. Self-hosting avoids per-request costs.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for unclecode/crawl4ai and "Scrape a page with one curl call"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer