DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to convert web pages to markdown for an LLM with Browser

Use lightpanda fetch to render a page, run its JavaScript, and dump clean markdown or structured data while obeying robots.txt.

Short answer: Install with brew install lightpanda-io/browser/lightpanda, or download the nightly binary and check it with ./lightpanda version; then Dump a page with ./lightpanda fetch --obey-robots --dump html --log-format pretty --log-level info https://demo-browser.lightpanda.io/campfire-commerce/; Swap in --dump markdown for markdown, or --dump structured_data for JSON-LD, OpenGraph and meta tags. All steps.

  • Top pick
  • Software and DevOps
  • Beginner
  • 20 minutes
  • From the README
  • Claude Code
  • Codex
  • Cursor
  • Gemini CLI

privacy. Lightpanda sends usage telemetry by default; set LIGHTPANDA_DISABLE_TELEMETRY=true to turn it off.

legal. Respect site terms and robots.txt; keep --obey-robots on.

Who it is forA developer feeding web pages into a RAG pipeline or agent.

The problemRaw HTML is noisy and launching full Chrome for each page is heavy.

The outcomeMarkdown or JSON of a page, produced from the command line.

What you need first

  • Linux or macOS (x86_64 or aarch64); Windows only through WSL2; Linux binaries need glibc
  • Install with `brew install lightpanda-io/browser/lightpanda`, a nightly binary, Nix or `docker run -d --name lightpanda -p 127.0.0.1:9222:9222 lightpanda/browser:latest`
  • puppeteer-core or Playwright if you connect your own scripts over CDP

Steps

  1. Install with brew install lightpanda-io/browser/lightpanda, or download the nightly binary and check it with ./lightpanda version
  2. Dump a page with ./lightpanda fetch --obey-robots --dump html --log-format pretty --log-level info https://demo-browser.lightpanda.io/campfire-commerce/
  3. Swap in --dump markdown for markdown, or --dump structured_data for JSON-LD, OpenGraph and meta tags
  4. Use --wait-selector or --wait-ms when content loads late

Prompt for your agent

Using lightpanda-io/browser, run lightpanda fetch with --obey-robots and --dump markdown on each URL in <my list> and save one markdown file per page.

Caveat. Downloads are nightly builds, so pin a version for production.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for lightpanda-io/browser and "Convert web pages to markdown for an LLM"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer