DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to pull a REST API into a local database with DLT

Describe an API declaratively and let DLT handle requests, pagination, schema inference and typing.

Short answer: Install the library and the destination extra with pip install dlt and pip install "dlt[duckdb]"; then build a source with rest_api_source, giving the client base_url and a resource with its endpoint path; create the pipeline with DLT.pipeline, naming pipeline_name, destination and dataset_name, then call pipeline.run(source). All steps.

  • Top pick
  • Data and analytics
  • Beginner
  • 45 minutes
  • From the README
  • Claude Code
  • Codex
  • Cursor
  • Gemini CLI

Credentials. Source and destination credentials come from secrets.toml or environment variables, so keep them out of your code.

Who it is forA developer who needs API data in a queryable table today.

The problemHand written extract scripts break on pagination and schema changes.

The outcomeA DuckDB dataset you can query, plus a DataFrame of the loaded table.

What you need first

  • Python 3.10 through 3.14, with 3.14 support described as experimental
  • `pip install dlt` plus the extras for your sources and destinations
  • Credentials for your destination in secrets.toml or environment variables

Steps

  1. Install the library and the destination extra with pip install dlt and pip install "dlt[duckdb]"
  2. Build a source with rest_api_source, giving the client base_url and a resource with its endpoint path
  3. Create the pipeline with DLT.pipeline, naming pipeline_name, destination and dataset_name, then call pipeline.run(source)
  4. Read the data straight back with pipeline.dataset() and .df()

Prompt for your agent

Using dlt-hub/dlt, write a pipeline that loads <my API endpoint> into DuckDB with rest_api_source and prints the first rows back as a DataFrame.

Caveat. Add the matching extra for any other destination, for example DLT[bigquery] or DLT[s3].

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for dlt-hub/dlt and "Pull a REST API into a local database"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer