DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to load only new rows and upsert on a key with DLT

Use the resource decorator to declare primary key, merge write disposition, column hints and a schema contract.

Short answer: Install the library with pip install dlt; then Decorate your generator with @DLT.resource and set primary_key and write_disposition to merge for upserts; add updated_at=dlt.sources.incremental("updated_at") as an argument and fetch only rows after updated_at.last_value. All steps.

Who it is forAn engineer running the same pipeline every hour.

The problemReloading everything each run is slow and duplicates rows.

The outcomeA pipeline that upserts on your key and only pulls changed rows.

What you need first

  • Python 3.10 through 3.14, with 3.14 support described as experimental
  • `pip install dlt` plus the extras for your sources and destinations
  • Credentials for your destination in secrets.toml or environment variables

Steps

  1. Install the library with pip install dlt
  2. Decorate your generator with @DLT.resource and set primary_key and write_disposition to merge for upserts
  3. Add updated_at=dlt.sources.incremental("updated_at") as an argument and fetch only rows after updated_at.last_value
  4. Guard the shape with schema_contract, choosing evolve, freeze or discard per tables, columns and data_type

Prompt for your agent

With dlt-hub/dlt, convert <my full reload script> into an incremental merge resource keyed on id, and freeze the columns so unexpected fields are rejected.

Caveat. Freeze rejects the record while discard drops the offending row or column, so pick deliberately.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for dlt-hub/dlt and "Load only new rows and upsert on a key"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer