# AI use cases for DLT (dlt-hub/dlt)

Repo: [dlt-hub/dlt](https://github.com/dlt-hub/dlt) | Use cases: 3 | Page: https://dedupedai.com/use-cases/dlt-hub-dlt/

dlt (data load tool) is an open source Python library that loads data from APIs, SQL databases, files, DataFrames and Arrow tables into typed datasets in more than twenty destinations. It handles credentials, DDL in the target dialect, type mapping, staging and schema drift, and adds decorators for incremental loading, merge write dispositions and schema contracts. Loaded data reads back through a Dataset API or as Ibis expressions.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Pull a REST API into a local database: https://dedupedai.com/use-cases/dlt-hub-dlt/load-a-rest-api-into-duckdb-with-dlt/
- Load only new rows and upsert on a key: https://dedupedai.com/use-cases/dlt-hub-dlt/load-incrementally-with-dlt-decorators/
- Query what you loaded without writing SQL: https://dedupedai.com/use-cases/dlt-hub-dlt/read-and-transform-loaded-data-with-the-dlt-dataset-api/

## How to pull a REST API into a local database with DLT

Repo: [dlt-hub/dlt](https://github.com/dlt-hub/dlt) | Difficulty: Beginner | Time: 45 minutes | Basis: From the README

Describe an API declaratively and let DLT handle requests, pagination, schema inference and typing.

**Who it is for:** A developer who needs API data in a queryable table today.

**The problem:** Hand written extract scripts break on pagination and schema changes.

**The outcome:** A DuckDB dataset you can query, plus a DataFrame of the loaded table.

### Steps

1. Install the library and the destination extra with `pip install dlt` and `pip install "dlt[duckdb]"`
2. Build a source with rest_api_source, giving the client base_url and a resource with its endpoint path
3. Create the pipeline with DLT.pipeline, naming pipeline_name, destination and dataset_name, then call pipeline.run(source)
4. Read the data straight back with pipeline.dataset() and .df()

### Prompt to paste into your agent

```text
Using dlt-hub/dlt, write a pipeline that loads <my API endpoint> into DuckDB with rest_api_source and prints the first rows back as a DataFrame.
```

### Caveat

Add the matching extra for any other destination, for example DLT[bigquery] or DLT[s3].

Risk, credentials: Source and destination credentials come from secrets.toml or environment variables, so keep them out of your code.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/dlt-hub-dlt/load-a-rest-api-into-duckdb-with-dlt/

## How to load only new rows and upsert on a key with DLT

Repo: [dlt-hub/dlt](https://github.com/dlt-hub/dlt) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Use the resource decorator to declare primary key, merge write disposition, column hints and a schema contract.

**Who it is for:** An engineer running the same pipeline every hour.

**The problem:** Reloading everything each run is slow and duplicates rows.

**The outcome:** A pipeline that upserts on your key and only pulls changed rows.

### Steps

1. Install the library with `pip install dlt`
2. Decorate your generator with @DLT.resource and set primary_key and write_disposition to merge for upserts
3. Add `updated_at=dlt.sources.incremental("updated_at")` as an argument and fetch only rows after updated_at.last_value
4. Guard the shape with schema_contract, choosing evolve, freeze or discard per tables, columns and data_type

### Prompt to paste into your agent

```text
With dlt-hub/dlt, convert <my full reload script> into an incremental merge resource keyed on id, and freeze the columns so unexpected fields are rejected.
```

### Caveat

Freeze rejects the record while discard drops the offending row or column, so pick deliberately.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/dlt-hub-dlt/load-incrementally-with-dlt-decorators/

## How to query what you loaded without writing SQL with DLT

Repo: [dlt-hub/dlt](https://github.com/dlt-hub/dlt) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README

Reattach to a pipeline by name and compose transformations as Ibis expressions that compile to SQL.

**Who it is for:** An analyst who wants to work in Python on data already loaded.

**The problem:** Switching to the warehouse console breaks the workflow.

**The outcome:** Aggregated results computed in the destination, returned to Python.

### Steps

1. Install the library with `pip install dlt`
2. Reconnect with DLT.attach, passing pipeline_name, destination and dataset_name
3. List tables with dataset.tables and pull one as a lazy relation, then call .df(), .arrow() or .to_ibis()
4. Compose group_by and aggregate on the Ibis expression and call .to_pyarrow(), which compiles to SQL and runs on the destination

### Prompt to paste into your agent

```text
Using dlt-hub/dlt, attach to <my pipeline>, aggregate <my table> by one column with Ibis and bring the result back as an Arrow table.
```

### Caveat

Nothing runs until you ask for the result, so time the final call, not the expression.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/dlt-hub-dlt/read-and-transform-loaded-data-with-the-dlt-dataset-api/