Clean up a resource or a source¶
This guide removes one resource from a live destination — its table, its incremental state, its checkpoints — then removes a whole source, and verifies at every step that nothing else was touched.
Prerequisites
dlt-opswith the DuckDB extra — see installation.- A checkout of the repository for the
examples/basic_projectused throughout. - DuckDB is full tier, so remote cleanup (the destination-side half) is available; step 6 shows the core-tier refusal.
Steps at a glance
- Start state: two sources sharing one DuckDB file
- Dry-run the selective clean
- Clean the resource and verify
- Clean the whole source
- Verify: rows deleted, tables kept
- Core tier:
--local-only
Why clean exists and what it touches¶
dlt covers the destination half of this, and clean uses it. dlt pipeline <name> drop <resource> drops the resource's tables and resets its incremental state as one load package, with resource regexes, --drop-all, --state-only, and --state-paths; dlt -y pipeline <name> drop <resource> runs it non-interactively (upstream marks the command experimental, and the Python entry point is dlt.pipeline.helpers.pipeline_drop). Reach for it directly when the destination is all you need to touch — and never hand-delete just the table, which leaves the incremental cursor behind, so the next run silently resumes from where the deleted data ended instead of re-extracting it.
pipeline clean calls that same pipeline_drop for the destination-side work, so the drop is dlt's own transactional, append-only operation and dlt-ops neither decodes nor rewrites dlt state. Two things follow. Tables are whatever dlt says it would drop — nested child tables included, which a name-based guess would miss and leave orphaned. And the drop runs on a pipeline whose working dir is a throwaway temp dir synced from the destination, so --remote-only never mutates local state and a pending local load package can never block it.
pipeline clean exists for the rest of the footprint, outside the scope of the drop itself: a --dry-run that prints the plan and changes nothing, the local pipeline working directory, the rows in dlt-ops' own _dlt_custom_checkpoints table that dlt has no knowledge of, the pipeline's leftover rows in dlt's shared system tables on a full clean, and a source-level unit — clean every resource of a source named the dlt-ops way, without enumerating them.
clean operates on both halves of a pipeline's footprint: remote (data tables in the destination, dlt resource state, rows in the _dlt_custom_checkpoints table) and local (the pipeline working directory under ~/.dlt/pipelines/, or DLT_DATA_DIR). By default it cleans both; --local-only / --remote-only narrow it. All the dlt-ops-owned remote SQL goes through the DestinationAdapter boundary, which makes remote cleanup a full-tier verb — step 6 shows what happens without an adapter.
-r takes exact resource names. dlt reads a re:-prefixed selector as a regex; clean promises exact names, so a selector that is not a resource this source declares is refused before anything is touched, rather than forwarded and silently widened into a matching one.
1. Start state: two sources sharing one DuckDB file¶
The start state is the repository's examples/basic_project: two sources that will share one DuckDB file. Copy it out — github_events_api (an events resource with an incremental cursor and checkpoints, plus a cursor-less actors) and github_events_full (an event_types catalog), both loading into the github_events_raw dataset:
By default dlt gives each pipeline its own DuckDB file, which would hide the most important thing this guide demonstrates — what happens to shared system tables when one of several pipelines in a dataset is cleaned. Point both pipelines at one file by appending to .dlt/secrets.toml (dlt refuses credentials keys in config.toml, even when the "credential" is just a file path; the path is cwd-relative, so run commands from the project root):
Run both sources:
2026-07-17 10:24:15|[INFO]|root|[events] Checkpoint saved: page 2, 6 records, value: 2026-01-01T05:00:00+00:00
...
1 load package(s) were loaded to destination duckdb and into dataset github_events_raw
Now look at what actually exists in the destination — the data tables, plus dlt's shared system tables and the dlt-ops ones:
python - <<'PY'
import duckdb
con = duckdb.connect("github_events.duckdb", read_only=True)
print(con.sql("SELECT table_name FROM information_schema.tables WHERE table_schema='github_events_raw' ORDER BY table_name"))
print(con.sql("SELECT pipeline_name, resource_name, status FROM github_events_raw._dlt_custom_checkpoints"))
con.close()
PY
┌─────────────────────────┐
│ table_name │
├─────────────────────────┤
│ _dlt_custom_checkpoints │
│ _dlt_loads │
│ _dlt_ops_runs │
│ _dlt_pipeline_state │
│ _dlt_trace │
│ _dlt_version │
│ actors │
│ event_types │
│ events │
└─────────────────────────┘
┌────────────────────────────┬───────────────┬───────────┐
│ pipeline_name │ resource_name │ status │
├────────────────────────────┼───────────────┼───────────┤
│ github_events_api_pipeline │ events │ completed │
│ github_events_api_pipeline │ events │ completed │
│ github_events_api_pipeline │ events │ completed │
└────────────────────────────┴───────────────┴───────────┘
The three _dlt_* state tables (_dlt_pipeline_state, _dlt_loads, _dlt_version) are dlt's, and they are shared by every pipeline writing to this dataset — both sources' rows live in the same tables. That fact drives cleanup's cardinal safety rule: data tables are dropped, but system tables only ever have rows deleted, never the table itself.
2. Dry-run the selective clean¶
Remove the events resource from github_events_api — but dry-run first. --dry-run prints the plan and exits without touching anything:
Discovering sources
Cleanup Plan:
Source: github_events_api
Pipeline: github_events_api_pipeline
Resources: events
Local: update state.json + schema (keep working dir)
Remote: github_events_raw
- 1 data table(s): events
- 1 resource state(s)
- state: reset the selected resources (dlt drop, append-only)
- checkpoints for 1 resource(s)
Dry-run mode: no changes will be made
The plan names the three remote targets: the events data table (dropped), the resource's dlt state (reset by dlt's drop, which appends a new state version rather than editing the old one in place), and its checkpoint rows. Locally, a selective clean edits state.json and deletes the schema file but keeps the working directory, because the source's other resources still live there.
The remote half of that plan is dlt's own dry run: pipeline_drop is constructed and its .info read, never called. So the table list is what dlt would actually drop — nested child tables and all — rather than a guess from the naming convention. Building it syncs the schema from the destination first, which is also why clean works on a machine that never ran the pipeline.
3. Clean the resource and verify¶
The real command asks for confirmation; --auto-approve skips the prompt (the flag for scripts):
Cleaning...
10:25:16|dlt_ops.discovery.cleanup|dlt drop complete for schema 'github_events_api'
10:25:16|dlt_ops.discovery.cleanup|Deleted checkpoints for resource: events
10:25:16|dlt_ops.discovery.cleanup|Updated local state.json
10:25:16|dlt_ops.discovery.cleanup|Deleted local schema: ~/.dlt/pipelines/github_events_api_pipeline/schemas/github_events_api.schema.json
Local:
- state.json (resource entries removed)
- schema (deleted github_events_api.schema.json)
Remote:
- table: events
- state: reset events
- checkpoint: events
Cleanup complete
Verify the blast radius — events gone, everything else intact, checkpoint rows deleted:
python - <<'PY'
import duckdb
con = duckdb.connect("github_events.duckdb", read_only=True)
print(con.sql("SELECT table_name FROM information_schema.tables WHERE table_schema='github_events_raw' AND table_name NOT LIKE '\\_%' ESCAPE '\\' ORDER BY table_name"))
print(con.sql("SELECT count(*) AS checkpoint_rows FROM github_events_raw._dlt_custom_checkpoints WHERE resource_name = 'events'"))
con.close()
PY
┌─────────────┐
│ table_name │
├─────────────┤
│ actors │
│ event_types │
└─────────────┘
┌─────────────────┐
│ checkpoint_rows │
├─────────────────┤
│ 0 │
└─────────────────┘
The incremental state went with it. Re-run the source and events starts over from its initial cursor value and re-extracts everything — which is the point of cleaning a resource, and exactly what dropping only the table would not have done:
2026-07-17 10:25:50|[INFO]|root|[events] Run isolation: value=2026-01-01T00:00:00+00:00, run_id=2db0bb0b60653f75
...
events: 20 | Time: 0.09s | Rate: 230.05/s
1 load package(s) were loaded to destination duckdb and into dataset github_events_raw
4. Clean the whole source¶
Omit -r and the unit becomes the source: all its data tables, all its state, its local working directory — and row-level deletes from the shared system tables:
Cleanup Plan:
Source: github_events_api
Pipeline: github_events_api_pipeline
Resources: all (2 total)
Local: ~/.dlt/pipelines/github_events_api_pipeline
Remote: github_events_raw
- 2 data table(s): actors, events
- 2 resource state(s)
- system tables: DELETE rows from _dlt_pipeline_state, _dlt_loads, _dlt_version
- checkpoints for 2 resource(s)
Dry-run mode: no changes will be made
Cleaning...
10:26:17|dlt_ops.discovery.cleanup|dlt drop complete for schema 'github_events_api'
10:26:17|dlt_ops.discovery.cleanup|Deleted _dlt_pipeline_state rows where pipeline_name = github_events_api_pipeline
10:26:17|dlt_ops.discovery.cleanup|Deleted _dlt_version rows where schema_name = github_events_api
10:26:17|dlt_ops.discovery.cleanup|Deleted _dlt_loads rows where schema_name = github_events_api
10:26:17|dlt_ops.discovery.cleanup|Deleted _dlt_custom_checkpoints rows where pipeline_name = github_events_api_pipeline
10:26:17|dlt_ops.discovery.cleanup|Removed local pipeline directory: ~/.dlt/pipelines/github_events_api_pipeline
Local:
- ~/.dlt/pipelines/github_events_api_pipeline
Remote:
- table: actors
- table: events
- state: _dlt_pipeline_state (rows deleted)
- state: _dlt_version (rows deleted)
- state: _dlt_loads (rows deleted)
- state: _dlt_custom_checkpoints (rows deleted)
Cleanup complete
dlt's drop reports the tables that physically held data; the four state: lines are the dlt-ops half, deleting this pipeline's rows from the shared bookkeeping tables that dlt's append-only drop leaves in place. Without that half a "cleaned" source would still appear in the dataset's load history.
5. Verify: rows deleted, tables kept¶
This is the shared-system-tables fact, checked against the live file. The state tables still exist; only github_events_api's rows are gone; the sibling source's row in each — and its event_types data — survived untouched:
python - <<'PY'
import duckdb
con = duckdb.connect("github_events.duckdb", read_only=True)
print(con.sql("SELECT table_name FROM information_schema.tables WHERE table_schema='github_events_raw' ORDER BY table_name"))
print(con.sql("SELECT schema_name, count(*) AS rows FROM github_events_raw._dlt_loads GROUP BY 1 ORDER BY 1"))
print(con.sql("SELECT count(*) AS event_type_rows FROM github_events_raw.event_types"))
con.close()
PY
┌─────────────────────────┐
│ table_name │
├─────────────────────────┤
│ _dlt_custom_checkpoints │
│ _dlt_loads │
│ _dlt_ops_runs │
│ _dlt_pipeline_state │
│ _dlt_trace │
│ _dlt_version │
│ event_types │
└─────────────────────────┘
┌────────────────────┬───────┐
│ schema_name │ rows │
├────────────────────┼───────┤
│ _dlt_traces │ 3 │
│ github_events_full │ 1 │
└────────────────────┴───────┘
┌─────────────────┐
│ event_type_rows │
├─────────────────┤
│ 4 │
└─────────────────┘
Had cleanup dropped _dlt_loads instead of deleting rows from it, github_events_full would have lost its load history and its persisted state — that is why "DELETE rows, never DROP" is the rule for anything shared. Two tables are deliberately not in clean's scope at all: _dlt_ops_runs (the runs ledger — the operational record of what ran survives the removal of what it ran on) and _dlt_trace; the _dlt_traces rows above are that machinery's own internal pipeline state, equally untouched.
6. Core tier: --local-only¶
Remote cleanup speaks SQL to the destination, so it needs a registered DestinationAdapter. On a core-tier destination — here a scaffolded demo project (dlt-ops init demo --example) pointed at a local filesystem bucket and run once — the remote half refuses:
Error: destination 'filesystem' has no DestinationAdapter — remote cleanup needs one (core mode). Clean local state with --local-only, or register an adapter; see docs/reference/destinations.md.
The command exits 1 and nothing is deleted, on either side. --dry-run refuses identically — the plan's remote half is built by opening the destination, so there is no plan to draw, and printing a partial one you could approve would be worse than the refusal. The local half needs no adapter — --local-only works at any tier:
Cleanup Plan:
Source: demo_events
Pipeline: demo_events_pipeline
Resources: all (1 total)
Local: ~/.dlt/pipelines/demo_events_pipeline
Cleaning...
10:32:56|dlt_ops.discovery.cleanup|Removed local pipeline directory: ~/.dlt/pipelines/demo_events_pipeline
Local:
- ~/.dlt/pipelines/demo_events_pipeline
Cleanup complete
The bucket's files are untouched — on core tier, removing destination-side data is your storage tooling's job; clean refuses to half-do it.
Any dlt version at or above the floor
clean gates on nothing but the adapter. The destination-side drop is dlt's own pipeline_drop, and the shared bookkeeping rows dlt-ops deletes afterwards are addressed by table and column names read off the live schema, so they follow whatever naming convention that destination uses rather than a hardcoded guess. The one file cleanup still edits by hand — the local state.json — checks the state engine version dlt stamped into it and refuses an unfamiliar layout before any destructive step runs. See compatibility.
The first run after a selective clean¶
Selective cleanup deletes the pipeline's local schema file — dlt re-derives it from the source on the next run, and surgically editing it instead would break dlt's schema-hash verification. That re-run rebuilds the schema as it extracts and loads normally (step 3 above re-extracted all 20 events rows), and it records its runs-ledger row like any other run: the ledger writer reaches the destination's _dlt_ops_runs table through the DestinationAdapter boundary, independently of the local schema, so a wiped local schema never costs a run its status row. dlt logs one restored from destination with inconsistent state notice while it re-derives the schema — expected, and absent from the following run once the file is back.
Where next¶
- Destinations and tiers — why remote
cleanis a full-tier verb, and what core tier keeps - Runs ledger — the operational record
cleandeliberately leaves alone - Compatibility — the verified dlt matrix, and why it gates nothing at runtime