Skip to content

Schema-drift reconciler

pipeline reconcile diffs the live destination schema against the Pydantic models your resources declare via columns=, and routes findings to pluggable alert sinks. It is strictly read-only — it never mutates pipeline state and never blocks runs — and it exists because schema drift is silent at ingest in both directions: columns can appear in the destination behind your model's back, and columns your model declares can quietly stop carrying data. Read this for both detection mechanisms, what deliberately does not count as drift, and how findings reach you.

At a glance

What it is When it applies Requires On failure Canonical detail
pipeline reconcile — a read-only diff of the live destination schema against your Pydantic columns= models, routed to alert sinks On demand or your own cron; against every discovered source regardless of contract mode Full tier (a DestinationAdapter); removal detection also needs load_timestamp_column Findings exit 0 (an alert to route, not a failure); only a reconciler error exits 1; sinks are best-effort Failure semantics for where read-only diagnostics sit

Schema contracts and the reconciler split the work: the canonical freeze contract rejects unknown columns at ingest, while the reconciler catches what no ingest-time contract can see — after-the-fact schema surgery and data that goes dark inside columns that still exist. That is why it runs against every discovered source regardless of contract mode:

  • Evolve sources — the primary drift signal: an evolving contract admits the new column at ingest by design, so the reconciler is the only thing that tells you upstream shipped a field your model doesn't know.
  • Freeze sources — the patched-schema failure mode: an engineer hand-patches the destination schema to unblock ingest and forgets (or defers) the model PR. Freeze is satisfied at ingest, but the live table now has more columns than the model — the reconciler fires.

Additive detection

Additive detection is the default scan: for each resource it fetches the live column list from the destination, computes the column set the resource's Pydantic model produces at the destination, and reports live columns the model doesn't know. The examples below run in a copy of the repository's examples/basic_project (the same fuller demo the checkpoints page uses) after one run of its github_events_api source; a fresh project reconciles clean:

dlt-ops pipeline reconcile -s github_events_api --dry-run

Dry-run: alert emission suppressed

Source: github_events_api  |  Findings: 0  |  Duration: 0.90s
  ✓ No drift

Now commit the classic crime — add a column directly in the destination, behind the model's back:

python - <<'PY'
import duckdb
con = duckdb.connect("github_events_api_pipeline.duckdb")
con.execute("ALTER TABLE github_events_raw.events ADD COLUMN payload_size BIGINT")
con.execute("UPDATE github_events_raw.events SET payload_size = 512 + id * 7")
con.close()
PY
dlt-ops pipeline reconcile -s github_events_api

2026-07-16 18:25:09|[WARNING]|dlt_ops.reconciler._emission|schema drift (additive): github_events.github_events_api.events — 1 column(s): payload_size | reproduce: SELECT "payload_size" FROM "github_events_raw"."events" WHERE "loaded_at" >= TIMESTAMP '2026-07-16T16:25:09.758696+00:00' LIMIT 5
Source: github_events_api  |  Findings: 1  |  Duration: 0.88s
  • events: additive drift (1 column(s))
      Columns: payload_size
      First seen: 2026-07-16T16:25:09.758696+00:00

The first line is the default logging alert sink emitting the finding; the block under it is the CLI's own rendering, printed with or without emission.

Detection emits one finding per drifted resource — carrying the full drifted column set, the destination-reported type per column, up to five recent sample values per column (one windowed query per resource; a failed sample query never blocks the alert), and a copy-pasteable reproduce SELECT — rather than one alert per column, so a single upstream change cannot fan out into twenty pages.

Three column families are deliberately not drift:

  • dlt's own system columns (anything prefixed _dlt_) are skipped by prefix.
  • Columns your infrastructure stamps on every row are subtracted via config — project-wide [dlt_ops] injected_columns, per-source [sources.<X>.dlt_ops] injected_columns, and the configured load_timestamp_column, which is auto-ignored because the runner stamps it (it is never part of a Pydantic model by design). Adding a new stamped key is a one-line TOML edit, not a reconciler change.
  • A resource with no Pydantic columns= model has nothing to diff against, so it is skipped — as is a resource whose table doesn't exist at the destination yet. Both detectors skip these resources, so a declared resource that never landed produces no query and no error. A resource's table is looked up under the resource name normalized with the source's naming convention (the name dlt uses when the resource sets no table_name), in the configured dataset normalized with the same convention (dlt's default dataset-name normalization), so a resource OrderItems in dataset RAW_ORDERS is found as table order_items in raw_orders.

One subtlety does the correctness heavy lifting: both sides of the diff speak destination-side names. For each model field, the reconciler uses one name: its alias if set, else its attribute name. This is the rule dlt uses to name a field's column when it builds table columns from a model. That name is run through the same dlt naming convention the write path uses — the source's own schema convention, not a hardcoded default — so a Pydantic field startTime matches the persisted start_time column instead of surfacing as false-positive drift. A validation_alias or serialization_alias does not add a name, and the attribute name of an aliased field is not used.

Removal detection: --include-removal

Additive drift is half the story. A provider can stop sending a field without any schema change: the column still exists, ingestion accepts null under every contract mode, and nothing fails — the data just goes dark.

--include-removal adds a windowed non-null-coverage scan over the model's columns: for each column, one query computes the fraction of non-null values in the recent window (last 6 hours of loads) and in the baseline window (the 7-day window ending where the recent one begins), timestamped on the table's time column. A column whose baseline coverage exceeded 20% and whose recent coverage fell below 1% went dark.

Simulating exactly that — backdating the existing rows into the baseline window, then appending fresh loads where actor_login (a nullable model column with 80% historical coverage) is only NULL:

python - <<'PY'
import duckdb
con = duckdb.connect("github_events_api_pipeline.duckdb")
con.execute("UPDATE github_events_raw.events SET loaded_at = loaded_at - INTERVAL 3 DAY")
for i in range(5):
    con.execute(
        "INSERT INTO github_events_raw.events "
        "(id, event_type, actor_login, occurred_at, loaded_at, _dlt_load_id, _dlt_id) "
        f"VALUES ({100 + i}, 'push', NULL, now(), now(), 'manual', 'manual-{i}')"
    )
con.close()
PY
dlt-ops pipeline reconcile -s github_events_api --include-removal --dry-run

Dry-run: alert emission suppressed

Source: github_events_api  |  Findings: 1  |  Duration: 0.82s
  • events: additive drift (1 column(s))
      Columns: payload_size
      First seen: 2026-07-16T16:25:24.842742+00:00

Source: github_events_api (removal)  |  Findings: 1  |  Duration: 0.66s
  • events: removal drift (1 column(s))
      Columns: actor_login
      First seen: 2026-07-16T16:25:25.474803+00:00

The thresholds are deliberately conservative in both directions. A window with no rows at all yields no coverage ratio, and the detector refuses to flag on it — an idle resource is indistinguishable from a dropped field, and guessing would alert on every quiet weekend.

Removal findings carry empty samples (there are no recent non-null values to sample, by definition) and empty inferred types; the reproduce SELECT still ships. The windows and thresholds are keyword parameters of the Python API (detect_removal, defaults exported as DEFAULT_*); the CLI runs the canonical defaults.

Removal detection needs a time axis, so it requires load_timestamp_column — the example project sets it (loaded_at); the scaffolded demo project does not, and asking anyway degrades honestly instead of guessing:

Source: demo_events (removal)  |  Findings: 0  |  Duration: 0.26s
  ✓ No drift
  ! removal detection skipped: [dlt_ops] load_timestamp_column is not set — windowed coverage needs a time axis. Set it to enable removal-drift detection.

Time column per table

Both detectors pick a time column for each table, in this order:

  1. The configured load_timestamp_column, normalized with the source's naming convention, if the table has that column.
  2. Otherwise, the column the destination reports as the table's partition column. The BigQuery adapter reads it from INFORMATION_SCHEMA.COLUMNS; the DuckDB, Postgres, and derived adapters report no partition column.
  3. Otherwise, the table has no time column.

Removal detection skips a table with no time column: it logs one INFO line naming the table, runs no query, and reports no error. Additive detection uses the same time column to window and order its sample query, and every reproduce SELECT filters on it. A table without one gets an unordered LIMIT 5 for both.

The windowed queries compare the time column with a timestamp parameter, so the destination must support that comparison for the column it reports as the partition column. An unsupported comparison is reported for that table only, through the sink's error path, and the sweep continues: removal yields no coverage for the table, while additive still reports its finding, without samples.

Findings and alert-sink routing

Every finding and every reconciler-internal error goes through the AlertSink protocol — the contract of the alert_sink plugin axis (emit_drift, emit_error, flush), so no alerting SDK ever loads unless you configured its sink. Sinks are selected per project, every configured sink receives every event, and per-sink non-secret options live in a table named after the sink:

[dlt_ops]
alert_sinks = ["logging", "sentry"]

[dlt_ops.alert_sink.sentry]
environment = "prod"

The default is ["logging"] — the core sink that needs zero configuration and writes one structured log line per event (shown in the additive demo above), so a zero-config project still surfaces drift in whatever collects its logs.

The sentry sink ships with the [sentry] extra and reads its DSN from dlt secrets ([alert_sinks.sentry] dsn in .dlt/secrets.toml; the bare SENTRY_DSN environment variable is deliberately not read — one source of truth). It fingerprints findings as ["schema-drift", pipeline, source, resource], so additive and removal drift on the same resource collapse into one Sentry issue by design — one model PR closes both — with a drift_type tag to disambiguate, and reconciler-internal errors land under a separate fingerprint so a drift triage never wades through reconciler bugs.

Third-party sinks register on the same axis — see plugins and the plugin-writing guide.

Sink plumbing follows the observability side of the failure-semantics contract:

  • One failure never starves the rest — one sink raising never crashes the sweep or the other sinks.
  • A sink that fails to load is dropped for the invocation — validate's alert_sink_registered rule and the Tier-2 preflight are the enforcement points for misconfiguration.
  • Emission is never silently lost — if every configured sink drops, emission falls back to the core logging sink; an explicitly empty alert_sinks = [] disables emission on purpose.
  • Sinks flush on exit — every public entry point flushes on the way out, bounding background transports (Sentry's queue) before a short-lived CLI or orchestrator task exits.
  • --dry-run outranks everything — it suppresses all emission (drift and error events alike) while still printing and returning the findings.

Read-only, and what failure means

The reconciler only ever reads: live column metadata and windowed SELECTs, both through the DestinationAdapter boundary in canonical SQL — which also makes reconcile a full-tier verb that refuses on a core-tier destination rather than half-working (the refusal is shown in failure semantics).

Because it can only observe, its exit code reports on the reconciler, not on your schema: findings alone exit 0 — drift is an alert to route, not a command failure to retry — and only a reconciler error (unreachable destination, unknown source) exits 1.

Failures are isolated at every level so one broken resource cannot hide drift elsewhere:

  • a per-resource failure is reported through the sink's error path and the sweep continues;
  • a source-level failure lands in that source's error field; a failed schema fetch is also reported once through the sink's error path;
  • reconcile --all gives every discovered source its own result block, each resolving — and reconciling against — its own destination and dataset from the config chain, so multi-destination projects sweep without cross-destination credentials.

The sink owns the error report. When the reconciler hands an error to the sink's error path, it also logs one WARNING line with the exception text and no traceback. The logging sink writes its own ERROR line with the traceback. Because the reconciler's own line is a WARNING, a process-wide logging integration that turns ERROR records into events, such as Sentry's, does not report the same failure a second time.

A reconcile --all sweep, one source drifted and one clean:

dlt-ops pipeline reconcile --all --dry-run

Source: github_events_api  |  Findings: 1  |  Duration: 0.21s
  • events: additive drift (1 column(s))
      Columns: payload_size
      First seen: 2026-07-16T16:25:42.081234+00:00

Source: github_events_full  |  Findings: 0  |  Duration: 0.02s
  ✓ No drift

The same machinery is importable for orchestrated sweeps — reconcile_source, reconcile_all, detect_removal, plus the DriftFinding / ReconcileResult models and the AlertSink protocol, all public exports of dlt_ops — returning findings as data so a scheduled task can post-process instead of parsing CLI output. There is no built-in scheduler for reconciliation, deliberately: it is one read-only verb, and it runs on whatever cadence your cron or orchestrator gives it.

Where next