Databricks Delta Lake Real-Time Ingestion Patterns

A pipeline that ran in twenty minutes at launch can take six hours at production scale without anyone touching a line of code, because a full load reads every row on every run regardless of what actually changed. Delta Lake offers a spectrum of patterns built to answer a different, narrower question: what changed since the last run, and each pattern fits a different combination of source behavior and acceptable trade-off.

Why full loads degrade as data grows

Source tables that hold tens of millions of rows at launch routinely reach hundreds of millions within months of going into production, and the pipeline built and tested at the smaller scale keeps running long after that scale stops applying. Three costs compound as the table grows: compute spend per run scales with row count, data freshness is bounded by how long the full load takes to finish, and long write locks create downstream contention that queues every reader waiting on the table. The inefficiency at the center of all three is simple to state: when only a small fraction of rows change between runs, reprocessing the entire table means paying for nearly all of that work just to confirm that most of it didn't need to happen. Incremental ingestion reframes the job. Instead of reading everything, the pipeline asks what changed since the last run, and that question turns out to have several valid answers depending on the source system and the trade-offs a team is willing to accept. Delta Lake supports three of those answers directly, each suited to a different situation, and choosing among them requires understanding what each one costs and what each one refuses to do.

Delta Lake's transaction log and incremental processing

All three patterns rest on the same mechanical foundation: the DeltaLog, a transaction log that records every change made to a table. That log is what separates a Delta table from a directory of Parquet files with a schema attached to it, and it's the reason incremental processing is possible at all rather than something bolted on after the fact.

The DeltaLog stores its change history as JSON files, and for every ten of those JSON log files, Delta writes a Parquet checkpoint and updates a _last_checkpoint pointer. Reads start from the most recent checkpoint and replay forward from there, so the log stays queryable without forcing a scan of the table's full history on every read. Delta also captures column-level statistics, min, max, and null count, for the first 32 columns by default, and the query planner uses those statistics to skip data files that cannot possibly contain the rows a query is asking for. That file-skipping behavior is a large part of why incremental reads stay fast even as a table grows into the hundreds of millions of rows.

Change Data Feed sits on top of this same log as a logical abstraction rather than a separate storage mechanism. For UPDATE, DELETE, and MERGE operations, Delta writes change records to a _change_data folder alongside the table, but for insert-only operations and full partition deletes, it computes the change directly from the transaction log without writing anything to that folder. Every CDF record carries three metadata fields, _change_type (with values of "insert", "update_preimage", "update_postimage", or "delete"), _commit_version, and _commit_timestamp, which together let a downstream consumer reconstruct what changed and when it changed. The log has a real limit, too: it tracks row-level data changes but does not capture schema changes such as a newly added column, so DDL events have to be handled at the ingestion layer before CDF ever sees them. That becomes a production concern the moment a source schema evolves. Understanding this foundation matters because pattern choice doesn't just set latency, it determines how the table writes files, how those files get scanned, and how query performance behaves as the table keeps growing.

Watermark filtering: the lowest-complexity pattern

Watermark filtering sits at the simple end of the spectrum, and its mechanism is easy to describe. The pipeline reads a stored watermark value, a timestamp or an auto-incrementing ID, queries the source for rows where that column exceeds the watermark, processes only those rows, and then saves the new high-water mark for the next run. The pattern works correctly only when the watermark column increases monotonically and rows are never modified after they're inserted. Event streams, log tables, and clickstream data all fit that description cleanly, and watermark filtering shows up so often in those pipelines for that reason.

It also fails in four specific, predictable ways. A deleted source row leaves no updated timestamp behind, so watermark filtering cannot detect deletes. It misses updates entirely, since an updated row retains its original creation timestamp and watermark filtering never re-reads a row it has already passed. Late-arriving data with an older timestamp gets silently dropped, because it falls below the current watermark the moment it arrives. And re-processed or corrected historical records stay invisible to the pattern for the same reason.

Implementing watermark filtering correctly requires a reliable checkpoint store. The watermark has to persist across runs and update only after a successful write, or a retry will either reprocess rows it already handled or skip rows it never got to, and that checkpoint-store detail is the most common gap in production implementations of this pattern. For genuinely append-only sources, though, watermark filtering is the right tool for the job: it adds no schema requirements on the target table, demands no Delta-specific table properties, and keeps operational complexity close to zero. The moment a source allows updates or deletes, that simplicity runs out, and the pipeline needs a pattern built to handle a fuller change payload.

MERGE INTO for direct upserts: handling inserts, updates, and deletes atomically

MERGE INTO picks up where watermark filtering stops. It handles inserts, updates, and deletes atomically in a single operation, which solves the exact gap watermark filtering leaves open, though its own performance degrades under high-frequency small upserts in a way that matters once a pipeline reaches real production scale. The mechanism depends on the source sending a CDC payload that includes explicit operation flags. MERGE matches each incoming record against the target table by a business key and applies a DELETE, UPDATE, or INSERT according to the flag attached to that record, all inside one atomic operation. Delta's ACID guarantees back that operation: the MERGE either completes in full or rolls back completely, with no partial writes and no inconsistent intermediate state visible to concurrent readers.

The performance cost is specific: an engineer sizing a pipeline needs more than "performance degrades. Each MERGE has to scan the target table's files to find matching keys. At high update frequencies, that file-scan cost accumulates run over run, write amplification increases as the same files get rewritten repeatedly, and the table can end up with a large number of small files that slow down every subsequent read against it. The 2026 Delta Lake incremental guide flags that degradation pattern for high-frequency small upserts.

MERGE INTO also has a hard input requirement: the source has to deliver a structured CDC payload with explicit operation flags. At moderate upsert volumes, none of this is disqualifying. MERGE INTO remains a well-understood, widely supported pattern that works against any Delta target without enabling any additional table properties. The real cost appears in the edge cases: teams routinely write 150-line custom MERGE scripts to handle out-of-order events and SCD Type 2 history, and that hand-written logic is where the operational burden concentrates. That specific problem, the growing pile of custom logic wrapped around a MERGE statement, is what AUTO CDC in Lakeflow Declarative Pipelines is built to remove. There's a related but distinct pattern that handles propagation between Delta tables that are already inside the lakehouse.

Change Data Feed as a downstream propagation mechanism between Delta tables

Change Data Feed solves a different problem than either of the previous two patterns, a problem in its own right rather than a lesser version of upstream CDC. CDF is not a replacement for CDC coming out of an operational database; it's a propagation mechanism for moving changes efficiently between Delta tables that already live inside the lakehouse. That constraint is the first thing to understand about it, because misapplying CDF as a substitute for upstream extraction is the most common way teams get this wrong.

CDF has to be explicitly enabled on the source Delta table through delta.enableChangeDataFeed = true. Once it's on, consumers read changes by version range or by timestamp range using readChangeFeed = true. The four _change_type values, insert, update_preimage, update_postimage, and delete, give a downstream consumer the full before-and-after picture of every change rather than just the final resting state of a row.

The structural context where CDF earns its place is the medallion architecture. The Bronze layer captures every incoming change as it arrives, and CDF lets Silver and Gold consumers read only what changed since the last version they processed. A job that would otherwise reprocess the full table in six hours becomes a minutes-long incremental read instead. A team cannot point CDF at Postgres, MySQL, or any non-Delta source and expect it to produce a change feed; that's a different problem, requiring a different ingestion path.

The schema evolution gap described earlier in the DeltaLog internals carries forward here directly. Adding a column to a Bronze table doesn't appear in CDF records, so downstream Silver jobs consuming that feed have to handle DDL changes through schema evolution mechanisms at the ingestion layer rather than expecting CDF to surface them. Delta's schema enforcement helps on the other side of that boundary: it rejects incoming records that violate the table's defined schema, so downstream consumers of CDF can trust that the change records they receive are structurally consistent. CDF is directly applicable to continuous ML retraining pipelines and data quality validation workflows where only changed rows need to flow through the validation or feature pipeline.

AUTO CDC in Lakeflow Declarative Pipelines: where the operational complexity concentrates

The hardest part of a production CDC pipeline is rarely the happy path, but the accumulation of edge cases: out-of-order events, SCD Type 2 history, and schema widening, all of which AUTO CDC addresses with a declarative API that shrinks the amount of custom code where bugs can hide. The concrete reduction is dramatic: a hand-coded MERGE script that can run to 150 lines becomes a 7-line declarative SQL specification, with the engine handling deduplication, ordering, and key matching internally rather than leaving each of those to be reimplemented per pipeline.

Out-of-order events are handled by a user-specified sequence column rather than arrival order, so a record that arrives late but carries an earlier timestamp doesn't corrupt the current state of the table. Slowly changing dimensions are controlled the same way: Type 1 overwrite behavior and Type 2 history-retaining behavior are both set with a single keyword, which eliminates the custom window logic that Type 2 normally demands. For the large population of source systems that don't expose a real-time change stream at all, AUTO CDC FROM SNAPSHOT compares consecutive table snapshots, identifies which rows were inserted, updated, or deleted between them, generates a synthetic change feed from those changes, and applies the same SCD logic on top of it. That capability matters specifically because a real-time feed isn't universal. Plenty of production sources simply don't offer one, and snapshot comparison is what makes AUTO CDC usable against them anyway.

As of June 2026, AUTO CDC target tables support liquid clustering, enabled either by primary key or through the ingestion spec, which improves query performance on CDC targets without manual clustering maintenance. As of May 2026, AUTO CDC supports bitemporal tracking, maintaining both system time (when a record was stored) and valid time (when it was true in the real world), which enables point-in-time correctness for financial, audit, and regulatory workloads. As of February 2026, Lakeflow pipelines support type widening, letting a column type broaden safely, INT to LONG, FLOAT to DOUBLE, without forcing a full pipeline reset, removing a disruption that has historically forced teams to rebuild pipelines over a change as small as a column type.

The strongest evidence for what this collapses in practice comes from Block, formerly Square. Block uses AutoCDC in Lakeflow Spark Declarative Pipelines to replace hand-coded CDC and merge logic, and according to Yue Zhang, Staff Software Engineer, Data Foundations at Block, "the time required to define and develop a streaming pipeline has gone from days to hours". That's the practical measure of what a declarative CDC specification buys a team: not a theoretical simplification, but a change in how long it takes to stand up a pipeline that used to require days of custom MERGE logic and edge-case handling.

Connecting upstream operational databases to Delta Lake: what each source requires

Every pattern described so far, watermark filtering, MERGE INTO, CDF, AUTO CDC, is only as reliable as the change feed arriving at its input. Each upstream operational database exposes a fundamentally different mechanism for surfacing changes, and each comes with its own configuration requirements that a team has to satisfy before any of Delta's internal patterns can do their job.

PostgreSQL exposes changes through logical replication, which decodes the write-ahead log (WAL) into row-level change events and ships them over a publish-subscribe model. Unlike physical replication, logical replication can cross major PostgreSQL versions and replicate a subset of tables rather than the whole database. Lakeflow Connect requires wal_level = logical on the source and supports PostgreSQL version 13 and later. Its PostgreSQL connector covers AWS RDS PostgreSQL, Aurora PostgreSQL, Amazon EC2, Azure Database for PostgreSQL, Azure virtual machines, GCP Cloud SQL for PostgreSQL, and on-premises PostgreSQL reached over Azure ExpressRoute, AWS Direct Connect, or VPN. Databricks tracks both extraction and application position, so if a pipeline is interrupted, it can resume from that tracked point as long as the replication slot and the WAL data it depends on are still present on the source.

Application telemetry follows a different path. Zerobus Ingest, part of Lakeflow Connect, lets applications push events directly to a Delta table over gRPC, with no message bus and no Structured Streaming job sitting in between. It delivers sub-5-second latency, up to 100 MB/sec per connection, and the data is immediately queryable in Unity Catalog once it lands.

MongoDB and DynamoDB sit in a less settled position. Official Lakeflow Connect support for DynamoDB streams isn't confirmed in available sources, and MongoDB has been announced as an upcoming managed connector but isn't yet listed as generally available. That gap is a reasonable thing to plan around: any team building on Postgres or on application telemetry has a documented, supported path into Delta today, while teams on MongoDB or DynamoDB are working with a narrower and less finished set of options.