Exactly-Once Delivery Semantics in CDC Pipelines

Exactly-once delivery semantics in CDC pipelines are a layered guarantee. Durable capture, safe log-position advancement, ordered processing, and idempotent sink writes must work together. That guarantee must also survive schema changes and the transition from an initial snapshot to streaming, or a pipeline can deliver events reliably while silently corrupting destination data.

Why exactly-once is a compound problem, not a single switch

A connector crashes after it sends a change downstream but before it records how far it got. On restart, it retries the event, although the destination may already have applied it. A consumer restart after a deploy creates the same uncertainty if its checkpoint does not reflect durable downstream work. Neither the source database nor the transport layer can resolve that uncertainty alone.

Three nested failure modes explain the problem, namely the durability gap, the ordering constraint, and storage accumulation. A crash between receiving and confirming an event can cause loss or replay. Parallel workers can apply related changes in the wrong order and produce a destination state that no valid source ordering could have produced. Meanwhile, a lagging consumer forces the source to retain log data, and truncating that log to recover disk space destroys the events needed for recovery.

Retaining every event is not enough if workers apply those events out of order, and ordered delivery is not enough if retries change the result. A pipeline that leaves these gaps unresolved provides at-least-once delivery at best, with duplicates or out-of-order state still possible at the destination.

How log-based CDC creates the raw material for the guarantee

Log-based CDC provides the necessary foundation because it preserves the complete, ordered sequence of changes, including deletes, with minimal overhead on the source system. That sequence supplies both the changes to process and the positions from which processing can resume. Without it, the pipeline lacks the raw material needed to account for changes across a failure. The log does not, however, make downstream processing exactly-once by itself.

MongoDB Change Streams expose an ordered stream of write operations from a cluster. Consumers track their position through resume tokens stored durably in a MongoDB collection, Redis, or a file. DynamoDB Streams deliver changes to items in order within a shard, and each record appears exactly once at the stream level. Yet a Lambda crash mid-batch can cause redelivery to the consumer, so stream-level uniqueness does not remove the need for consumer idempotency.

Consumers need the right database permissions, and log retention must last through the CDC processing window. In Postgres, a replication slot pins WAL on the primary until the consumer advances its confirmed LSN, allowing a lagging consumer to cause unbounded disk accumulation. Postgres logical decoding also leaves a separate gap because DDL does not appear in the stream. The event sequence is therefore a foundation for recovery.

The durability gap: why advancing the log position is the most dangerous moment

The critical durability decision is when the consumer tells the source that earlier log data is no longer needed. In Postgres, advancing confirmed_flush_lsn tells the database that it is safe to remove WAL up to that point. If the consumer advances that position before persisting the corresponding events downstream, a crash can leave neither a durable destination write nor a source record available for replay. The checkpoint has claimed progress that the pipeline has not actually secured.

The safe sequence is to persist the event to the destination or an intermediate durable store before advancing the source position. This closes the loss window, but it leaves a replay window. If persistence succeeds and the consumer crashes before confirming progress, recovery can read the event again. That is why durability discipline and idempotency are complementary approaches.

Multiple destinations make checkpoint advancement more demanding. Under the minimum unflushed LSN pattern, the safe position is the minimum confirmed LSN across all destinations. Advancing to the highest confirmed position would release source history that a lagging destination still needs.

Kafka offers mechanisms for the broker portion of this problem. Its idempotent producer and transactional API, available since Kafka 0.11, support writes that survive broker failures without creating duplicate records. The producer assigns sequence numbers per topic-partition, and the broker uses its assigned Producer ID to reject duplicates. These mechanisms protect writes to Kafka, but they do not replace the discipline required when acknowledging the source log or writing to destinations.

The ordering problem: how parallel processing breaks sequential guarantees

Once events are durable, the next question is whether they reach the destination in a valid order. A source log delivers changes serially, but distributing those changes across worker threads can discard that sequence. A later change may finish before an earlier change for the same record. The pipeline can then preserve every event and still produce the wrong state.

An UPDATE followed by a DELETE illustrates the failure. If the sink receives them in reverse order, the result can leave intact a row that should have been deleted. The defect is not a missing event but a broken relationship between events. A row count check cannot establish that this ordering relationship was preserved.

Partition-by-primary-key keeps that relationship intact while allowing parallel work. All events for a given record go to the same processing lane, where they arrive in source order. Events for different records can run in parallel while those for each record stay in order. In-flight tracking sets belong to the progress-tracking work around this parallel execution, but they do not replace ordered routing.

For Kafka-based pipelines, the practical decision is the partition key. A consistent key, typically the source record’s primary key, sends all changes for that record to the same partition. Kafka keeps events in order within a partition, so how the pipeline sets up its partitions helps keep the data correct.

Idempotent sinks: the final layer that makes replay safe

Durability and ordering cannot eliminate every replay. Connector restarts, broker failovers, and checkpoint recovery can all bring an already-processed event back to the sink. The destination must therefore give the same result even when it applies an event more than once, turning safe recovery into an exactly-once effect on stored state.

At a warehouse, one implementation is to express CDC events as MERGE operations keyed on the source record’s primary key. An INSERT that arrives twice then produces the same state as one INSERT. An UPDATE can also be idempotent when it carries the full record state, because repeating a state assignment differs from repeating a delta that changes that state. The MERGE pattern must account for insert, update, and delete operations.

Another way to make DynamoDB Streams consumers idempotent is to use the stream record’s sequence number as an idempotency key. The key belongs in a durable store with a TTL longer than the maximum retry window. This addresses the gap between a record appearing exactly once in the stream and that record being delivered more than once to a consumer. The deduplication state must remain available for as long as a retry can occur.

Dead-letter queues are part of this recovery design. Events that fail repeatedly must go to a DLQ. The queue preserves them for inspection and reprocessing without holding up downstream commits. But replay-safe writes still need the sink to read the event correctly, so the guarantee also depends on schema compatibility.

Schema changes as a silent threat to exactly-once guarantees

A pipeline can preserve durability, ordering, and idempotency while misinterpreting the data it delivers. A developer may add a column, rename a field, or change a data type without producing an explicit schema-change event in Postgres logical decoding. The consumer encounters the change as a structural surprise.

Different changes place different demands on consumers. Forward-compatible consumers commonly handle a new nullable column by ignoring fields they do not know. Adding a NOT NULL column requires a DEFAULT value so that replayed events do not break the consumer. For renames, drops, and type changes, source and consumer teams need to work together and often backfill the destination. Treating these changes as the same hides the work needed to keep old events readable.

A schema registry makes that contract explicit. Avro serialization with Schema Registry is the most widely adopted approach in Kafka-based pipelines, while Protobuf and JSON Schema are also supported. Schema versions become artifacts that consumers can resolve at read time.

Consumers should handle missing or extra fields gracefully from the start. Schema evolution belongs in the initial pipeline design as a planned workflow. Advanced CDC tools can detect DDL changes outside the logical decoding stream and apply them to the destination. That capability addresses a gap that duplicate handling and ordered delivery cannot close.

The initial snapshot problem and how the snapshot-then-stream handoff works

The transition from existing data to ongoing changes tests all of these guarantees at once. During an initial snapshot, the source continues to change. Those writes can fall into a gap between the snapshot and streaming, or appear in both. A pipeline that works correctly in steady state can still start with wrong data at the destination if its handoff is poorly defined.

The required sequence is to mark the log position first and then take the snapshot. Changes made while the snapshot runs are then available in the log from the marked position onward. Streaming starts from that position rather than from an uncertain boundary established after the snapshot. Combined with idempotent application, this preserves changes made during the initial load without counting overlap twice.

Reversing the sequence removes that clear boundary. If the snapshot starts before the streaming position is established, changes during the snapshot window have an uncertain status. They may appear in the snapshot, in the stream, or in both. The handoff must therefore be designed as part of recovery.

MongoDB migration architectures use Change Streams to capture writes to the source cluster during the snapshot phase. An idempotent consumer, a durable checkpoint store tracking resume tokens, and a dead-letter queue support the transition without loss. DynamoDB Streams impose a harder time constraint because records expire after 24 hours, with no configuration option. If a snapshot takes longer than that window or a consumer falls too far behind, records are lost for good. Pipeline design must account for that ceiling before relying on the stream to bridge the initial load.

Where managed replication services fit into this engineering picture

Building durable capture, LSN-safe advancement, ordered parallel delivery, and idempotent sink writes from scratch is a multi-month engineering project. It also needs ongoing upkeep as databases, brokers, and warehouse targets each change on their own. Choosing an in-house implementation therefore means taking ownership of both the correctness mechanisms and the conditions that keep them working.

At the source, that work includes managing the replication slot, monitoring WAL retention, and tracking the LSN across all destinations. In processing and storage, it includes primary-key-keyed partitioning, in-flight tracking sets, Redis deduplication or warehouse MERGE logic, and retry state that survives restarts. Dead-letter queues, schema change detection, and snapshot handoff sequencing complete the recovery path. Postgres failover adds slot recovery before PostgreSQL 17 or sync_replication_slots configuration for PostgreSQL 17+.

AWS DMS is frequently evaluated as an alternative to building this stack. It supports full load, CDC only, and full load + CDC modes and is effective for one-time migrations. For long-running CDC workloads, its documented issues include memory saturation, task stalls during write spikes, and schema change handling. It also has no built-in logic to automate merges for modern cloud warehouses. Those limits matter because supporting a migration takes different work from keeping exactly-once behavior at the destination over time.

A replication tool that takes more than an hour to configure is still deferring engineering cost. More broadly, a managed service changes who implements the guarantees the pipeline still needs. The decisive question remains whether every layer preserves correct destination state through failure, replay, schema change, and the initial handoff.