MongoDB Change Streams for Real-Time Replication

MongoDB change streams solve a specific engineering problem: how to know, in real time and without guesswork, that a document changed. The method teams reached for before change streams existed, polling the database on a timer, fails in ways that are structural rather than incidental, and understanding why it fails is the fastest way to understand what change streams actually buy you.

Why Polling for MongoDB Changes Fails

Polling asks a database the same question over and over: has anything changed since the last check? The trade-off baked into that question has no good answer. There's no setting that fixes both sides of that trade at once, because the trade is built into the approach.

Polling also depends on an assumption that rarely holds across a real codebase: every document needs a reliably maintained updatedAt field, written correctly on every code path that touches the collection. A find() against a timestamp field returns rows that exist; it has no way to report the absence of a row that used to be there. Any pipeline built on polling is structurally blind to deletions, full stop as a design property, not a tuning problem.

Before change streams existed, teams who needed real-time visibility into MongoDB writes tailed the oplog directly. It changed between MongoDB versions, and teams tailing it directly had to track those changes themselves, handle oplog rollover (the point at which old entries get overwritten), and build their own logic for sharded clusters where the oplog lives separately on every shard. It was a workable hack for teams with the engineering capacity to maintain it, and a liability for everyone else.

Change Streams and the Oplog

Change streams give MongoDB a supported, stable API for what oplog tailing was trying to accomplish: ordered, filtered notification of every write, without requiring the application to understand the oplog's internal format.

The oplog itself is a capped collection that lives in the local database on every replica set member. A change stream consumer is, mechanically, just another reader of that same stream. Subscribing to change events adds close to no additional overhead: the write to the oplog was happening anyway for replication purposes, and the change stream mechanism just gives applications a structured way to read it.

Change streams can be opened at three different scopes; the right one depends on how much of the deployment needs visibility. A standalone mongod has no oplog at all, so a replica set is a hard requirement, even a single-node one with replSetName configured for local development.

A basic subscription looks like this in one driver:

const changeStream = collection.watch();
changeStream.on('change', (event) => {
console.log(event.operationType, event.documentKey);
});

Every event that comes through carries a consistent shape: operationType (insert, update, replace, delete, drop, rename, or invalidate), clusterTime, ns for the namespace, documentKey, and fullDocument where applicable. The aggregation framework sits in front of all of this as the filtering layer: stages like $match, $project, $addFields, $replaceRoot, and $unset run on the server, before events ever leave MongoDB. A collection taking tens of thousands of writes a second can still deliver a narrow, targeted stream to a consumer, because the filtering happens before the network, not after.

One more property underpins everything downstream: change stream events are only delivered once they've been majority-committed. That means every event a consumer reads has already been replicated to a majority of voting members, and it will survive a primary failover. A consumer never has to worry about processing an event for a write that later gets rolled back.

The fullDocument Options

Change events don't automatically carry the full document; the setting that controls this, fullDocument, is a real consistency decision, not a minor configuration knob. Getting it wrong doesn't throw an error. Getting it wrong produces subtly incorrect data that appears only under load, which is worse.

For insert events, this question doesn't come up: fullDocument is always the complete document, since there's nothing to look up. For update events, the picture changes. Setting fullDocument: "updateLookup" tells MongoDB to run a findOne for the document's current state at the moment the consumer reads the event:

const changeStream = collection.watch([], { fullDocument: 'updateLookup' });
changeStream.on('change', (change) => {
if (change.operationType === 'update') {
console.log('Full updated document:', change.fullDocument);
}
});

That lookup happens after the event was emitted, at some later moment when the consumer reads it, so a fast series of updates can produce a mismatch: a second update may land before the consumer gets around to the lookup for the first event, so the document returned reflects a later state than the one the event was actually describing. The fix is to carry a version field, such as updatedAt, and apply incoming updates only when the version is newer than what's already stored locally. That one check prevents an out-of-order lookup from overwriting newer data with older data.

Getting the document's state before a write requires a different setting: fullDocumentBeforeChange: "required". This returns the pre-image, the document as it looked before the write happened, but it only works if changeStreamPreAndPostImages: true is enabled on the collection, either at creation or afterward via collMod:

db.runCommand({
collMod: "orders",
changeStreamPreAndPostImages: { enabled: true }
});

Pre-images cost storage, and that cost is paid by the database, so it's worth enabling only on collections where the before-state genuinely matters downstream, such as audit logging or change-tracking systems that need to show what was overwritten. Pre-image capture is the only way to recover what a deleted document actually contained, since once the document is gone, there's no live state left to look up.

The practical decision tree is short: if a pipeline needs to know what a document looked like before a write, turn on pre-images and accept the storage cost. If it needs the current state after a write, use updateLookup and guard it with a version check. If it's an insert, there's nothing to decide.

Resume tokens and how change streams survive consumer failures and primary elections

A tailed oplog with no notion of position is fragile by construction: disconnect the consumer and there's no principled way to know where to pick back up. Resume tokens solve that problem directly. Every change event carries an opaque _id field that functions as its resume token, encoding the cluster time and the event's exact position in the oplog. A consumer holding that token can restart from that precise logical point, regardless of whether the interruption was a dropped network connection, an application crash, or a MongoDB primary election.

The correct pattern is to persist the resume token in a checkpoint document, stored in the same MongoDB cluster the change stream is reading from, so the checkpoint inherits the same durability and backup guarantees as the data it's tracking. Checkpointing only after the destination write succeeds closes that gap.

There are two ways to resume a stream from a token. resumeAfter picks up from the position right after the given token and covers the normal restart case. startAfter is for a different situation: when a stream has been closed by an invalidate event, such as a collection drop or rename, and needs to continue against a new namespace. Because every change event is idempotent, reapplying a sequence of events from any earlier point reaches the same consistent end state, which gives operators real flexibility when recovering from an incident.

At-least-once delivery and how to make consumers idempotent

Change streams deliver events at-least-once. They do not guarantee exactly-once delivery, and treating them as if they do will eventually produce duplicate writes somewhere in the pipeline. This is the guarantee the system actually makes, and building for it correctly means building idempotent consumers.

The fix lives entirely in the consumer. Processing the same event twice needs to produce the same end result as processing it once. For an Elasticsearch destination, this is handled with version_type=external, which rejects any write carrying a version number lower than what's already stored:

await es.index({
index: "products",
id: doc._id.toString(),
version: doc.version,

document: doc
});

The oplog window: the one constraint that can silently break your pipeline

The oplog is a capped collection with a fixed size, and it eventually starts overwriting its oldest entries to make room for new ones. That fixed size defines a window of history the oplog can hold at any given time, and if a consumer is down longer than that window, its resume token points to an oplog position that no longer exists. At that point, the pipeline can't simply reconnect. It has to be rebuilt from a fresh snapshot of the source data, and the failure mode that leads there gives almost no warning before it happens.

Default sizing makes this risk easy to underestimate. On 64-bit macOS, the default is a fixed 192 MB regardless of available disk space. On a collection under heavy write load, that percentage of disk space can translate into a window of just a few hours, far shorter than most teams assume when they first set up their replica set.

Expedia Group ran into this directly. Change streams had to be restarted after the replay to re-sync the cache with the underlying documents. It's a concrete illustration of what "oplog exhaustion" actually costs: not a warning, not a graceful degradation, but lost updates that required manual intervention to repair.

When a resume token has expired, resumeAfter fails. The metric that actually matters here is the oplog window remaining minus the age of the last processed token, and it needs an alert set before that number hits zero, not after.

Sharded Cluster Ordering in Cross-Shard CDC Pipelines

Teams running MongoDB as a single replica set can skip this section; the ordering caveat only applies once data is distributed across shards. On a sharded cluster, change stream events are merged across shards by cluster time, not by the order in which an application actually issued its writes. Two writes sent from the same application, to documents that happen to live on different shards, can be interleaved in a different sequence than the one the application wrote them in, because the mongos layer merges independent per-shard streams.

MongoDB's global logical clock does guarantee total ordering across the cluster in the sense that events are always delivered in the order they were applied internally, even across shards. For pipelines that require strict per-document ordering, the practical fix is to watch by document key, or to pre-shard the consumer so that each shard's stream is read and processed independently.

Schema evolution in MongoDB CDC: why the flexible document model creates a specific downstream hazard

MongoDB's lack of an enforced schema is what makes it fast to build applications against, and it turns into a reliability problem for any CDC pipeline reading its change stream. A relational database emits a DDL event when a table's structure changes, giving downstream consumers a clear signal to react to. MongoDB gives no equivalent signal. A new field, a renamed field, or an array element that suddenly has a different shape can all appear in a document with no database-level event marking that something structural just shifted.

A safe sequence for introducing a schema change keeps producers and consumers from drifting out of sync with each other. Following that order avoids the common failure where a consumer crashes on an unfamiliar field that appeared before anything downstream was ready for it.

A few practices make this manageable over time.

Lag monitoring, oplog consumer health, and the operational instrumentation a production pipeline requires

A change stream pipeline running without lag monitoring is, in effect, waiting for the oplog exhaustion scenario described earlier to happen to it. The single number that determines whether a pipeline will survive a consumer outage is the gap between the last event it successfully processed and the current leading edge of the oplog, and that number needs to be visible at all times, not discovered after the fact.

A handful of signals matter enough to track continuously. Oplog window remaining versus the age of the last processed token is the core health number discussed above. And invalidate events, which end a change stream cursor outright when a collection is dropped or renamed, need explicit handling in consumer code. Without a handler for them, a consumer stops silently, and a dashboard or downstream system quietly goes stale with no obvious cause.

Long-running transactions create a lag hazard that needs watching separately from oplog window size. A change stream only emits events for a transaction after it commits, so a transaction that runs for several minutes holds up every downstream consumer for that same duration, even though writes are actively occurring inside it the whole time. From the outside, that looks like the pipeline has stalled, when in fact it's waiting on a single long transaction to finish.

Where Managed Replication Fits

Everything covered here, resume token persistence, idempotent destination writes, oplog window sizing and alerting, sharded cluster ordering, and schema evolution sequencing, is infrastructure a team has to design, build, and keep operating if it implements change streams directly. None of it is optional once a pipeline runs in production; it's the set of problems that decides whether that pipeline survives a bad week.

DMS is built primarily for migration rather than ongoing real-time sync, and it requires one endpoint to be inside AWS, which limits its fit for architectures replicating directly between other clouds, such as GCP and Azure, without AWS in the path.

Whatever direction a team chooses, the value of understanding the internals doesn't depend on which option it picks. A team that understands resume tokens, oplog windows, delivery guarantees, and schema evolution hazards is equipped to evaluate any replication tool, managed or self-built, by asking the one question that actually matters: what happens when this fails, and who notices first?