Six Source Databases One Warehouse Why the Replication Topology Matters More Than the Tool Brand

Fan-in CDC replication from multiple operational databases into a single warehouse requires a composition layer in addition to a connector for each source. With six heterogeneous databases, reliable correctness depends on how the warehouse end handles ordering conflicts, schema drift collision, lag asymmetry, and snapshot-to-streaming handoff. Tool choice governs capture, but topology governs whether the combined data represents a coherent application state.

Why fan-in topology is a distinct engineering problem from change capture

A team replaces a replication tool because of broken schemas, duplicate records, or stale query results, then runs into the same failures again. The new tool may capture changes correctly and still leave the warehouse with data it cannot trust. The failure sits beyond the individual connector. It appears when the target combines streams whose relationships the replication architecture does not govern.

Change data capture shows what changed in a source since the last cycle. Log-based CDC reads the transaction log, query-based CDC polls a timestamp column, and trigger-based CDC fires on every write. None determines how changes from independent databases should interact when they reach the same warehouse table.

Topology is the shape of that data flow, including which sources connect to which destinations and in what configuration. Fan-in means many sources converge on one target. That convergence creates four distinct risks, namely ordering conflicts, schema drift collision, lag asymmetry, and snapshot-to-streaming handoff. Some also exist within single-source replication, but combining sources introduces conflicts that no connector can resolve from its own stream alone.

How the six source databases each produce a structurally different change stream

PostgreSQL exposes row-level operations through WAL-based logical decoding, with replication slots anchoring each consumer’s position. A stalled slot keeps WAL segments, so it uses more disk space as more WAL is generated. Slot lag can also hold back the xmin horizon and constrain autovacuum cleanup. DDL does not automatically replicate through native logical replication, so a source column addition can leave the consumer receiving data whose schema change it has not received.

MongoDB change streams read the oplog used for replication between replica set members, adding minimal source overhead. Only majority-committed changes enter the stream, but consumer delivery is at-least-once. After a crash, the last event may be processed again depending on when its resume token was saved. This mechanism requires a replica set or sharded cluster because standalone deployments have no oplog.

DynamoDB records inserts, updates, and deletes in streams with a fixed 24-hour retention window. Changes to a single item come through in order, and the stream itself has exactly-once semantics. Consumer processing is a separate guarantee. Lambda processes batches at-least-once, so retries can deliver records already processed and consumers must be idempotent. Once a consumer falls beyond the retention window, it needs a new full snapshot to recover the missing changes.

MySQL exposes changes through its binlog, while Oracle uses redo logs. SQL Server CDC uses change tables populated by a continuously running SQL Server Agent job that reads the transaction log. Its Change Tracking option records only that a row changed. Even the label “log-based” therefore covers different capture paths, while the six inputs differ in ordering scope, replay behavior, retention, and schema handling before they reach the warehouse.

Ordering conflicts: why global event order across six sources cannot be assumed

PostgreSQL has a monotonically advancing LSN, MongoDB has an ordered oplog, and DynamoDB preserves per-item order within its retention window. An LSN cannot establish whether a change happened before an event identified by another database’s sequence, even when both connectors faithfully preserve their source order.

The conflict arises when the sink has to apply related changes from different sources. Consider a customer record updated in one relational database at the same wall-clock moment that a related order is inserted in MySQL. Connector lag, network jitter, and batch size determine arrival order. They do not express the application’s causal intent. Without additional metadata, the warehouse cannot infer the relationship merely from the order in which the records arrive.

Wall-clock timestamps are an appealing substitute for a global sequence, but source clocks can skew. Even synchronized clocks may not distinguish high-frequency writes at the available timestamp granularity. A timestamp is not a causal guarantee. Treating it as one can leave the order unclear and cause phantom conflicts and missed updates. For dashboards and AI agents, the consequence is a warehouse view that supports answers correct for the visible records but wrong relative to the application state that generated them.

This is not necessarily a transport defect. A connector can deliver every event in the correct local order while the combined target applies related events in the wrong order. The missing authority spans sources. Replacing a connector leaves that boundary unchanged unless the replacement also changes how the sink governs cross-source order.

Schema drift collision: when independent schema changes from multiple sources meet in one target

A source adds a column, later events include its data, and the warehouse loader must handle the field without breaking the consumers that already use it. PostgreSQL’s native logical replication makes the distinction between data and DDL especially important because the schema change does not propagate automatically. An unprepared loader can drop or misroute the new column. The source’s data and the target’s expectations have diverged.

Fan-in introduces a second, independent source of intent. Suppose source A renames a column while source B continues writing the old name into the same warehouse table through a different schema mapping. Both changes can be locally valid. Yet the target now contains two logical versions of one field, and downstream queries may silently read the wrong one. Neither source team needs to have made a mistake for this collision to occur.

The shared namespace is the point of conflict. A connector following source A cannot decide what source B intends, while source B’s connector cannot arbitrate a rename it does not control. Automated schema evolution can handle single-source additions and renames without resolving that disagreement. It needs knowledge beyond one stream. Detecting the collision requires visibility into the in-flight schema states of all sources that contribute to the target.

A schema coordination layer at the warehouse end provides the place for that decision. Its role differs from simply allowing a loader to add columns automatically. It must reconcile independent mappings before they conflict in the shared target namespace.

Lag asymmetry: what happens when the six streams fall behind at different rates

Six streams each have their own lag value, which leaves the warehouse without one value that shows how fresh its data is. Cross-source queries can combine incompatible states when the streams’ lag values diverge. One current stream cannot compensate for another that is hours behind. A join between a current customer table and an orders table four hours behind mixes two moments rather than representing either the current application or a consistent historical snapshot.

Each source can fall behind for different reasons and with different consequences. With PostgreSQL slot retention, the risk to disk space depends on the bytes of WAL kept, while DynamoDB’s fixed retention window sets a hard limit on how far back recovery can go. MongoDB’s oplog window depends on the size of its ring buffer. High write volume can rotate the oplog past a lagging consumer’s resume token, making a re-snapshot necessary. The same latency values can therefore leave different amounts of room for recovery.

Teams can track replica lag in seconds per connector while omitting bytes retained per replication slot. A consumer that appears only seconds behind can still retain gigabytes of WAL. Latency alone does not describe source pressure. Nor does a set of separate connector dashboards necessarily reveal the temporal mismatch in a query that joins their outputs.

This leaves two different questions unanswered by one lag indicator. The first is whether a consumer can recover before source retention or disk pressure becomes a problem. The second is whether the warehouse combines streams within an acceptable window of one another. A connector may report its own position accurately without answering the second question. No individual stream can correct the relative freshness of the other five.

Snapshot-to-streaming handoff: the transition that breaks most implementations

The snapshot-to-streaming handoff is where most implementations go wrong because the pipeline must connect a full source copy to ongoing changes without losing or duplicating events. The snapshot needs a known log position, such as an LSN, binlog coordinate, or oplog timestamp. Streaming must align with that position. A gap between the snapshot boundary and the stream’s start misses events, while an overlap can produce duplicates. Correct capture on either side does not, by itself, make the boundary correct.

Fan-in compounds that boundary problem because the six sources do not finish their snapshots together. Source A may already be streaming current changes while source B is still copying the state it started with. Both processes can be operating as intended. Yet a query joining them combines current A data with historical B data, and may give no sign that B is not yet live.

Adding a table, introducing another source database, or recovering a failed slot can require a snapshot of only part of the running pipeline. Other streams continue advancing during that work. The warehouse can therefore return to an asymmetric handoff state after it first goes live, so the transition remains a risk to availability beyond setup.

Retention failures can force that return as well. A DynamoDB consumer that falls beyond the 24-hour window cannot resume from its saved position and must take a new snapshot while the other five sources continue. The missing interval cannot be recovered by waiting. The architectural implication is a per-source concept of “stream readiness,” with queries and downstream consumers prevented from reading a source stream while it is in handoff.

Why these four failures survive a tool swap and what must change instead

A duplicate record, broken schema, or stale result can look like a fault in the connector even when capture and delivery worked correctly. Replacing that connector does not repair missing cross-source state. The architectural response is a single composition layer with a global sequence or causal ordering mechanism spanning all six streams. A monotonic write order enforced at the sink must be paired with the information needed to respect causal relationships. The same layer needs a cross-source schema registry, relative lag monitoring, and a consumer-visible “consistency horizon” identifying the latest moment for which all six streams are current. Readiness flags for each source must also control when downstream joins can run during handoff.

These requirements favor a system that manages fan-in directly over six independently operated connectors, because one system must hold the streams’ composition state. A managed service’s source and destination support is therefore only part of its suitability. Shared controls also provide the place to enforce data residency and access policy, with passthrough handling, BAAs, and audit logs across streams supporting the compliance requirement. Before evaluating replication tools, the decisive question is not “which sources and destinations does it support?” but “how does it manage the composition of all six streams at the warehouse end?”