Fan-In CDC Without Custom Pipelines What the Managed Service Model Actually Delivers

Fan-in CDC replication from multiple operational databases into a single warehouse needs log capture for each source, schema handling that works across sources, and writes to the destination that are replay-safe. A managed service turns much of that infrastructure into configuration, but leaves schema policy, sink correctness, and cross-source consistency decisions with the team.

Why fan-in CDC is structurally harder than any single-source pipeline

A hard delete in one production database never reaches the warehouse, while changes from the other sources keep arriving as usual. The shared destination now includes a row that no longer exists upstream. Nothing about the healthy streams reveals that omission. A query joining those streams can therefore return an incorrect result even when most of the pipeline is working.

Fan-in means many sources feeding one destination, unlike fan-out, which distributes changes to multiple consumers. Polling-based capture misses hard deletes, while log-based CDC reads inserts, updates, and deletes straight from the transaction log with almost no load on the source. For this workload, log-based capture is a prerequisite.

Each engine exposes different events, ordering rules, and recovery constraints, while its schema evolves independently of the others. Those differences converge at the shared destination. A column addition can conflict with another source’s column, and a lagging stream can leave joined data partially current without making any individual connector appear incorrect.

How each major source database exposes its change log, and what that means for a multi-source pipeline

PostgreSQL exposes changes through its Write-Ahead Log, or WAL. Its wal_level must be set to logical, because the default replica setting does not contain enough information to reconstruct row-level changes. Changing that setting requires a server restart. Logical decoding also excludes DDL, so an ALTER TABLE requires a separate schema-detection process.

Replication slots create a second operational obligation. A lagging or abandoned slot can retain WAL until the disk fills and PostgreSQL crashes or refuses new writes. Clients receive no error during the preceding accumulation, although PostgreSQL may emit WARNING-level log messages. Log access therefore needs active lifecycle management.

MongoDB Change Streams require a replica set or sharded cluster, because a standalone instance has no oplog to stream. By default, update events have only the fields that changed. The fullDocument option set to 'updateLookup' performs a separate read for the current document, but that read occurs after the event is emitted. The fetched document may already reflect a subsequent change, so a full document is not necessarily the document as it stood at the event.

DynamoDB Streams presents a different boundary again. Each partition has a corresponding stream shard, with ordering inside the shard but no cross-shard ordering guarantee. Events remain available for only 24 hours. An outage that lasts beyond that window calls for a full table re-scan, so recovery depends on both how much time has passed and whether the data is correct. A pipeline reading all three engines must consequently maintain separate cursors, interpret different event shapes, and apply different recovery strategies rather than treating log management as interchangeable infrastructure.

Why schema evolution is the most common production failure in a fan-in pipeline

Independent release schedules make schema drift a continuous operating condition. The log-reading layer must read changed records correctly even when it learns of a change to their structure through a separate detection process. That distinction is especially important for PostgreSQL. If schema inspection misses an alteration, later row events can contain unexpected columns or missing fields even though log capture itself remains healthy.

The difficulty depends on the kind of change. Many managed tools can add columns automatically, but for a rename they must track that it is still the same column, since treating it as a drop-and-add loses history. Drops usually leave the destination column in place while updates to it stop, so downstream consumers need to know it is deprecated. Type changes are harder still. They often require a full re-sync of the affected table.

The Flink CDC 3.x project gives useful names to these policy choices. Its evolve mode applies upstream schema changes to the destination, while lenient converts changes into non-destructive equivalents, such as handling a type change by renaming and adding a column. Its ignore mode suppresses schema changes while continuing to propagate data from unchanged columns. Every pipeline commits to a stance on changed schemas, whether that stance is explicit or embedded in service defaults.

Fan-in adds conflicts that no universal rule resolves. Two sources can independently add columns with the same name but different types, leaving the destination to reconcile incompatible definitions. AWS DMS illustrates the distinction between replication capability and turnkey handling, because schema-change replication typically still requires manual configuration and additional scripts to keep downstream tables consistent. Schema evolution needs a planned workflow with rollback procedures. Teams must decide how type changes and drops will behave before those changes reach production.

What delivery guarantees actually mean when one source in a fan-in pipeline lags or fails

Exactly-once is not an end-to-end property that the transport layer can supply by itself. Retries and duplicates are normal in production CDC, so correctness depends on idempotent writes and deterministic versioning at the sink. The practical target is at-least-once delivery paired with an idempotent destination. For most warehouses, that means MERGE operations keyed on primary keys, so replaying an event produces the same result as processing it once.

If source A is hours ahead of source B, each source’s replicated data can be individually correct while a join combines incompatible temporal states. Independent clocks create a sequencing problem as well. A delete in source A caused by an update in source B can arrive first because of relative lag, and applying events in wall-clock order does not restore causal consistency.

Retention expiry makes that distinction operationally urgent. When a DynamoDB consumer outage lasts longer than the stream window, recovery needs a re-scan, during which the destination can hold both snapshot data and live-stream changes. Recovery must take the other sources into account too, along with loading the table for the failed stream. Idempotency addresses duplicate application, but it does not make this mixed state uniformly current.

AWS DMS documentation states that there are no SLAs for CDC latency and that latency can rise to several minutes or longer. Source workload, network conditions, replication instance resources, target ingestion capacity, and data characteristics all contribute. An uptime commitment is therefore not a freshness commitment. For practitioners, the useful questions are how a tool makes sink writes idempotent and what latency it commits to.

What a managed service actually absorbs versus what it leaves to the team

A managed CDC service takes over source-specific log reading, cursor management, initial snapshots, schema detection, and transport delivery. In concrete terms, the team is not building WAL slot handling, oplog readers, or stream shard iterators. The service manages those mechanisms as infrastructure. It also coordinates initial source-table snapshots with change streaming so the destination has a consistent starting state, without thereby imposing a unified ordering on independent databases.

That infrastructure remains useful after the initial load. Schema detection and destination updates can automate column additions, while coverage of renames, drops, and type changes depends on the service. After an outage, recovery resumes from the last confirmed cursor rather than re-scanning the entire table unless retention has expired. Moving them into a service reduces the maintenance attached to every additional source.

Policy does not transfer in the same way. A service can execute a re-sync or a coercion, but the team must decide which response is acceptable for a type change. Sink idempotency design and the response to an outage beyond retention also remain architectural responsibilities. Likewise, correct delivery from a source that is two hours behind another does not produce a unified view of time.

Managed services in the pipeline model route changes to a single destination warehouse. A team that also needs a cache and a search index needs a broker layer. That requirement changes how much of the overall architecture the service replaces.

How to evaluate a managed service for multiple databases feeding one warehouse

For a workload consolidating five production databases into one cloud data warehouse, evaluation should test each source’s behavior. A support label is only the starting point. For PostgreSQL, does the service manage replication slot lifecycle, monitor lag, and alert before retained WAL fills the disk? For MongoDB, does it handle the full-document timing gap, or send partial events that need more data added to them? For DynamoDB, does retention expiry start re-scanning and reconciliation with the live stream automatically, or must the team step in?

Schema evaluation needs equally specific failure cases. What happens by default when a column is dropped or its type changes, and can that behavior be configured per source? Does the service inspect schemas itself, or depend on the team to signal changes out-of-band? The shared destination needs its own test. If two sources add the same column name with different types, the proof of concept should check what reaches the warehouse as well as whether both connectors keep running.

Does the service provide at-least-once or exactly-once transport, and how does it make sink writes idempotent? What latency is stated, what latency is observed under normal conditions, and are a CDC latency SLA and an uptime SLA provided? Outage behavior deserves a direct question too. Are in-flight events replayed from a last confirmed cursor, and is that cursor maintained per source or globally?

Rules for deployment can rule out some services before teams test performance. Healthcare and finance teams may require VPC deployment or data-residency guarantees, while storing customer data in transit raises compliance concerns for any service that does so. Retention is the key distinction to establish here. Does the service retain change-event payloads beyond the time required for delivery?

Where the managed service model fits, where it does not, and what the realistic deployment path looks like

The managed model is the right default when engineering time is the binding constraint. Time to first replication is measured in hours rather than the months associated with building and validating a custom multi-source pipeline. The relevant comparison is the whole fan-in system. Its ongoing value includes schema handling and recovery as sources change, where custom pipelines otherwise accumulate maintenance debt.

At very high change volumes, though, pricing can become non-linear, so teams should work out the costs at their expected event rates before committing. Strong infrastructure capability and strict fan-out requirements can also justify a different foundation. The warehouse leg may still remain managed. The decision is whether that service complements a necessary broker layer or duplicates work the broader architecture already requires.

Teams that need rigorous cell-level validation must build comparisons between destination data and an independent table snapshot to catch silent failures. Replication success alone is not that test.

For deployment, the practical sequence starts with the source that has the highest change rate and the most clearly defined schema. That exposes behavior under real conditions before adding heterogeneous complexity. Sources should then be added incrementally, with destination consistency validated after each addition. Teams need a clear schema policy for each source before its first production schema change, and should monitor lag for each source alongside native observability. Alerts tied to retention windows can warn of an approaching DynamoDB recovery cliff before events expire.

The managed model handles the infrastructure build, making fan-in CDC primarily a configuration and policy problem. The decision is therefore more precise than whether to outsource replication. Can the service execute this workload’s source-specific capture, schema, and recovery requirements while leaving the team with policies it can define and correctness it can validate?