Kafka-Based CDC Architecture Total Cost of Ownership

Fan-in CDC replication from multiple operational databases into a single warehouse uses source connectors, Kafka topics, and downstream consumers to bring database changes into one destination. Kafka-Based CDC Architecture Total Cost of Ownership includes the cost of infrastructure, the ongoing work of engineers, recovery work for each database, and software licensing. An honest build-vs-buy comparison prices those obligations at production scale against a managed replication service’s fee and the responsibilities it actually removes.

Why Kafka feels free until it runs in production

Kafka is open-source, and CDC connectors are widely available. That makes the first design look cheap before anyone has agreed to keep it running reliably. A working laptop setup proves that changes can move, while recovery at 3am on a Saturday still needs to be proven. The license covers the software, while production ownership remains a separate obligation.

CDC makes that distinction particularly important because the pipeline remains tied to a live database log. Falling behind can leave data inconsistent as well as delay events. The apparent bargain therefore needs to be separated into the resources, operating work, and recovery obligations that make “free” software usable.

How do multiple production databases feed one warehouse through Kafka?

A production fan-in pipeline coordinates source database logs, connectors, Kafka brokers, internal metadata, schema history, and downstream consumers. Each source sends changes through its connector, and consumers take those changes to the warehouse. These components must work together continuously. Consolidating five production databases into one cloud data warehouse is therefore not simply a connector installation multiplied across sources.

On Postgres, logical decoding reads the write-ahead log through a replication slot. The slot is a named, crash-safe cursor that tracks how far the consumer has read and keeps the WAL it still needs. MongoDB change streams instead build on the oplog and require a replica set or sharded cluster. The source mechanism changes the recovery contract. Postgres also omits DDL such as ALTER TABLE and DROP COLUMN from its logical replication stream, although it includes DML and, since Postgres 11, TRUNCATE.

The Kafka layer requires brokers and replication across availability zones for fault tolerance. Kafka 4.0 replaced ZooKeeper with native KRaft mode, removing a separate coordination system that older deployments still need to operate and eventually migrate. Metadata is another production dependency. Connector offsets, task status, and schema history support recovery, while captured tables add data topics and often schema history, dead-letter queue, and retry topics.

Infrastructure costs: the part of the bill teams actually see

Brokers need more compute as throughput and partition count grow, but they must have enough capacity for peak demand. Disk-based Kafka architectures do not automatically scale down that provisioned capacity when traffic falls. The expense persists during quieter periods as well. Kafka Streams or Apache Flink adds another compute layer, especially when CDC processing maintains state.

A production replication factor of three stores three copies of every retained byte, before spare capacity is considered. AutoMQ illustrates how storing 50 TB in a disk-based cluster can require roughly six times that amount in provisioned storage once replication and headroom are included. Replay makes this more than an archival choice. CDC consumers depend on retained changes for backfill and sink recovery, so retention becomes a platform guarantee.

The network can cost more than that storage because every replicated write crosses availability-zone boundaries. Cloud ingress is typically free, but cross-AZ traffic is not. AutoMQ identifies this traffic as often the largest individual infrastructure charge in self-hosted deployments, potentially accounting for a majority of the raw bill at scale. These are substantial costs in their own right. Even so, they cover only the machines and data movement.

Engineering and operational costs: the number that is never on the invoice

Engineering time dominates self-managed Kafka total cost because sustained specialist work is a structural requirement of operating the system. Conduktor says teams typically need at least one full-time engineer per few clusters, while larger organizations have teams devoted to the platform. Capacity planning, security patching, version upgrades, and incident response remain obligations after the initial deployment succeeds.

Garbage collection pauses can stall a broker’s service, so teams need to keep working on JVM tuning. Page cache behavior, file descriptor limits, and network buffers also require settings that differ from standard web server defaults. Disk write speed constrains throughput. Filesystem selection, RAID configuration, and mount options therefore become part of maintaining each broker.

Partition management adds a different balancing problem. Too few partitions restrict throughput, while too many slow leader elections and lengthen rebalancing. Adding or losing brokers can trigger data movement that saturates bandwidth for hours or days on large clusters. Older coordination systems introduce further work. ZooKeeper session timeouts can incorrectly mark healthy brokers dead, while migration to KRaft requires a multi-step, multi-day procedure with planning and rollback preparation.

Monitoring platforms, schema registries, and tools such as Cruise Control help operators perform this work, but they also need deployment, configuration, and debugging. Streamkap notes that a single outage can consume more engineering hours than a year of managed service fees. Beyond incident costs, routine maintenance consumes capacity that would otherwise build product.

CDC-specific failure modes that generate costs Kafka TCO models ignore

A generic Kafka budget can include operators and still omit failures that originate in the source database. Postgres WAL retention is a common example. With the default max_slot_wal_keep_size = -1, a logical replication slot that is inactive keeps WAL pinned with no time limit. If a connector stops unnoticed, the primary’s pg_wal directory can fill the disk until Postgres refuses writes.

Recovery then extends beyond restarting the connector. Engineers must step in to reconcile gaps in the data and may need to run a full re-snapshot. That snapshot increases Kafka storage demand while competing with production database traffic. PostgreSQL 17 added synchronization for slots created with failover = true, allowing a promoted standby to resume CDC without re-snapshotting, whereas earlier versions required external tooling or a fresh snapshot.

Schema changes create another failure path because Postgres does not send DDL through logical replication. A new column can lead to schema registry validation failure and stop the pipeline, with stale reports becoming the first visible symptom. Diagnosis is only the beginning of recovery. Engineers may need to repair connector state, replay or reconcile missing changes, and change the migration process to prevent recurrence.

Consumers can also lose their recovery position without a broker outage. When lag exceeds Kafka’s retention window, deletion of needed segments can force a new source snapshot. Loss or corruption of connector metadata can likewise require re-snapshotting every captured table. MongoDB has a corresponding window constraint. Its opaque resume tokens become unusable when the oplog window expires before the consumer catches up, requiring a new snapshot. Each failure mode needs a tested recovery runbook, and maintaining that runbook is an engineering cost.

Exactly-once delivery and schema evolution are engineering projects, not configuration flags

Kafka supports at-most-once, at-least-once, and exactly-once delivery guarantees. Idempotent producers and transactional APIs, available since Kafka 0.11, provide mechanisms for exactly-once behavior, but do not make an entire CDC path exactly-once by default. The guarantee depends on the complete chain. Database log retention, connector offsets, Kafka durability, consumer logic, and target writes can each determine what the pipeline actually delivers.

Teams should treat CDC pipelines as at-least-once unless they have designed them for end-to-end exactly-once behavior. Consumers need idempotent behavior, typically through primary-key upserts and explicit handling of deletes. Target behavior is part of the contract. Flink can provide exactly-once processing semantics for CDC workloads, but introducing it adds another system to deploy, operate, debug, and support on call.

Because Postgres logical replication omits DDL, changes must be detected, versioned, and propagated through a separate mechanism, typically a schema registry with compatibility rules. Prevention reaches beyond the connector configuration alone. Explicit compatibility policies and tested migration procedures require database administrators and data platform engineers to coordinate, while warehouses need automatic column additions or versioned table strategies. These capabilities are reasons to count implementation work accurately.

How the cost structure changes as CDC pipelines scale

A design affordable for five databases can have a very different operating profile at fifty. Each data topic can bring supporting topics, while additional partitions require decisions about distribution, consumer assignment, replication, retention, and monitoring.

Consumer growth also changes the storage commitment. As more teams depend on replay, retention must accommodate the slowest consumer’s recovery window, increasing the volume of replicated data held in the cluster. Recovery capacity becomes a shared platform obligation. Meanwhile, higher incoming throughput increases cross-AZ replication traffic, and broker additions or failures require moving larger retained datasets during reassignment.

Governance grows with the number of teams consuming database changes. Access controls, schema contracts, lineage, and deletion semantics for compliance need explicit rules and enforcement. Support expectations can change as well. Contractual SLAs introduce enterprise support costs through per-node subscriptions, consumption-metered options, or custom-quoted retainers, each responding differently to growth.

Modeling ownership at target scale requires going beyond extrapolation from the pilot’s cloud bill. That model should include how many sources and consumers there will be in production, the recovery windows, the governance rules teams must meet, and the staff they will need. Growth changes more than the broker count. Engineering, storage, and network costs can grow faster than the number of source tables because each addition changes what the team must do for downstream users.

What a managed replication service costs by comparison, and what the tradeoff actually is

The useful comparison is between permanent responsibilities. Self-managed Kafka preserves control and customizability while leaving the team responsible for brokers, connectors, recovery, and delivery semantics. A managed service takes over a defined set of operating duties, while the team pays a recurring fee and depends on the vendor. The boundary of that transfer determines its value. A full comparison of the numbers needs fees for the workload and a clear account of who does what.

Managed connectors and managed replication are not equivalent purchases. Moving Kafka Connect into a managed layer reduces work around deployment, scheduling, upgrades, retries, and plugins. The team may still have to run the Kafka cluster. In that arrangement, the self-managed budget still needs to cover retention economics, replay costs, and consumer catch-up behavior.

A managed replication service that handles schema evolution, failure recovery, fan-in, and compliance takes more of the engineering work off the team’s hands. A passthrough replication design, in which the infrastructure does not store customer data, also addresses storage-related security and compliance requirements. Scope is therefore the decisive comparison criterion. For simultaneous replication from Postgres and other relational databases into one warehouse, the relevant comparison includes source coverage, fan-in support, schema handling, recovery responsibilities, and data storage behavior.

Self-management can still be good value for hyperscale throughput when a team has engineers devoted to the platform. For most growth-stage and enterprise teams, however, the sustained engineering expense exceeds the managed service fee before that point. The decision turns on what ownership will cost over three years and what the same engineering capacity could build instead.