Keeping CDC Pipelines Running When Upstream Source Schemas Change
Schema changes break CDC pipelines at predictable boundaries between layers.
Automatic schema evolution for live CDC pipelines requires coordinated schema detection, compatibility checks, ordered propagation, destination DDL, and exactly-once delivery. Keeping pipelines running when upstream schemas change depends on that coordination across the source log, streaming layer, and warehouse. Failures arise at predictable boundaries, and safe changes require a sequence that respects each layer’s schema model.
Why schema changes break CDC pipelines and where the damage lands
A source database, a streaming connector, and a destination warehouse do not interpret a schema change in the same way. Each maintains its own model of the data. DDL executes at the source first, after which new events enter the log while older events remain in flight. The connector must handle both shapes before the warehouse can receive them. A failure therefore depends on which layer first encounters a change it cannot accommodate.
Streamkap’s schema evolution playbook describes this propagation through the WAL, the CDC engine’s internal mapping, the streaming topic, and downstream consumers. Each handoff is a separate failure point. An added nullable column may pass through those boundaries, while a rename can invalidate both serialization rules and consumer field references. A primary key change or table drop-and-recreate can require a full re-snapshot. The operation determines the failure path.
What the transaction log actually records when a schema changes — and what it omits
PostgreSQL logical replication does not replicate DDL. An ALTER TABLE on the publisher does not automatically alter the subscriber, and PostgreSQL 18 did not change that behavior. Schema information reaches the connector through Relation messages when data is modified. Detection therefore waits for the first modified row. If that row includes a new column that the subscriber cannot accept, replication fails until the subscriber schema catches up.
Workaround tools add the missing DDL path. pglogical replays statements passed to pglogical.replicate_ddl_command() through the replication stream. pgcopydb uses event triggers to capture DDL and emits each command through pg_logical_emit_message. Its apply process executes that command on the target within the same apply transaction as the rest of the source transaction. These mechanisms supply information that core logical replication does not propagate automatically.
MySQL’s binlog records DDL as statement events. A connector can therefore encounter ALTER TABLE before the first data row in the new schema. MongoDB’s oplog instead records logical document operations, including inserts, updates, and deletes, without enforcing a column schema. DynamoDB streams likewise do not enforce schemas, while records contain keys, old images, new images, or both, depending on StreamViewType. A new attribute does not itself generate a schema failure signal in either system.
That flexibility moves responsibility downstream. For WAL-based pipelines, a stalled connector also faces a retention risk. If the source recycles required WAL segments before recovery, the connector loses its position and must re-snapshot the entire table. A schema error can consequently become a recovery operation involving much more than the changed column.
How the streaming layer reacts to events that arrive in two different schemas simultaneously
During a schema transition, one Kafka topic or equivalent transport can contain events in both the old and new schemas. Consumers must deserialize both versions during that window. A schema registry runs compatibility checks, but its mode sets which changes it accepts. The presence of a registry alone is insufficient. Its rules must match how producers change and how consumers read historical data.
The playbook recommends BACKWARD compatibility as the default to use for CDC because new schemas must read older events. Its guidance treats optional additions and numeric widening as compatible, while identifying drops, renames, and narrowing as dangerous changes. FORWARD compatibility reverses the reading direction, allowing older schemas to read newer data and rejecting required additions without defaults. FULL requires both directions, while NONE performs no checks. The proposed Avro or JSON schema still needs testing against the registry’s compatibility endpoint before DDL executes.
Renames expose the limits of treating schema changes as field updates. To most connectors, a rename looks like a drop plus an addition. A name-based consumer loses the field it expects even when the value itself stays the same. Xata’s pgstream v1.0.0, released in February 2026, takes an event-driven approach by replicating schema changes as they occurred. It captures DDL without storing schema state in the source database and preserves ordering between schema and data changes.
AWS DMS illustrates a different connector constraint. AWS’s knowledge-center guidance describes a case where the task captures the table structure at startup and does not refresh it when a column is added. The new column may consequently be ignored, and recovery requires stopping the task and restarting with reload-target. AWS also documents continuous replication use cases, but its CDC documentation states that it does not provide real-time replication or CDC latency SLAs. Continuous replication and automatic schema adaptation are distinct capabilities.
Classifying DDL operations by the risk they carry through the pipeline
Low-risk changes preserve the interpretation of existing events. Adding a nullable column lets consumers handle older events without a value for that field, while a column with a default supplies a consumer-side fallback. Widening INT to BIGINT preserves the validity of old values. Adding a new table to CDC preserves existing streams, and these operations still depend on compatible serialization and destination handling.
Risky changes require coordination. Adding a NOT NULL column without a default leaves existing events without the required field and may mean the source table must be rewritten. Changing INT to DECIMAL depends on consumer parsing and type coercion. Adding an index is safe for CDC data flow, but execution can affect source performance.
Dangerous changes remove information or change how it is identified. Drops and renames first threaten connector compatibility and field-based consumers, while narrowing BIGINT to INT can leave values outside the new range. Dropping and recreating a table can invalidate the pipeline’s log position and require a full re-snapshot. Primary key changes belong in the same high-risk planning category. Their effect extends beyond whether a new field can be serialized.
How do teams execute safe schema changes without stopping the CDC pipeline?
For additive changes, the procedure begins before the database changes. Test the proposed Avro or JSON schema against the registry’s compatibility endpoint, and fix any rejected definition before proceeding. Then run ALTER TABLE so that a capable CDC engine detects the new column during its next schema detection cycle. Verify that subsequent events include the field, allowing for in-flight events that lack it and require null handling. Update downstream consumers to read the field only after the new schema is flowing.
A rename requires an expand-and-contract sequence. First add the new column while retaining the old one, then backfill the new column from the old column at the source. Next, update application code to write to both columns at the same time. Only then should downstream consumers switch to reading the new column. Both names remain available during the transition.
After consumers have switched, stop the application from writing to the old column. Drop it only when no consumer depends on it. This prevents the consumer failure, but the drop must still pass the pipeline’s compatibility checks. The application team and data team must agree on when to carry out the dual-write and consumer-switch stages. Coordination failure is the dominant risk when one team advances before the other.
For a drop without a replacement field, use the same consumer-retirement sequence without a backfill. Primary key changes are the exception to the uninterrupted procedure. In most CDC systems, the safest approach is to pause replication, apply the change, trigger a full re-snapshot, and validate row counts and checksums before bringing consumers back online.
How the destination warehouse layer absorbs schema changes — and where it still fails
A valid event can still fail at the warehouse. The destination must apply the schema change before, or atomically with, the first event that requires it. Across destinations such as Snowflake, BigQuery, Redshift, and Databricks, handling of unknown columns varies. A write may be rejected, or an unknown field may be ignored even though the rest of the row arrives. Automatic destination DDL removes this ordering failure only when it is coordinated with event delivery.
Schema evolution belongs in the raw, flexible bronze layer, where Databricks Auto Loader can adapt to upstream changes. From silver onward, strict schemas protect merge operations and downstream transformations. Column type changes, drops, and nested structure changes can still call for fixes in downstream DLT processing.
Keeping full raw data in bronze makes that intervention recoverable. Transformation logic can be updated and data replayed without returning to the source system. But rebuilding a materialized view or adding a new consumer may still need a source snapshot followed by the changes captured during that snapshot. An uncoordinated overlap can create duplicates or inconsistencies. The safe pattern uses snapshot-then-stream with a transaction-log-position-based cutover and a shared consistency watermark, followed by row-count and checksum validation before the old pipeline state is decommissioned.
What automated schema evolution actually requires a CDC system to do
A system must detect the new source schema before serializing the first event that uses it. Detection after serialization is already too late. It must also check compatibility against the downstream schema before propagating the change. These are separate requirements. Discovering a new field does not establish that consumers can safely read it.
The system must then keep schema and data events in order. Sending them through separate asynchronous paths can bring a row to its destination table before the table is ready. The warehouse DDL must execute before, or atomically with, the first new-schema event. This coordination cannot end at the streaming topic. The destination writer is part of the schema evolution mechanism.
Exactly-once delivery must survive the transition as well. Kafka transactions, idempotent sinks, and offset management must work together when a schema change causes a connector restart. Otherwise, recovery can introduce duplicate destination events even if normal operation avoids them. Exactly-once is a system property.
A new column may contain PHI, making automatic forwarding a compliance risk in HIPAA or SOC 2 contexts. The system must allow the column to be checked against redaction or masking rules before it moves downstream. Accepting a schema is not permission to expose its contents.
Why DIY CDC pipelines rarely survive schema changes in production and what a managed service handles instead
A log reader, transport, and warehouse writer can each work as they should while the pipeline as a whole fails. DIY teams may discover that they lack pre-flight compatibility checks, ordered schema propagation, automatic destination DDL, or offset handling that preserves exactly-once delivery through restarts. Closing those gaps can take months. Configuring each component independently does not create the shared sequence that schema evolution requires.
A purpose-built managed replication service handles detection, compatibility enforcement, ordered DDL propagation, destination changes, and delivery guarantees as one coordinated system. Those behaviors are the criteria for evaluating it. A service that supports added columns still needs to show that it can safely handle drops, renames, primary key changes, or warehouse replay. A service’s useful scope is defined by what it can propagate automatically and where it requires a coordinated migration.
That engineering is not necessary for every workload. For a single small table without a real-time requirement, timestamp-based polling is simpler and cheaper than log-based CDC. A managed service makes more sense when application developers decide when schemas change and an overnight outage is unacceptable. In that setting, the purchase is coordinated handling of the boundaries where independently correct components would otherwise leave recovery to the data team.
