DynamoDB Streams Architecture and Replication Limits

DynamoDB Streams is a capable change data capture primitive whose limits are architectural, not incidental. Teams building replication pipelines on it discover that shard topology, view-type immutability, and a two-reader-per-shard ceiling impose real constraints that no amount of Lambda tuning removes. This piece maps where those ceilings sit, how they compound, and what a production replication system must add on top of them.

What DynamoDB Streams actually is, and what it gives you before any limits apply

DynamoDB Streams captures a time-ordered sequence of item-level modifications to a DynamoDB table and stores them in a log for up to 24 hours, available in near-real time to downstream consumers. Every successful write produces a stream record describing a single change: an INSERT, MODIFY, or REMOVE event, accompanied by a monotonically increasing sequence number and a timestamp. The sequence number is the key to ordering, and it allows consumers to track their position in the stream with precision.

The stream is organized into shards, each holding a strictly ordered sequence of records. DynamoDB creates, splits, and deletes shards automatically as the underlying table partitions change, and each open shard maps to exactly one table partition at any given time. This design keeps write amplification low and preserves per-partition ordering, which is the property most consumers actually need.

When the stream is enabled, the table owner selects one of four view types that determine what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES. Stream data is encrypted at rest automatically. What it does not provide begins to matter the moment a team attaches a real pipeline to it.

The four view types and why choosing one locks your pipeline's schema options

The four view types differ in record size and semantic completeness, and each one forecloses certain consumer patterns. KEYS_ONLY delivers just the primary key attributes, the smallest possible record. NEW_IMAGE carries the full item after the change, which covers most synchronization use cases, but it loses the prior state entirely, making conflict detection impossible. OLD_IMAGE holds the full item before the change, useful for audit trails yet unable to reconstruct current state on its own. NEW_AND_OLD_IMAGES carries both, the largest record, and enables diff-based synchronization and full audit, but a REMOVE event's NewImage will be null, requiring null checks in consumer code.

For replication to a data warehouse, teams often select NEW_IMAGE for its moderate size. When downstream schema inference depends on attribute deltas, knowing that an attribute existed before and disappeared is precisely the information that NEW_IMAGE discards. What looked like a reasonable size optimization at enable time becomes a semantic gap in the pipeline.

The view type is also immutable. Changing it requires disabling and re-enabling the stream, which creates a new stream with a different ARN and breaks any consumer holding the old ARN. The decision made in the first hour of setup binds every downstream capability for the life of the table.

What ordering DynamoDB Streams guarantees — and what it does not

DynamoDB Streams guarantees that records for a given item appear in the same sequence as the actual modifications to that item. Ordering is per primary key, not per partition and not across the whole table. Within a single shard, records are strictly ordered, but across shards there is no ordering guarantee whatsoever. A shard split or a table repartition silently redistributes keys, and any consumer that assumed global ordering has been relying on a property the stream never possessed.

The practical consequence lands on any downstream aggregate that spans multiple partition keys. A running total computed from stream events, a cross-item consistency check, a multi-entity event sequence: none of these can trust the stream to deliver events in global order. Teams that genuinely require global ordering must move to Kinesis Data Streams for DynamoDB, which does not itself provide global ordering either but does expose the consumer-side coordination patterns, sequence numbers and watermarking, that make it possible to impose one. That is a meaningful distinction: the stream gives per-key ordering, which is enough for entity-level processing and nothing more.

What "exactly once in the stream" does and does not mean for end-to-end delivery

AWS guarantees that each stream record appears exactly once in the stream itself. That guarantee is real, but it governs only the boundary between DynamoDB and the stream. The moment a Lambda consumer reads a batch, the semantics change.

End-to-end delivery to a Lambda consumer is at-least-once. Idempotency therefore is not optional in any production consumer. A composite of the sequence number and the item primary key serves as a reliable idempotency key, but applying it correctly is entirely the consumer's responsibility. The stream does not track what a consumer has successfully processed, and it does not suppress redelivery.

The failure pathology without mitigation is stark. A single failing record in a batch blocks the entire shard. Available mitigations exist: bisect_batch_on_function_error, ReportBatchItemFailures, bounded maximum_retry_attempts, and an on-failure destination such as an SQS queue or SNS topic. The stream has no built-in poison-pill handling, no dead-letter queue, no partial-failure acknowledgment. It offloads the hardest part of reliable delivery entirely to the consumer.

The two-reader-per-shard ceiling and what it prevents

For single-Region tables, the concurrent-reader limit per shard is two simultaneous processes. For global tables, AWS recommends limiting simultaneous readers to one to avoid request throttling. The limit applies independently to each shard, and because the shard topology is opaque and automatically managed by DynamoDB, teams cannot provision their way around it. Adding read capacity does nothing. The ceiling is fixed.

A replication pipeline that needs one process for the data warehouse and a second for audit logging already saturates the limit. The canonical pattern is one Lambda reading the stream and publishing to SNS, which fans out to multiple SQS queues, each consumed by a dedicated Lambda. An alternative routes through EventBridge, which adds another hop and another failure domain.

The deeper problem is that neither fan-out pattern is a native feature of DynamoDB Streams. The stream gives any single consumer a clear path to the data. It does not give a team of consumers, each with different needs, a clear path to share it.

How the 24-hour retention window and hot-shard behavior compound the consumer-limit problem

Stream records are retained for up to 24 hours, and the window is not configurable. Combined with the blocked-shard failure pathology, the compounding effect is the point. A consumer that stalls for hours under a bad batch is burning through its retention window while accumulating lag, and a team that waits to notice the stall may find the window has closed on the records they needed.

DynamoDB Streams also does not increase shard count in response to high write traffic. Shard count follows table partition count, not throughput demand. The result is a hot shard: a processing bottleneck the stream itself cannot relieve, because adding compute changes nothing when the shard is the unit of parallelism and the shard is singular.

The two constraints interact directly. A hot shard under sustained high throughput, plus a blocked batch retry loop, creates a window where a consumer can fall so far behind that records expire before they are processed, at which point data is silently lost with no native alerting from the stream. Retention, parallelism, and failure handling are not independent dials. Each one makes the others' limits arrive sooner.

What enabling Streams on an existing table does not do: the no-backfill problem for replication bootstrapping

Enabling DynamoDB Streams on an existing table is a non-breaking, zero-downtime operation. The stream begins capturing changes from the moment it is enabled and does not backfill historical data. Zero downtime for the source table, however, does not mean zero gap for the downstream system: any rows written before the stream was enabled are invisible to every stream consumer, forever.

The required workaround is a full table scan first, then enabling the stream to capture incremental changes. A write that happens after the scan reads a row but before the stream is enabled falls into the gap. Reconciling those two paths is not a trivial batch job. It must be correct under concurrent writes.

This is not an edge case. Every new replication target for an existing DynamoDB table faces this bootstrapping problem, and the stream provides no tooling to solve it. The engineering burden falls entirely on the team building the pipeline. Bootstrapping is where most replication pipelines face their first real engineering decision, and the stream itself offers no guidance past it.

When Kinesis Data Streams for DynamoDB is the right answer, and what it costs to get there

Kinesis Data Streams for DynamoDB addresses the three hardest constraints of native Streams directly. It supports many more simultaneous consumers per shard with enhanced fan-out, extends retention to up to 365 days, and supports both provisioned and on-demand shard management. Enhanced fan-out uses a push model over HTTP/2 rather than the DynamoDB Streams pull model, which reduces per-consumer read latency and gives each consumer its own dedicated throughput instead of shared shard read bandwidth. For teams whose workloads have genuinely hit the two-reader ceiling or the 24-hour wall, this is the AWS-native path forward.

The cost side is equally concrete. DynamoDB Streams read request units are billed separately and at a lower rate than Kinesis Data Streams for DynamoDB change data capture units, and Kinesis charges add on top of that. Kinesis is not a simpler version of the same thing. It is a more capable system with more of its internals exposed.

How the schema-less DynamoDB model creates a live schema-evolution problem at the warehouse destination

DynamoDB tables have no enforced schema. Different items in the same table can carry different attributes, and new attributes appear on any write without a table-level migration. That flexibility is a source-side virtue. It becomes a destination-side problem the moment the data lands in a warehouse, because Redshift, BigQuery, Snowflake, and ClickHouse all require defined columns. When a new attribute appears in a stream record for the first time, a new column must be created in the destination table, without a manual ALTER TABLE and without restarting the pipeline.

The stream view type chosen at enable time determines which attributes are visible in stream records, and that choice compounds the problem. A team that chose KEYS_ONLY cannot detect new attributes at all. Schema evolution is not an operational afterthought for DynamoDB replication. It is a first-class production problem, and the view-type decision made at stream enable time determines how solvable it will be.

What production replication pipelines on DynamoDB Streams actually require teams to build and operate

A production-grade DynamoDB-to-warehouse replication pipeline on native Streams requires five distinct pieces of engineering, none of which the stream provides: initial bootstrapping logic with full scan and overlap coordination; idempotent consumer code; shard-level failure handling with a dead-letter queue, partial batch response, and retry bounds; fan-out architecture if more than two consumers are needed; and destination-side schema evolution. Each is a separate problem with its own failure modes, and they interact. A fan-out retry in one branch diverges from another while the underlying shard approaches its 24-hour expiry, and the team discovers the interaction only in production.

The shard lifecycle itself is opaque. DynamoDB creates, splits, and closes shards automatically, and consumers must handle shard enumeration, checkpoint management, and the transition from parent to child shards without dropping records. There is no native checkpoint store. Teams typically use DynamoDB itself or a Lambda-managed offset store. Teams at financial-services scale, where the 24-hour retention ceiling, the two-reader limit, and audit-trail compliance apply simultaneously, typically route DynamoDB changes through Kinesis before landing in a warehouse, adding another system to the stack.

What a managed replication service handles that the stream itself does not

The constraints DynamoDB Streams imposes, no backfill, at-least-once end-to-end delivery, no schema evolution, no checkpoint store, an opaque shard lifecycle, the two-reader ceiling, are not defects in the stream. They are the natural scope of a CDC primitive, and a production replication system requires a layer above them. A managed replication service handles bootstrapping coordination, a full scan followed by stream catch-up with correct overlap semantics, and maintains consumer checkpoints and shard lifecycle transitions, including parent-to-child handoffs, without operator intervention.

Exactly-once delivery at the destination, as distinct from the stream's exactly-once-in-the-log guarantee, requires idempotent writes at the warehouse layer. A managed service applies this consistently across all event types and failure scenarios rather than delegating it to each team's consumer code. The practical argument is straightforward: the stream gives you the data, and a managed service gives you the pipeline. Building the pipeline from DynamoDB Streams primitives is a project that takes months and produces infrastructure maintained indefinitely. A managed service makes sub-minute replication from DynamoDB to the warehouse available on day one.