Database Migrations and DynamoDB Streams: Real-Time Event Processing

Previously in this course, we explored advanced DynamoDB query patterns and how to handle API traffic. While querying allows you to pull data on demand, sometimes your application needs to "react" the moment data changes. This is where DynamoDB Streams come in.

By enabling a stream, you turn your database into a source of truth for an event-driven architecture. Instead of polling for updates, your system listens for changes—inserts, updates, or deletes—and triggers downstream logic automatically.

Understanding DynamoDB Streams from First Principles

A DynamoDB Stream is an ordered flow of information about changes to items in your table. When you enable it, DynamoDB captures a time-ordered sequence of modifications at the item level.

Think of it like a ledger of "what happened." If a user updates their profile, the stream records the old version of the item and the new version. You can configure the level of detail captured:

• KEYS_ONLY: Only the primary key attributes of the modified item.
• NEW_IMAGE: The entire item as it appears after the modification.
• OLD_IMAGE: The entire item as it appeared before the modification.
• NEW_AND_OLD_IMAGES: Both the old and new versions (essential for calculating deltas).

Why use event-driven data processing?

In traditional systems, you might perform a write and then manually trigger a secondary process (like sending an email or updating a search index). This is fragile; if the second process fails, your data gets out of sync. With DynamoDB Streams, the event is guaranteed to be captured by AWS, and the Lambda function will retry automatically if it fails, ensuring eventual consistency.

Worked Example: Triggering Lambda from a Stream

To implement this, we need to do two things: enable the stream on the table and grant the Lambda function permission to "read" from that stream.

1. Enabling the Stream in CDK

In your existing infrastructure stack (where you defined your DynamoDB table), update your table definition to include the stream property:

TYPESCRIPT
import { Table, StreamViewType } from CE9178">'aws-cdk-lib/aws-dynamodb';

const myTable = new Table(this, CE9178">'MyTable', {
  partitionKey: { name: CE9178">'pk', type: AttributeType.STRING },
  stream: StreamViewType.NEW_AND_OLD_IMAGES, // Enables the stream
});

2. Processing Stream Records in Lambda

When a change occurs, DynamoDB invokes your Lambda function with an event object containing a list of Records. Here is how you process them in Node.js:

JAVASCRIPT
exports.handler = async (event) => {
  for (const record of event.Records) {
    if (record.eventName === CE9178">'INSERT') {
      const newItem = record.dynamodb.NewImage;
      console.log(CE9178">'New item detected:', newItem);
      // Logic to trigger downstream tasks, e.g., send welcome email
    } else if (record.eventName === CE9178">'REMOVE') {
      console.log(CE9178">'Item deleted:', record.dynamodb.OldImage);
    }
  }
};

3. Wiring it up

Finally, you must tell the Lambda function to listen to this specific stream:

TYPESCRIPT
import { DynamoEventSource } from CE9178">'aws-cdk-lib/aws-lambda-event-sources';

myFunction.addEventSource(new DynamoEventSource(myTable, {
  startingPosition: lambda.StartingPosition.LATEST,
  batchSize: 5, // Process 5 records at a time
}));

Hands-on Exercise

1. Modify your CDK stack: Add the stream property to your project's DynamoDB table.
2. Deploy: Run cdk deploy to update the infrastructure.
3. Create a Processor: Write a simple Lambda function that logs the eventName of every record it receives.
4. Test: Use the AWS CLI or the console to PutItem into your table. Check your CloudWatch logs to see the stream event being processed.

Common Pitfalls

• Forgetting IAM Permissions: Even if you wire the event source, the Lambda execution role needs dynamodb:GetRecords, dynamodb:GetShardIterator, and dynamodb:DescribeStream permissions on the table. (CDK's addEventSource usually handles this for you, but keep it in mind if writing raw CloudFormation).
• Infinite Loops: Be careful. If your Lambda function writes back to the same table that triggers the stream, you will create an infinite loop. Always filter your events or write to a different table/service.
• Stream Expiration: Data in a DynamoDB Stream is only available for 24 hours. If your Lambda function is disabled or failing for longer than a day, those events are lost.

FAQ: Real-Time Data Processing

Q: Can I have multiple Lambda functions reading from the same stream? A: Yes. You can attach multiple Lambda functions to the same stream, allowing different services to react to the same data changes independently.

Q: Does enabling streams cost money? A: Yes, you are charged for the amount of data written to the stream and the number of read requests made by your Lambda function.

Q: What happens if my Lambda function fails? A: DynamoDB will retry the batch of records until the function succeeds or the data expires (24 hours). Ensure your code is idempotent—able to handle the same event twice without side effects.

Recap

We’ve moved from simple CRUD operations to event-driven architecture. By enabling DynamoDB Streams, we decouple our data storage from our business logic. This pattern is foundational for building scalable systems that maintain data integrity, much like the Change Data Capture patterns used in large-scale distributed systems.

Up next: We will shift focus back to the user interface to handle frontend state management and sync it with our backend updates.