Handling Asynchronous Events: SQS and Lambda for Resilient Systems

Previously in this course, we covered Triggering Lambda Functions: Mastering Event-Driven Architectures, where we focused on direct invocation. This lesson adds asynchronous processing to your toolkit, allowing you to decouple services so that your system remains responsive even when downstream tasks take time or experience spikes in traffic.

Why Asynchronous Architectures Matter

In a synchronous system, if a user uploads a file and your backend immediately tries to process it, resize it, and save it to a database, the user must wait for all those steps to finish. If one step fails or slows down, the entire request hangs.

By using an Event-driven approach with SQS (Simple Queue Service), you change this flow:

1. The user uploads the file.
2. Your API puts a message (an "event") into an SQS queue.
3. Your API returns an immediate "Accepted" response to the user.
4. A separate Lambda function "polls" the queue and processes the message whenever it's ready.

This pattern ensures that a burst of traffic doesn't overwhelm your database, as the queue acts as a buffer.

Creating Your First SQS Queue

To integrate SQS into your project, we will add a queue to your existing CDK stack. In your lib/backend-stack.ts, define the queue:

TYPESCRIPT
import * as sqs from CE9178">'aws-cdk-lib/aws-sqs';

const taskQueue = new sqs.Queue(this, CE9178">'TaskQueue', {
  visibilityTimeout: cdk.Duration.seconds(300), // Time Lambda has to process
});

The visibilityTimeout is crucial. It defines how long AWS keeps a message "hidden" from other consumers while your Lambda processes it. If the Lambda doesn't finish within this time, the message becomes visible again, and another consumer might pick it up.

Pushing Events to the Queue

To send a message to the queue from your existing Lambda function, you need the AWS SDK (v3). Ensure your function has the sqs:SendMessage permission via your IAM role (as discussed in Managing Lambda Execution Environments).

JAVASCRIPT
const { SQSClient, SendMessageCommand } = require("@aws-sdk/client-sqs");
const sqsClient = new SQSClient({});

exports.handler = async (event) => {
  const params = {
    QueueUrl: process.env.QUEUE_URL,
    MessageBody: JSON.stringify({ userId: CE9178">'123', action: CE9178">'process_data' }),
  };
  
  await sqsClient.send(new SendMessageCommand(params));
  return { statusCode: 202, body: CE9178">'Task Queued' };
};

Triggering Lambda from SQS

Finally, you must connect the queue to a second Lambda function—the "worker." In your CDK code, use the SqsEventSource:

TYPESCRIPT
import { SqsEventSource } from CE9178">'aws-cdk-lib/aws-lambda-event-sources';

workerLambda.addEventSource(new SqsEventSource(taskQueue, {
  batchSize: 5 // Process up to 5 messages at once
}));

When your workerLambda runs, the event object will contain an array of records from the queue. You must iterate through them to handle the processing logic.

Hands-on Exercise

1. Define the Queue: Update your CDK stack to include a new sqs.Queue.
2. Grant Permissions: Use taskQueue.grantSendMessages(yourMainLambda) to allow your main function to push to the queue.
3. Deploy: Run cdk deploy to provision the infrastructure.
4. Test: Manually trigger your main Lambda function and verify that the message appears in the SQS console under "Monitoring."

Common Pitfalls

• Visibility Timeout Too Short: If your processing takes 40 seconds but your timeout is 30, the message will reappear in the queue while the first Lambda is still working, leading to duplicate processing.
• Missing Error Handling: If your processing logic throws an error, the message returns to the queue. If you don't have a "Dead Letter Queue" (DLQ) configured, it can loop indefinitely. Always attach a DLQ to your main queue for failed messages.
• Permission Gaps: Forgetting to grant sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes to your worker Lambda will result in it being unable to pull or remove messages.

FAQ

Q: How does SQS compare to SNS? A: SQS is a point-to-point queue (one message, one consumer). SNS is a pub/sub system (one message, many subscribers). You often use them together: an event is published to SNS, which then fans out to multiple SQS queues.

Q: What happens if my Lambda fails? A: If the function crashes or returns an error, SQS will wait for the visibility timeout to expire and then make the message available again for a retry.

Recap

We've moved from simple synchronous triggers to a more robust, asynchronous pattern. By using SQS to decouple your service, you've learned how to buffer requests, enable retries, and ensure your system remains stable under load. These are the fundamental building blocks of modern, event-driven cloud architectures.

Up next: We will learn how to manage different versions of your code and use Aliases to point traffic to specific deployments.