Queueing Tasks: Asynchronous Processing with Cloudflare Workers

Previously in this course, we explored working with KV storage to manage fast, global state. Now, we're taking a leap into Asynchronous Processing, where we'll learn how to offload time-consuming tasks to keep our primary request handlers snappy and responsive.

Why Use Queues?

In a standard HTTP request-response cycle, your Worker must finish its work and send a response before the connection closes. If you need to send an email, process an image, or perform a heavy database migration every time a user triggers an action, your response time will skyrocket.

By using Queues, you decouple the trigger (the user action) from the execution (the heavy lifting). You "fire and forget" a message into a queue, and a separate, dedicated consumer Worker processes that message whenever it's ready. This is a core pattern in API design for asynchronous processing, allowing your system to handle spikes in traffic without crashing.

The Architecture of Cloudflare Queues

Cloudflare Queues consist of two main components:

1. The Producer: A standard Worker that accepts user requests and pushes data into a Queue.
2. The Consumer: A separate Worker that listens for messages in the Queue and processes them.

Flow diagram: User → HTTP Request Producer Worker; Producer Worker → Push Message Cloudflare Queue; Cloudflare Queue → Trigger Consumer Worker; Consumer Worker → Process D1 / R2

Worked Example: Building the Queue

First, define your queue in your wrangler.toml file. This tells Cloudflare that your project needs a queue resource.

TOML
[[queues.producers]]
queue = "my-task-queue"
binding = "MY_QUEUE"

[[queues.consumers]]
queue = "my-task-queue"

1. The Producer Worker

In your main worker, use the send method to push tasks to the queue.

JAVASCRIPT
export default {
  async fetch(request, env) {
    const task = { userId: 123, action: "generate_report" };
    // Push the task to the queue
    await env.MY_QUEUE.send(task);
    return new Response("Task queued successfully!");
  }
};

2. The Consumer Worker

The consumer is a specialized worker that exports a queue handler. This handler processes batches of messages.

JAVASCRIPT
export default {
  async queue(batch, env) {
    for (const message of batch.messages) {
      console.log(CE9178">`Processing task: ${message.body.action}`);
      // Perform your heavy lifting here, e.g., calling an external API
      message.ack(); // Acknowledge completion
    }
  }
};

Hands-on Exercise

1. Create the Queue: Run npx wrangler queues create my-task-queue in your terminal.
2. Implement: Update your wrangler.toml with the producer and consumer blocks shown above.
3. Deploy: Run npx wrangler deploy to push both workers to the Cloudflare edge.
4. Test: Trigger your producer endpoint and check your Cloudflare Dashboard under the "Queues" tab to see the message flow and processing logs.

Common Pitfalls

• Forgetting to Acknowledge (Ack): If you don't call message.ack(), Cloudflare assumes the job failed and will retry it based on the queue's retry policy. This can lead to duplicate processing if your logic isn't idempotent.
• Assuming Instant Execution: Queues are asynchronous. If your UI relies on the result of the task, you need a way to poll for status or use WebSockets, which we will cover in a later lesson.
• Infinite Retries: By default, if a consumer throws an error, the message stays in the queue. Always implement try/catch blocks inside your consumer to prevent "poison pills" (malformed messages) from blocking your queue.

FAQ

Q: Are queues strictly FIFO? A: Cloudflare Queues provide "at-least-once" delivery, but they prioritize performance and availability. Do not rely on strict ordering for critical business logic.

Q: Can I use queues with D1 or R2? A: Yes. Queues are the recommended way to perform background writes to D1 or R2 when the operation doesn't need to be immediate.

Q: Is this similar to background jobs in other frameworks? A: It is conceptually identical to background workers in Node.js or Laravel background processing, just optimized for the serverless edge.

Recap

We've moved beyond simple request-response cycles. By using Queues, we've enabled our application to handle heavy background tasks, significantly improving user-perceived performance. We created a Producer to ingest work and a Consumer to process it in the background.

Up next: Cron Triggers, where we’ll learn how to automate tasks based on a schedule rather than a user action.