Handling Background Tasks: Worker Services and Queues

Previously in this course, we explored the introduction to message brokers, where we decoupled our services by sending messages to a broker. Now, we will operationalize that concept by building a worker service to process those messages, effectively shifting heavy lifting away from the request-response cycle to improve overall system performance.

Why We Use Worker Services

In a typical web application, the user makes a request and waits for a response. If that request triggers an expensive operation—like generating a PDF report, resizing an image, or sending a batch of emails—the user is left staring at a loading spinner. This increases latency and risks request timeouts.

By using background jobs, we acknowledge the user's intent immediately (e.g., "We've started processing your report") and offload the actual execution to a worker—a separate process or service that listens to a queue.

Creating a Worker Service

A worker service is essentially a long-running process that follows a simple loop:

1. Poll/Subscribe: Get the next available task from the queue.
2. Execute: Perform the business logic defined by the task.
3. Acknowledge: Signal that the task is finished so the broker can remove it from the queue.

For our running project, let's assume we need to process user avatar uploads. Instead of resizing in the controller, we push the file path to a queue.

Worked Example: A Simple Node.js Worker

Using a library like bull or a standard message broker client, your worker logic looks like this:

JAVASCRIPT
// worker.js
const queue = require(CE9178">'./task-queue');

async function processImageUpload(job) {
    console.log(CE9178">`Processing image: ${job.data.filePath}`);
    // Simulate heavy image processing
    await new Promise(resolve => setTimeout(resolve, 2000));
    console.log(CE9178">'Done!');
}

// The worker loop
queue.process(async (job) => {
    try {
        await processImageUpload(job);
    } catch (error) {
        // Handle failure logic here
        throw error; 
    }
});

Handling Job Failures

In distributed systems, failures are inevitable. A worker might crash, a database might be down, or an external API might return a 500 error. If a job fails, we cannot simply delete it from the queue, or the work is lost forever.

We implement retry logic and dead-letter queues (DLQ):

• Retries: Configure the queue to re-queue the job after a short delay (exponential backoff).
• Dead-Letter Queue: If a job fails a predefined number of times (e.g., 3 retries), move it to a special "dead-letter" queue for manual inspection. This prevents "poison pills"—bad jobs that crash workers repeatedly—from blocking your pipeline.

Hands-on Exercise

1. Define a Job: Create a function that simulates sending a welcome email.
2. Dispatch: Update your API endpoint to push a "SEND_EMAIL" job to your broker instead of calling the mail service directly.
3. Process: Create a worker script that pulls these jobs and executes them.
4. Simulate Failure: Force the worker to throw an error 50% of the time and verify that your queue configuration successfully retries the task.

Common Pitfalls

• Assuming Success: Never assume a background job will succeed. Always log failures with enough context (e.g., job.id, error.message) to debug them.
• Large Payloads: Do not store massive files in the queue itself. Store a reference (like an S3 URI) in the message and let the worker fetch the data.
• Lack of Idempotency: If a job is retried, it might run twice. Ensure your job logic is idempotent (performing the action multiple times has the same result as performing it once). We dive deeper into this in our guide on reliable background jobs.

Frequently Asked Questions

• How many workers should I run? Start with one and monitor your "queue depth" (how many jobs are waiting). If the depth grows, spin up more workers (horizontal scaling).
• Can background jobs update the UI? Not directly. Use WebSockets or polling to notify the user once the background job completes.
• What if the worker crashes mid-task? Most modern queues track "in-flight" messages. If the worker doesn't acknowledge the job, the message becomes visible again after a timeout, allowing another worker to pick it up.

Recap

Background jobs are the key to building responsive, high-performance systems. By offloading heavy tasks to a dedicated worker service, you ensure the main application remains free to handle incoming user requests. Always design for failure by implementing retries and monitoring your queues for persistent errors.

Up next: We will discuss API Versioning and Documentation, ensuring that as our services grow, our contracts remain stable.