Laravel Job Queuing: Architecting Weighted Fair Queuing with Redis

We hit a wall last year when our primary notification pipeline started drowning out critical billing synchronization tasks. Standard Laravel queues are First-In-First-Out, which is fine for simple apps, but once you're processing 50,000+ jobs an hour, FIFO becomes a liability. We needed a way to ensure that high-priority billing events always jumped the line without completely starving the background notification workers.

If you’re struggling with similar bottlenecks, you might have already tried the standard priority queue approach. We did too. We split our queues into high, default, and low, but we quickly found that if the high queue is flooded, the low queue never gets a CPU cycle. It’s a classic starvation problem. That’s when we pivoted to implementing Weighted Fair Queuing (WFQ) using Redis Sorted Sets.

Moving Beyond FIFO with Redis Sorted Sets

Instead of relying on Laravel’s native queue drivers, we decided to treat Redis as a custom priority buffer. By using Sorted Sets (ZSET), we can assign a score to each job. Redis then automatically sorts the set, allowing us to fetch the "highest" priority item in constant time.

Here is the basic architecture:

1. Producer: When a job is dispatched, we push it to a Redis ZSET where the score is the timestamp plus a priority weight.
2. Consumer: A custom worker process polls the ZSET, pops the member with the lowest score, and dispatches it into a local Laravel queue instance.

This approach gives you granular control over Laravel Job Queuing logic without fighting the framework’s internal queue runners.

Implementing the Middleware

We didn't want to rewrite our entire job infrastructure, so we encapsulated the priority logic within a Job Middleware. This keeps the implementation clean and allows us to toggle prioritization on a per-job basis.

PHP
namespace App\Jobs\Middleware;

use Illuminate\Support\Facades\Redis;

class WeightedPriority
{
    public function handle($job, $next)
    {
        $priority = $job->priority ?? 0; #6A9955">// Higher is more important
        $score = microtime(true) - ($priority * 100);

        Redis::zadd('job_priority_buffer', $score, serialize($job));

        return; #6A9955">// We stop the standard dispatch flow
    }
}

Wait—why microtime(true) - ($priority * 100)? Since Redis ZSET sorts in ascending order (lowest score first), subtracting the priority from the current timestamp forces higher-priority jobs to have a "smaller" score, pushing them to the front of the line. It's a simple, effective trick for Performance Optimization that avoids complex Lua scripts unless you're handling massive concurrency.

Addressing the Trade-offs

The biggest trade-off here is observability. When you move jobs into a custom ZSET, you lose the native Laravel Horizon dashboard metrics for those jobs. We had to build a custom monitoring tool to track the size of our ZSET and the latency between "queued" and "processed."

If your application requires strict consistency, consider how this interacts with Database Caching: Mastering the Cache-Aside Pattern for Scale. You don't want your background jobs pulling stale data while they're being re-prioritized. We eventually moved our job state into Redis itself, which helped bridge the gap between the queue and our Database caching: Implementing Redis Write-Through for Consistency strategy.

Scaling with System Architecture

When you're designing this System Architecture, keep in mind that Redis is single-threaded. If you perform too many ZADD and ZRANGE operations, you’ll see latency spikes in your primary cache. We offloaded the job processing to a dedicated Redis instance, separate from our application cache.

For teams dealing with complex, multi-step processes, you might find that Laravel Workflow: Architecting Asynchronous State Machines for Reliability is a better fit than managing raw priority queues. We use Workflows for the heavy, long-running tasks, while our Weighted Fair Queuing handles the high-frequency, short-lived jobs.

FAQ: Common Implementation Hurdles

Q: Does this replace Laravel Horizon? A: Not entirely. You can think of it as a pre-processor. You’re routing jobs through a priority filter before they ever touch the standard Laravel queue workers.

Q: What happens if the worker crashes mid-process? A: That’s the danger of custom queues. You lose the native "failed_jobs" table functionality. You’ll need to implement a "processing" set in Redis to track jobs in-flight and move them back to the main set if the worker heartbeat fails.

Q: Is this overkill for most apps? A: Yes. If you have fewer than 10,000 jobs per day, stick to standard priority queues. The complexity of managing System Architecture with custom Redis sets only pays off when you hit the limits of standard FIFO processing.

I'm still not 100% satisfied with our error handling. If a job fails in our custom worker, it’s currently a "black hole" unless we manually pipe it back into the ZSET. Next time, I’d probably look at using a dedicated package for this or investing more time into a more robust retry mechanism using Laravel Redis Lua Scripting for Deterministic Rate Limiting to ensure our queue workers don't overwhelm the database during spikes. It’s messy, but it’s real-world Performance Optimization that keeps our production systems humming.