Laravel Queues: Implementing Redis Lua Scripting Rate Limiting

Last month, our primary job processor hit a wall. We were firing thousands of webhooks to a third-party API, and despite using Laravel’s built-in RateLimiter facade, we kept tripping their circuit breakers. The issue wasn't the queue itself; it was the "thundering herd" effect where multiple workers checked the rate limit, saw available capacity, and simultaneously flooded the endpoint before the counter could update.

We needed a way to enforce strict, atomic concurrency control across our distributed fleet of workers. That’s when we moved from simple application-level checks to Redis Lua scripting.

Why Standard Rate Limiting Fails at Scale

When you use the standard RateLimiter::attempt() in Laravel, the logic happens in two distinct steps: read the current count from Redis, and increment it if it's below the threshold. In a distributed environment, two processes can read the same "available" count at the exact same microsecond.

This creates a race condition. By the time both workers write their increment back to Redis, you’ve effectively bypassed your limit. If you're interested in the theory behind protecting your multi-tenant infrastructure, API Rate Limiting with Token Bucket Algorithms for Multi-Tenant SaaS breaks down the math, but here we’re concerned with the implementation.

We first tried adding a simple Redis::lock() around the check-and-set operation. It worked, but it added about 40ms of latency per job—an unacceptable tax when you're processing 500 jobs per second. The solution is to move the logic into the database engine itself.

Architecting with Redis Lua Scripting

Redis executes Lua scripts atomically. No other command can run while your script is executing, which effectively gives us a transaction-like guarantee without the overhead of heavy locking.

Here is a simplified version of a sliding window script we deployed to manage our worker throughput:

LUA
-- KEYS[1] = rate_limit_key
-- ARGV[1] = current_timestamp
-- ARGV[2] = window_size (e.g., 60 seconds)
-- ARGV[3] = max_requests

redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])
local current_count = redis.call('ZCARD', KEYS[1])

if current_count < tonumber(ARGV[3]) then
    redis.call('ZADD', KEYS[1], ARGV[1], ARGV[1] .. math.random())
    return 1
else
    return 0
end

By using a Sorted Set (ZSET), we keep track of individual timestamps as members. We clean up expired entries with ZREMRANGEBYSCORE before checking the count. Because this runs inside Redis, the "check-and-add" happens in a single, uninterruptible block.

Integrating with Laravel Queues

To make this useful, you shouldn't manually call Redis::eval() in every job class. We wrapped this in a custom middleware. This allows us to keep our worker logic clean while enforcing Laravel Queues constraints globally.

PHP
namespace App\Http\Middleware;

use Illuminate\Support\Facades\Redis;

class RateLimitMiddleware
{
    public function handle($job, $next)
    {
        $allowed = Redis::eval($this->script, 1, 'webhook_limit', time(), 60, 100);

        if (! $allowed) {
            return $job->release(10);
        }

        return $next($job);
    }
}

This approach significantly reduced our error rate. We effectively transformed a non-deterministic distributed system into a predictable pipeline. If you're handling complex task deduplication, you might also want to look at Laravel Horizon Idempotency: Building Deterministic Redis Task Keys to ensure that retries don't inflate your throughput metrics unnecessarily.

Trade-offs and Lessons Learned

The primary trade-off here is observability. When logic lives inside a Lua script, standard Laravel debuggers won't show you exactly what's happening inside the execution loop. We had to implement custom logging within the script to track why specific jobs were being throttled.

Also, keep your Lua scripts lean. If you perform heavy computation inside the script, you will block the entire Redis instance, which can cause a cascading failure across your entire High-Throughput Architecture. Keep the logic to simple set operations and counters.

FAQ

Does this work with Redis Cluster? Yes, but you must ensure that your keys use a hash tag (e.g., {webhook_limit}) so that all related keys reside on the same shard. Without hash tags, Redis Cluster will throw an error because it cannot guarantee atomicity across shards.

What if the Redis instance goes down? Your workers will likely fail fast. We prefer this "fail-closed" approach because it prevents us from accidentally overwhelming the downstream API if our monitoring tools aren't functioning correctly.

How does this compare to Laravel's built-in RateLimiter? Laravel’s native implementation is great for HTTP requests, but for high-frequency background jobs, the Lua-based approach is significantly faster because it reduces the number of round-trips to the Redis server.

I’m still experimenting with how to handle "burst" capacity—where we allow a sudden spike if the long-term average is low. For now, the strict sliding window is keeping our workers stable. It’s not perfect, but it’s deterministic, which is exactly what I need when I’m on-call.