Laravel Distributed Task Scheduling: Implementing Redis Leader Election

When we scaled our primary application to run across four separate Kubernetes pods, our standard php artisan schedule:run setup immediately started firing duplicate tasks. It turns out that having every container hit the database simultaneously to check the jobs table isn't just inefficient; it's a recipe for double-billed invoices and corrupted state.

If you’re moving beyond a single-server deployment, you’ve likely realized that standard cron isn't built for high availability. To solve this, we need a way to ensure only one "leader" node triggers the schedule.

The Problem with Standard Cron

In a monolithic environment, * * * * * is fine. But in a distributed system, you aren't just dealing with one server—you're dealing with a cluster. If you have five nodes, you have five processes trying to execute the same logic at the same time.

We first tried using a central database flag to lock the schedule execution. That failed because if the node holding the lock crashed mid-execution, we were stuck with a "zombie" lock that required manual cleanup. It was fragile and added unnecessary latency to our database.

Implementing Distributed Redis Leader Election

To achieve true Laravel distributed systems orchestration, we need a primitive that handles TTL (Time To Live) automatically. Redis is perfect for this because we can use atomic SET operations with an NX (set if not exists) flag and an expiration time.

Here is the approach we landed on. Instead of relying on the native scheduler to run on every node, we gate the execution behind a custom Redis::set call.

PHP
#6A9955">// In app/Console/Kernel.php or a dedicated ServiceProvider
protected function schedule(Schedule $schedule)
{
    $schedule->call(function () {
        $lockKey = 'scheduler_lock';
        $acquired = Redis::set($lockKey, 'true', 'EX', 55, 'NX');

        if (!$acquired) {
            return;
        }

        #6A9955">// Proceed with your scheduled tasks
        $this->runMyTasks();
    })->everyMinute();
}

By setting the TTL to 55 seconds (just shy of our one-minute interval), we ensure that if a node dies, the lock expires before the next cycle, allowing another node to pick up the slack. This is the foundation for robust high availability in background processing.

Refining the Orchestration

While simple locks work for basic tasks, complex workflows often require more than a simple boolean flag. If you are handling mission-critical state, you should look into Laravel Workflow: Architecting Asynchronous State Machines for Reliability to ensure that your tasks are not just triggered reliably, but executed atomically.

However, locking is only half the battle. If your tasks are resource-intensive, you might accidentally starve your queue workers. When designing your task scheduling architecture, always keep in mind how your recurring tasks interact with Laravel Job Queuing: Architecting Weighted Fair Queuing with Redis. You don't want your scheduler to flood your workers and delay real-time user requests.

Lessons Learned in Production

We’ve been running this pattern for about 18 months, and it’s been rock solid. The biggest takeaway? Keep the leader election logic separate from the task logic.

If you find yourself needing to prevent race conditions beyond just scheduling—such as ensuring two processes don't update the same user record—you should implement Laravel Distributed Locks: Preventing Race Conditions with Redis. The concepts are similar, but the implementation details for granular locks require more care regarding deadlock prevention.

FAQ

Q: What happens if the Redis cluster is down? A: In our current setup, the scheduler fails silently. Depending on your needs, you might want to wrap the lock acquisition in a try-catch block and alert your on-call engineer if the scheduler cannot acquire a lock for three consecutive minutes.

Q: Is 55 seconds the optimal TTL? A: It depends on your task duration. If your tasks take longer than a minute, you need a different locking strategy, perhaps one that extends the TTL dynamically while the task is running.

Q: Can I use Laravel’s built-in onOneServer() method? A: Yes, Laravel actually has a built-in onOneServer() method that uses Redis under the hood. For 90% of use cases, you should use that instead of rolling your own. I built this manual implementation because we needed custom observability into which node was acting as the leader at any given timestamp.

I’m still not entirely satisfied with how we handle "missed" tasks. If the leader node restarts exactly at the start of a minute, we occasionally lose a cycle. Next time, I’d look into a secondary heartbeat mechanism, though that adds significant complexity to the infrastructure. Keep it simple until you absolutely have to make it complex.