Laravel Horizon Job Pre-emption: Managing Priority Queues with Lua

During a recent infrastructure migration, we ran into a classic distributed systems bottleneck: a flood of high-priority audit reports was burying our background worker fleet, causing the user-facing transactional emails to lag by nearly 40 seconds. We needed a way to perform Job Pre-emption without killing the worker processes or corrupting the job state.

Standard Laravel Horizon configurations are great for balancing throughput, but they don't natively support "interrupting" a job already in progress. If a worker has picked up a 2-minute PDF generation task, that worker is effectively blocked until completion.

The Problem with Standard Queues

We initially tried simply increasing the number of workers, but that led to CPU contention and memory exhaustion on our small 2GB RAM nodes. We then looked into pcntl_signal to kill processes, but that often left our database transactions in an inconsistent state.

The missing piece was a way to signal the worker to pause and re-queue its current task when a high-priority "interruption" event occurred. Since Laravel doesn't support this out of the box, we had to build a custom layer using Redis Lua Scripting.

Implementing Job Pre-emption with Lua

To achieve Priority Queuing without side effects, we store a "pre-emption flag" in Redis for each active worker process. If a high-priority job hits the queue, we push a signal to a specific Redis key that the worker checks during its processing loop.

Here is the Lua script we use to check the state atomicity:

LUA
-- check_preemption.lua
local worker_id = KEYS[1]
local flag = redis.call('GET', 'preempt:' .. worker_id)

if flag == '1' then
    redis.call('DEL', 'preempt:' .. worker_id)
    return 1
end
return 0

By using Laravel Redis Lua Scripting for Deterministic Rate Limiting patterns, we ensure that the check is atomic. The worker executes this check at safe "checkpoints" within the handle() method of our long-running jobs.

Architecting the Worker Interruption

To make this work within the Laravel ecosystem, I created a Preemptible trait. This trait forces the job to periodically verify if it should yield its execution slot.

PHP
trait Preemptible {
    public function shouldPreempt(): bool {
        return Redis::eval(file_get_contents('check_preemption.lua'), 1, $this->workerId) === 1;
    }

    public function handle() {
        foreach ($this->largeDataSet as $chunk) {
            if ($this->shouldPreempt()) {
                $this->release(); #6A9955">// Re-queue the job
                return;
            }
            $this->process($chunk);
        }
    }
}

This approach allows us to maintain Distributed Systems integrity. Because we use $this->release(), the job returns to the queue with its original priority, but it effectively clears the worker for the urgent task.

Why This Beats Simple Queue Priorities

You might ask why we didn't just use multiple queues. We already use Laravel Distributed Task Scheduling: Implementing Redis Leader Election to manage our cron jobs, but that doesn't solve the "long-running job hogging the worker" scenario.

When you use Laravel Queues and Redis Lua for Atomic Job Batching, you gain the ability to manipulate the queue state in real-time. By combining this with Horizon, you get:

1. Deterministic behavior: No more guessing if a job will finish.
2. Resource efficiency: High-priority jobs get the CPU cycles they need instantly.
3. Graceful degradation: Jobs that are interrupted simply resume where they left off later.

Caveats and Lessons Learned

The biggest risk here is "pre-emption loops." If your high-priority jobs are too frequent, low-priority jobs will never finish. We solved this by implementing a "max-retry" logic on the pre-emption: if a job has been pre-empted three times, we ignore any further pre-emption signals for that specific job instance.

I’m still not entirely convinced that this is the cleanest way to handle worker orchestration. If I were rebuilding this today, I’d likely look into moving the heavy lifting to a sidecar process written in Go, using Laravel only as a control plane. However, for a pure PHP environment, this Lua-based interruption pattern is the most reliable way I've found to keep our queues responsive without throwing away work.