Laravel Horizon job pre-emption allows you to interrupt low-priority tasks for urgent work. Use Redis Lua scripting to manage atomic queue state effectively.
During a recent infrastructure migration, we ran into a classic distributed systems bottleneck: a flood of high-priority audit reports was burying our background worker fleet, causing the user-facing transactional emails to lag by nearly 40 seconds. We needed a way to perform Job Pre-emption without killing the worker processes or corrupting the job state.
Standard Laravel Horizon configurations are great for balancing throughput, but they don't natively support "interrupting" a job already in progress. If a worker has picked up a 2-minute PDF generation task, that worker is effectively blocked until completion.
We initially tried simply increasing the number of workers, but that led to CPU contention and memory exhaustion on our small 2GB RAM nodes. We then looked into pcntl_signal to kill processes, but that often left our database transactions in an inconsistent state.
The missing piece was a way to signal the worker to pause and re-queue its current task when a high-priority "interruption" event occurred. Since Laravel doesn't support this out of the box, we had to build a custom layer using Redis Lua Scripting.
To achieve Priority Queuing without side effects, we store a "pre-emption flag" in Redis for each active worker process. If a high-priority job hits the queue, we push a signal to a specific Redis key that the worker checks during its processing loop.
Here is the Lua script we use to check the state atomicity:
LUA-- check_preemption.lua local worker_id = KEYS[1] local flag = redis.call('GET', 'preempt:' .. worker_id) if flag == '1' then redis.call('DEL', 'preempt:' .. worker_id) return 1 end return 0
By using Laravel Redis Lua Scripting for Deterministic Rate Limiting patterns, we ensure that the check is atomic. The worker executes this check at safe "checkpoints" within the handle() method of our long-running jobs.
To make this work within the Laravel ecosystem, I created a Preemptible trait. This trait forces the job to periodically verify if it should yield its execution slot.
PHPtrait Preemptible { public function shouldPreempt(): bool { return Redis::eval(file_get_contents('check_preemption.lua'), 1, $this->workerId) === 1; } public function handle() { foreach ($this->largeDataSet as $chunk) { if ($this->shouldPreempt()) { $this->release(); #6A9955">// Re-queue the job return; } $this->process($chunk); } } }
This approach allows us to maintain Distributed Systems integrity. Because we use $this->release(), the job returns to the queue with its original priority, but it effectively clears the worker for the urgent task.
You might ask why we didn't just use multiple queues. We already use Laravel Distributed Task Scheduling: Implementing Redis Leader Election to manage our cron jobs, but that doesn't solve the "long-running job hogging the worker" scenario.
When you use Laravel Queues and Redis Lua for Atomic Job Batching, you gain the ability to manipulate the queue state in real-time. By combining this with Horizon, you get:
The biggest risk here is "pre-emption loops." If your high-priority jobs are too frequent, low-priority jobs will never finish. We solved this by implementing a "max-retry" logic on the pre-emption: if a job has been pre-empted three times, we ignore any further pre-emption signals for that specific job instance.
I’m still not entirely convinced that this is the cleanest way to handle worker orchestration. If I were rebuilding this today, I’d likely look into moving the heavy lifting to a sidecar process written in Go, using Laravel only as a control plane. However, for a pure PHP environment, this Lua-based interruption pattern is the most reliable way I've found to keep our queues responsive without throwing away work.
Master Laravel Horizon idempotency by implementing content-addressable task keys and Redis deduplication for exactly-once processing in your background jobs.