Laravel Horizon Idempotency: Building Deterministic Redis Task Keys

Last month, a simple race condition in a payment processing queue cost us about two hours of manual reconciliation. We were firing background jobs via Laravel Horizon, but network retries caused the same transaction to hit our gateway twice, leading to duplicate charges. It's the classic distributed systems headache: how do you ensure a job runs exactly once when the infrastructure is inherently unreliable?

If you're relying on database state alone, you’re already behind. You need a way to track job intent before the execution even begins.

Why Laravel Horizon Idempotency Needs Redis

When we talk about Laravel Horizon and idempotency, we're really talking about state management. Database transactions are great, but they’re slow for high-throughput deduplication. If you have 500 jobs hitting the queue per second, checking a processed_jobs table in MySQL creates a bottleneck.

We initially tried using a unique database column with a UNIQUE constraint on the job payload hash. It worked, but the performance hit was noticeable. Our latency jumped by roughly 12ms per job because of the index contention. We needed something faster, so we moved the deduplication layer into Redis.

Implementing Content-Addressable Task Keys

The core concept here is content-addressability. Instead of using a random UUID, we generate a deterministic key based on the job's unique business intent. If the input data is the same, the key must be the same.

Here is how we handle this in our ShouldQueue jobs:

PHP
public function getDeduplicationKey(): string
{
    #6A9955">// Deterministic hash of the job's business parameters
    return 'job_dedup:' . md5(json_encode([
        'user_id' => $this->user->id,
        'action'  => 'process_payment',
        'amount'  => $this->amount,
        'token'   => $this->transactionToken,
    ]));
}

By hashing the core parameters, we create a signature that represents the "intent" of the job. Even if the job is dispatched three times due to a network glitch, the deduplicationKey remains identical.

Atomic Deduplication with Redis

Once we have the key, we need to ensure that only one worker processes it. We use a Redis SET command with the NX (Not Exists) and EX (Expire) flags. This is the gold standard for distributed systems locking.

I prefer putting this logic in a base job class or a middleware. Here’s a simplified version of a Deduplicatable trait:

PHP
public function handleDeduplication(): bool
{
    $key = $this->getDeduplicationKey();
    
    #6A9955">// Set the key with a 1-hour TTL, only if it doesn't exist
    $result = Redis::set($key, 'processing', 'EX', 3600, 'NX');

    return (bool) $result;
}

If handleDeduplication returns false, the job has already been processed or is currently in flight. We then simply release the job or delete it from the queue. This pattern is far more efficient than querying a SQL database, and it dovetails perfectly with the principles discussed in Laravel API integration idempotency: Handling Webhooks with Redis.

Handling Edge Cases and Failures

Nothing in Redis is perfectly permanent. If your worker crashes after setting the key but before finishing the job, you’ve created a "zombie" lock. This is why the EX (expire) flag is non-negotiable.

We set our TTL to slightly longer than the max expected execution time—usually around 10 minutes for our specific workload. If the job fails, we clear the key in the failed() method of the job:

PHP
public function failed(Throwable $exception)
{
    Redis::del($this->getDeduplicationKey());
}

This ensures that retries are possible if the failure was transient. If you're building complex state machines, you might want to look into Laravel Workflow: Architecting Asynchronous State Machines for Reliability to manage these transitions more formally.

Are There Trade-offs?

Yes, absolutely. The biggest trade-off is the assumption that Redis is always available. If your Redis cluster goes down, your deduplication layer vanishes, and you might experience duplicate processing. We mitigate this by using a dedicated Redis instance for job state, separate from our cache or session storage.

We also acknowledge that this isn't "perfect" exactly-once processing. It's "at-least-once delivery with effective deduplication." In distributed systems, perfect exactly-once is a myth—you’re always managing trade-offs between consistency and availability.

I’m still experimenting with using Lua scripts to combine the GET and SET operations into a single atomic round-trip. It would shave off a few more milliseconds, but it adds complexity to the deployment pipeline. For now, the standard SET NX approach is holding up under a load of about 400 jobs per second without any major incidents.

Next time, I’d probably look into moving this logic into a dedicated middleware layer to keep the job classes cleaner. It’s easy to forget to call handleDeduplication() when you’re in a rush to ship a feature.