Redis Idempotency Pattern: Preventing Duplicate API Requests

We’ve all been there: a client-side network hiccup causes a user to hammer the "Submit" button, or an automated service retries a POST request that actually succeeded but timed out. Without proper handling, you end up with double-charged credit cards or duplicated database entries. Solving this requires a reliable idempotency pattern in distributed systems: A practical guide, and Redis is usually the fastest tool for the job.

I once spent an entire afternoon debugging why our payment service was occasionally processing the same transaction twice. We were relying on database unique constraints, but the overhead of checking the DB before every request was adding about 30ms of latency to our critical path. Moving the check to Redis changed everything.

The Redis SETNX Strategy

The core of the Redis idempotency pattern is the SETNX command (SET if Not eXists). It’s atomic, meaning Redis guarantees that if two requests arrive at the exact same millisecond, only one will succeed in setting the key.

Here is the basic logic flow:

1. Generate or receive a unique idempotency_key from the client.
2. Attempt to SET the key in Redis with a short TTL (Time To Live), using the NX flag.
3. If the command returns 1 (success), proceed with your business logic.
4. If it returns 0 (failure), the request is a duplicate; return a cached response or a "409 Conflict" status.

While REST API Idempotency: Implementing Keys to Prevent Duplicates covers the general concept, the actual implementation requires careful handling of the TTL. If you set it too short, you risk a race condition where the key expires before the operation finishes. If you set it too long, you bloat your memory usage.

Implementation Details

When I implement this, I prefer using a Lua script to ensure atomicity between the "check" and the "set" steps, especially if I need to associate the key with a specific user ID.

Bash
# Basic SETNX with a 1-hour expiration
SET idempotency:order:12345 "processing" NX EX 3600

If you're looking for Laravel REST API Development expertise, I often wrap this logic in a middleware. The middleware intercepts incoming requests, checks the header (e.g., X-Idempotency-Key), and short-circuits execution if the key exists.

Why not just use the database?

I’ve seen engineers try to solve this using a simple UNIQUE constraint in Postgres. That works for data integrity, but it doesn't solve the "API response" problem. If the DB constraint triggers, you have to catch the exception, parse it, and then figure out what the original response was. With Redis, you can store the actual serialized API response as the value for the key. When a duplicate hits, you just return the cached JSON.

The Trade-offs

Nothing is free. Here are the hurdles I've hit:

• Clock Skew: If your distributed nodes don't have synchronized clocks, your TTLs might behave inconsistently. Use NTP.
• Memory Pressure: If you have millions of requests, you must set an expiration. Don't let keys live forever.
• False Positives: If your business logic fails after the key is set but before the transaction completes, you’re stuck. You need a way to clean up the key on failure (the DEL command).

Comparison Table: Idempotency Strategies

A Better Approach for Background Jobs

If you're dealing with background processing, the strategy changes slightly. For those cases, I often look at Laravel Horizon Idempotency: Building Deterministic Redis Task Keys to ensure that even if a job is retried by the queue worker, the downstream side effects remain consistent.

FAQ

Q: What happens if the Redis server goes down? A: Your idempotency check will fail. Depending on your business requirements, you should either fail-open (allow the request) or fail-closed (block the request until Redis is back).

Q: How long should the TTL be? A: It depends on your client's retry strategy. Usually, 24 hours is plenty, as most clients will give up on a request long before that.

Q: Do I need a unique key for every request? A: Yes, the key should be a hash of the request payload or a client-generated UUID. If the payload changes, the idempotency key should change too.

I still worry about edge cases where a process crashes exactly after setting the key but before committing to the DB. To handle this, we've started implementing a "two-phase" check where we mark the key as processing and only update it to completed once the DB transaction succeeds. It’s more code, but it’s the only way to be 100% sure in high-stakes environments.