REST API Idempotency: Implementing Keys to Prevent Duplicates
REST API idempotency is essential for reliable retries. Learn how to implement idempotency keys to stop duplicate requests and fix race conditions easily.
Last month, a client reported that their billing service was double-charging users during intermittent network timeouts. We traced the issue to a classic distributed systems problem: the client would send a POST request, the server would process it, but the ACK would get lost in transit. The client, assuming the request failed, would retry, triggering a second charge.
Implementing REST API idempotency is the only reliable way to fix this. If you don't account for the "at-least-once" delivery nature of network calls, your system will eventually corrupt its own data.
Why You Need Idempotency Keys
At its core, an idempotency key is a unique identifier (usually a UUID) generated by the client and sent in an HTTP header, typically Idempotency-Key. When your server receives a request with this header, it checks a persistent store—like Redis or Postgres—to see if that key has been processed before.
If the key exists, you return the cached response from the previous execution instead of re-running the logic. This is how you make idempotency keys in databases: preventing duplicate transactions effective, ensuring that even if a network packet is duplicated, your database state remains consistent.
The Implementation Flow
We initially tried to implement this using a simple database flag, but we ran into race conditions where two concurrent requests with the same key would both check the database, see no result, and both initiate processing. We had to move to a "lock-then-check" strategy.
Here is the standard flow for handling these requests:
Flow diagram: Receive Request → Key in Cache?; B -- Yes → Return Cached Response; B -- No → Create Pending Record; Create Pending Record → Execute Business Logic; Execute Business Logic → Store Result & Mark Done; Store Result & Mark Done → Return Result
When you handle API idempotency: implementing deterministic correlation IDs for safety, you essentially treat the request as a state machine. The state transitions from PENDING to COMPLETED. If a retry hits while the state is PENDING, you should return a 409 Conflict or implement a short-lived lock.
Managing State and Storage
I’ve found that using Redis with SETNX (Set if Not Exists) is the most performant way to implement this. You set the key with a TTL (Time To Live), typically around 24 hours. This prevents your cache from growing indefinitely while giving the client enough of a window to retry their failed requests.
When building your idempotency pattern in distributed systems: a practical guide, keep these rules in mind:
- Client-side generation: The client must generate the UUID. If the server does it, the client won't know the key to use on a retry.
- Standardized Responses: When a duplicate request arrives, don't just return a 200 OK. Return the exact same response body, status code, and headers as the original successful request.
- Atomic Operations: Use database transactions or distributed locks to ensure the check-and-set operation is atomic.
| Strategy | Pros | Cons |
|---|---|---|
| Database Unique Constraint | Simple, ACID compliant | Can be slow under high load |
| Redis SETNX | Extremely fast | Requires extra infrastructure |
| In-Memory Map | Fastest, zero latency | Won't work across multiple nodes |
Handling Errors Properly
When a request fails, you need to decide if the idempotency key should be discarded. If the failure was a transient network error, keep the key. If the failure was a validation error (e.g., 400 Bad Request), you might want to allow the client to fix the payload and retry with the same key.
Follow REST API error handling: standardizing with RFC 7807 to ensure your error responses are predictable. If a client retries a request that failed due to a server error, they should see a consistent error structure, not a generic "something went wrong" message.
Frequently Asked Questions
What happens if the client sends a different body with the same key?
You should validate the request body against the one stored with the initial idempotency key. If they differ, return a 400 Bad Request to signal a mismatch.
How long should I keep idempotency keys?
24 hours is usually sufficient for most client-side retry logic. If your system requires longer windows, ensure you have a TTL-based cleanup strategy to prevent storage bloat.
Should I use idempotency keys for GET requests?
No. GET requests should be naturally idempotent by design. Idempotency keys are specifically for state-changing operations like POST, PUT, or PATCH.
What if my request hits a timeout?
If you're interested in how to manage these boundaries, check out API design: enforcing request timeout budgets for distributed systems. It's critical to ensure your timeouts are shorter than your idempotency window.
I'm still tinkering with how to handle "in-flight" requests more gracefully. Right now, returning a 429 Too Many Requests when a duplicate key is currently being processed is our best bet, but it feels a bit heavy-handed. If you have a cleaner way to handle concurrent retries, I’d love to hear it.