Building a Distributed Lock: Synchronization Patterns in Redis

Previously in this course, we explored how to use Introduction to Lua Scripting to group commands into atomic blocks. In a distributed environment, however, we often need to synchronize access to a resource across multiple separate service instances. That is where distributed locks come in.

When you have multiple instances of an API writing to the same database row or updating a shared resource, you need a mechanism to ensure that only one instance performs the operation at a time. This is the essence of concurrency control.

The Anatomy of a Distributed Lock

A distributed lock isn't just a key in Redis; it’s a protocol. To build a robust lock, you must ensure three things:

1. Mutual Exclusion: Only one client can hold the lock at any given time.
2. Deadlock Prevention: If a process crashes while holding a lock, the lock must eventually expire so other processes can proceed.
3. Safety: A client should only release the lock it acquired.

The Simple Lock Pattern

In Redis, we use the SET command with the NX (Not Exists) and PX (Milliseconds) options. This is an atomic operation that sets the key only if it doesn't already exist and attaches an expiration time.

JAVASCRIPT
// Example: Acquiring a lock
const lockKey = CE9178">'lock:user:123:update';
const requestId = CE9178">'unique-node-id-xyz'; // Important for safe releases
const ttl = 5000; // 5 seconds

const acquired = await redis.set(lockKey, requestId, CE9178">'PX', ttl, CE9178">'NX');

if (acquired) {
  // We have the lock! Perform protected logic
  // ...
} else {
  // Lock held by someone else
}

Implementing a Safe Lock Release

A common mistake is simply calling DEL on the lock key. If a process hangs for longer than the TTL, the lock might expire and be claimed by another process. If the first process then calls DEL, it might accidentally delete the lock owned by the second process.

To fix this, we use a Lua script to verify the owner before deleting.

LUA
-- release_lock.lua
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end

By passing the requestId as ARGV[1], we ensure that our application only deletes the lock it actually created. This pattern is essential for preventing race conditions in distributed systems.

Addressing Edge Cases

Distributed systems are inherently unreliable. When designing your synchronization, keep these edge cases in mind:

• Clock Skew: Relying on system time across different servers is dangerous. Always use Redis-side expiration (PX).
• Lock Duration: Choose a TTL that covers the worst-case execution time of your protected logic. If your code takes longer than the TTL, your lock will expire while the task is still running, leading to race conditions.
• Redlock Theory: For high-availability setups, the Redis Distributed Lock: Preventing Race Conditions in Microservices documentation suggests using multiple Redis nodes to ensure that if one node fails, the lock remains consistent. For most beginners, a single-instance lock is sufficient, but keep this in mind as you scale.

Hands-on Exercise

1. Update your project to include a function acquireLock(key, ttl) and releaseLock(key, identifier).
2. Use the SET key value NX PX ttl pattern for acquisition.
3. Implement the Lua script provided above for the release function to ensure safety.
4. Test by spinning up two Node.js processes that try to "claim" the same resource simultaneously.

Common Pitfalls

• Forgetting the TTL: If you don't set an expiration time, a process crash will result in a permanent lock (deadlock), requiring manual intervention to clear the key.
• Ignoring the Identifier: Never delete a lock without checking if you still own it. You will inadvertently break concurrency for other processes.
• Over-locking: Only lock the specific resource you need. If you lock a generic "user" key when you only need to update "user_preferences," you will create unnecessary contention.

FAQ

Q: Why not just use SETNX and EXPIRE as separate commands? A: SETNX followed by EXPIRE is not atomic. If the process crashes between the two commands, the lock will never expire. Always use SET key value NX PX ttl to perform both in one atomic step.

Q: What if my task takes longer than the lock duration? A: You should either increase the TTL or implement a "heartbeat" mechanism that extends the lock TTL as long as the process is still running.

Q: Are distributed locks slow? A: They add network latency to your operation. If you find yourself locking extremely frequently, consider if you can redesign your logic to avoid the need for global synchronization.

Recap

We’ve learned that distributed locks rely on atomic SET operations with TTLs to ensure safety and prevent deadlocks. By utilizing unique identifiers and Lua scripts for release, we protect our systems against race conditions even when multiple instances are running.

Up next: We will combine our cache service and locking logic into a unified architecture in our final project refactoring.