Distributed Locking: Managing Concurrency in Scalable Systems

Previously in this course, we explored handling large data imports to manage high-volume workloads. In this lesson, we address the challenge of distributed locks, which are essential when multiple instances of a service must coordinate access to a shared resource without corrupting state.

Why Distributed Locks?

In a single-process application, you might use language-level primitives like mutex or synchronized blocks. However, when your architecture scales horizontally, these local locks are invisible to other nodes. If two instances attempt to update the same user account or inventory item simultaneously, you encounter a race condition.

A distributed lock acts as a global "traffic light" for your services. It ensures that only one process can perform a specific operation at any given time, maintaining consistency across your distributed environment.

Implementing Distributed Locks with Redis

We use Redis for application caching as our coordination layer because of its speed and atomic command support. The core principle is simple: use a shared key in Redis to represent the lock.

To avoid race conditions during the locking process itself, we use the SET command with arguments that ensure the operation is atomic:

PYTHON
# Pseudo-code for atomic lock acquisition
import time

def acquire_lock(redis_client, lock_key, request_id, ttl_seconds):
    # NX: Only set if the key does not exist
    # EX: Set expiration time in seconds
    return redis_client.set(lock_key, request_id, nx=True, ex=ttl_seconds)

def release_lock(redis_client, lock_key, request_id):
    # Use a Lua script to ensure we only delete the lock we own
    lua_script = CE9178">"""
    if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
    else
        return 0
    end
    """
    return redis_client.eval(lua_script, 1, lock_key, request_id)

The NX flag ensures that only one client succeeds in creating the key. If the key already exists, the SET operation returns None, indicating the lock is currently held by another process.

Handling Lock Expiration and Deadlocks

One of the most dangerous scenarios in distributed systems is a deadlock. This occurs if a service acquires a lock and then crashes or loses network connectivity before releasing it. Without a mechanism to expire that lock, the resource stays "locked" forever.

To prevent this, we must use a Time-To-Live (TTL). By setting an expiration on the Redis key, we guarantee the lock will be released automatically if the owner fails to finish its work.

However, choosing the right TTL is an art:

1. Too short: The lock expires while the process is still working, allowing a second process to acquire the lock and causing data corruption.
2. Too long: If a service crashes, other services are blocked for an unnecessarily long time.

Pro-tip: Use a "heartbeat" or "lock renewal" pattern if your tasks are long-running. Periodically extend the TTL of the lock while the process is still active.

Hands-on Exercise

For your running project, identify one resource in your system that requires exclusive access (e.g., updating a user's wallet balance or processing a payment).

1. Implement the acquire_lock and release_lock logic shown above.
2. Simulate a crash: Acquire a lock, then manually kill your service process before the release_lock function executes.
3. Verify that the lock expires after the TTL and that a subsequent process can then successfully acquire it.

Common Pitfalls

• Fencing Tokens: If a process experiences a long "stop-the-world" garbage collection pause, it might think it still holds the lock when the TTL has actually expired. Always include a unique request ID (fencing token) in your database updates to ensure that late-arriving requests from "expired" lock holders are rejected.
• Clock Skew: Relying on system clocks across different servers to manage lock duration is dangerous. Always rely on the centralized clock provided by your Redis instance.
• Deleting Others' Locks: Never simply call DEL on a key. Always verify that the current process owns the lock (using the Lua script pattern above) before deleting it, otherwise, you might accidentally delete a lock just acquired by another process.

FAQ

Is Redis the only way to implement distributed locks? No. You can use databases like PostgreSQL (via pg_advisory_lock) or coordination services like Zookeeper or Etcd. Redis is often preferred for its performance, but it provides "at-least-once" safety semantics; for "strictly consistent" requirements, consider consensus-based systems.

What is the impact of a Redis failure? If your Redis instance goes down, your locking mechanism fails. Ensure your Redis setup is highly available (using Sentinel or Cluster mode) to minimize this risk.

Recap

Distributed locks are critical for ensuring concurrency control. By using atomic SET NX operations and TTLs, you can prevent race conditions while ensuring the system recovers from process failures, avoiding permanent deadlocks.

Up next: We will explore Event-Driven Architecture to move toward asynchronous, decoupled system designs.