Advanced Key Expiration Patterns: Sliding Windows and Scheduling

Previously in this course, we covered Implementing Expiration and TTL in Redis for Data Management. While basic TTLs are perfect for cache invalidation, they lack the granularity needed for complex traffic shaping or task scheduling.

In this lesson, we move beyond static expiration to implement dynamic, time-based patterns that power professional-grade APIs.

Moving Beyond Fixed Windows

In our earlier lessons on rate limiting, we used fixed-window counters. These reset at the start of a minute or hour, leading to "boundary bursts"—where a user hits their limit at the end of one window and immediately again at the start of the next.

A sliding window solves this by calculating the request rate over a rolling time frame. Instead of a single counter, we treat time as a sequence of events.

Implementing a Sliding Window with Sorted Sets

We use a Sorted Set (ZSET) to store individual request timestamps. The timestamp (in milliseconds) serves as both the score and the unique identifier (or appended with a random ID to handle multiple requests at the exact same millisecond).

Here is the logic:

1. Remove elements older than the current window (e.g., now - 60s).
2. Count the remaining elements.
3. Add the current request timestamp.
4. Set a TTL on the key to ensure it eventually clears if the user stops sending requests.

JAVASCRIPT
const redis = require(CE9178">'redis');
const client = redis.createClient();

async function isAllowed(userId, limit = 5, windowMs = 60000) {
  const now = Date.now();
  const key = CE9178">`ratelimit:${userId}`;
  
  // Use a transaction(MULTI/EXEC) or Lua for atomicity
  const multi = client.multi();
  
  // 1. Remove old timestamps
  multi.zRemRangeByScore(key, 0, now - windowMs);
  // 2. Count current window
  multi.zCard(key);
  // 3. Add current timestamp
  multi.zAdd(key, { score: now, value: CE9178">`${now}-${Math.random()}` });
  // 4. Set TTL to keep Redis clean
  multi.expire(key, Math.ceil(windowMs / 1000));
  
  const results = await multi.exec();
  const count = results[1];
  
  return count <= limit;
}

TTL-based Task Scheduling

While we use EXPIRE for automatic deletion, we can also use TTLs to simulate "delayed" jobs. By storing a task in a Sorted Set with a future timestamp as the score, we can treat the set as a priority queue.

Pattern: The Delayed Task Queue

1. Enqueue: Use ZADD with a score of Date.now() + delay.
2. Process: Use a background worker to run ZRANGEBYSCORE key 0 Date.now() LIMIT 0 1 to find tasks ready for execution.
3. Handle: Perform the task, then ZREM the item from the set.

Managing Expiry Events

Sometimes you need to trigger logic the moment a key expires (e.g., closing a user session or cleaning up an external resource). Redis provides Keyspace Notifications.

Enable them in your redis.conf or via CLI: CONFIG SET notify-keyspace-events Ex

This allows your application to subscribe to a special channel: __keyevent@0__:expired. Your Node.js application can listen for these notifications to perform cleanup, but remember: these events are fire-and-forget. They do not guarantee delivery if your subscriber is offline. For critical tasks, use the polling pattern mentioned in the task scheduling section instead.

Hands-on Exercise

1. Modify the isAllowed function above to use a Lua script (see Introduction to Lua Scripting) to ensure the ZREM, ZCARD, and ZADD occur in one atomic step.
2. Implement a "Delayed Cleanup" job that adds a key to a Sorted Set with a 10-second delay and prints "Task Done" when you detect it in the set.

Common Pitfalls

• Non-Atomic Operations: Always use Lua scripts or MULTI/EXEC for sliding windows. If you perform the ZREM and ZADD as separate network calls, race conditions will result in inaccurate counts.
• Memory Bloat: If you have millions of users, every sliding window key stays in memory until its TTL expires. Ensure your TTLs are as short as necessary.
• Clock Skew: If you run Redis across multiple servers, rely on TIME from the Redis server itself rather than Date.now() from the application server to maintain consistency across distributed nodes.

Summary

Advanced expiration isn't just about deleting keys; it's about managing the flow of data over time. By combining ZSET operations with TTLs, you gain precise control over rate limiting and task timing.

Up next: Scaling Redis with Replication.