Handling Large Result Sets: SCAN, Pagination, and Streaming

Previously in this course, we covered Analyzing Memory Usage to identify bloat in your database. Now that you know how to find "big keys," this lesson teaches you how to retrieve that data without crashing your application or freezing your Redis instance.

In Redis, the golden rule of performance is never to block the event loop. Because Redis is single-threaded, a long-running command stops everything else, causing your API to hang.

Why You Must Avoid the KEYS Command

When searching for keys, beginners often reach for KEYS *. While it works in a development environment with five keys, it is catastrophic in production. KEYS is an O(N) operation—it scans the entire keyspace. If you have a million keys, Redis stops processing every other request until the command completes.

Instead, use SCAN. SCAN provides a cursor-based iterator that allows you to retrieve keys in small batches. It doesn't block the server, and it returns a new cursor you can use to fetch the next set of results.

Worked Example: Scanning Keys Safely

Here is how to perform an iterative scan in Node.js using ioredis:

JAVASCRIPT
async function scanAllKeys(pattern = CE9178">'*') {
  let cursor = CE9178">'0';
  do {
    const reply = await redis.scan(cursor, CE9178">'MATCH', pattern, CE9178">'COUNT', 100);
    cursor = reply[0]; // Next cursor
    const keys = reply[1]; // Batch of keys
    
    for (const key of keys) {
      console.log(CE9178">'Found key:', key);
    }
  } while (cursor !== CE9178">'0');
}

The COUNT argument is a hint to the server. By setting it to 100, we retrieve keys in small, manageable chunks, keeping the latency impact near zero.

Pagination for Lists and Sets

When you have a massive List or Set, you shouldn't fetch all members at once. If your "request_log" list (from Using Lists for Request Logging) grows to 50,000 items, LRANGE list 0 -1 will serialize all 50,000 items and send them over the wire, likely causing a memory spike in your application.

Instead, implement pagination using LRANGE with explicit indices.

JAVASCRIPT
// Fetch page 1(indices 0 to 49)
const page1 = await redis.lrange(CE9178">'my_large_list', 0, 49);

// Fetch page 2(indices 50 to 99)
const page2 = await redis.lrange(CE9178">'my_large_list', 50, 99);

For Sets, use SSCAN (the Set variant of SCAN) to iterate through members without fetching the entire set into memory.

Streaming Large Data

For truly large result sets, pagination can be tedious to coordinate. Streaming is the professional approach. In Node.js, we can use the scanStream method provided by most Redis clients to handle the cursor logic automatically.

JAVASCRIPT
const stream = redis.scanStream({
  match: CE9178">'user:session:*',
  count: 50
});

stream.on(CE9178">'data', (keys) => {
  // Process this batch of 50 keys
  console.log(CE9178">'Processing batch:', keys);
});

stream.on(CE9178">'end', () => {
  console.log(CE9178">'All keys processed.');
});

This approach is non-blocking and memory-efficient. It treats the Redis data as a continuous pipe, ensuring your application memory remains stable even if you are processing millions of records.

Hands-on Exercise

1. Create a loop in your script that populates a list with 1,000 dummy keys.
2. Write a function that uses LRANGE to fetch those keys in chunks of 50.
3. Verify that your script prints "Page processed" 20 times, rather than loading all items at once.

Common Pitfalls

• Ignoring the Cursor: If you don't save the cursor returned by SCAN, you'll start your search from the beginning every time.
• Assuming Count is Exact: The COUNT argument is only a hint. Redis might return more or fewer keys than requested; your code must be prepared to handle batches of varying sizes.
• Using KEYS in "Just one" script: There is no such thing as a "safe" use of KEYS in production. If you need it, you need SCAN.

FAQ

Q: Does SCAN guarantee that I'll see every key? A: SCAN guarantees that you will see all keys that existed from the start of the iteration to the end. Keys added or deleted during the process might or might not appear.

Q: Is SCAN slower than KEYS? A: It is technically more work for the server to manage the cursor, but it is significantly faster for your application because it avoids the "stop-the-world" effect.

Recap

Managing large result sets is about control. By replacing blocking commands like KEYS with SCAN and implementing pagination or streams for large collections, you ensure your Redis instance remains responsive under high load. Much like we discussed in Data Fetching Architecture, the goal is to break big tasks into manageable, asynchronous chunks.

Up next: We will connect our Redis data to the front end by Integrating Redis with WebSockets.