Redis Pub/Sub vs Streams: Building Reliable Notifications
Redis Pub/Sub lacks message durability. Learn how to implement Redis Streams and Dead Letter Queues to ensure reliable notifications in your architecture.
During a recent refactor, I spent two days debugging why our real-time notification service was dropping roughly 5% of user alerts during transient network spikes. We were using standard Redis Pub/Sub, which is essentially "fire-and-forget"—if the subscriber isn't listening at the exact microsecond the message is published, that message is gone forever.
If you’re building an event-driven architecture, you need to accept that networks are unreliable and services will crash. Moving from volatile pub/sub to persistent streams is the single most effective way to harden your messaging layer.
Why Redis Pub/Sub Fails for Notifications
Redis Pub/Sub is fast, but it’s not designed for critical data. It doesn't store messages, it doesn't track acknowledgments, and it doesn't provide a way to replay events. When a subscriber disconnects, it misses everything until it reconnects.
We first tried to patch this by adding custom logic in our application layer to retry failed deliveries, but the race conditions were a nightmare. We eventually realized we were fighting the tool's core design. If you need message durability, you need a system that persists events to disk and tracks consumer state.
Transitioning to Redis Streams
Redis Streams provides the persistence we were missing. Unlike Pub/Sub, a stream retains messages until you explicitly delete them. More importantly, it supports Consumer Groups, which allow multiple workers to share the processing load while ensuring every message is processed at least once.
The Architecture Shift
When we moved to Streams, we gained the ability to "claim" pending messages if a worker crashes. This is a massive upgrade over the black-hole nature of Pub/Sub.
Flow diagram: Producer → Redis Stream; Redis Stream → Consumer Group; Consumer Group → Worker 1; Consumer Group → Worker 2; Worker 1 → Success; Worker 1 → Fail; Fail → Dead Letter Queue
Handling Failures with Dead Letter Queues
Even with a persistent stream, some messages will fail to process—perhaps due to a malformed payload or a downstream API being down. You don't want these "poison pills" blocking your entire processing pipeline.
This is where a Dead Letter Queue (DLQ) becomes mandatory. When a worker fails to process a message after a fixed number of retries (we usually set this to 3), we move the message to a separate "dead-letter" stream for manual inspection or later reprocessing. Implementing this pattern is similar to how we handle jobs in Laravel Queues: Building a Dead Letter Queue for Production Jobs, where isolating failures is the primary goal for system stability.
The Implementation Logic
- XREADGROUP: A worker reads a message from the stream.
- Process: The worker executes the notification logic.
- XACK: If successful, the worker acknowledges the message.
- XADD (DLQ): If the retry threshold is reached, push the message to the DLQ and acknowledge the original to clear the stream.
This approach ensures that your system remains consistent, much like the patterns we use for the Transactional Outbox Pattern: Using WAL for Reliable Event-Driven Systems to maintain state across distributed services.
Comparison: Pub/Sub vs. Streams
| Feature | Redis Pub/Sub | Redis Streams |
|---|---|---|
| Durability | None | Disk-backed |
| Consumer Groups | No | Yes |
| Message History | No | Yes |
| ACK Support | No | Yes |
| Complexity | Low | Medium |
Practical Advice for Production
Don't over-engineer your first implementation. Start by migrating one notification type to a stream. Monitor the PEL (Pending Entries List) in Redis using XPENDING. If you see the list growing, your consumers are either crashing or failing to acknowledge messages, which is your immediate signal to investigate.
I’m still experimenting with automatic TTL settings for my DLQs. While it’s tempting to keep failed messages forever, they eventually become noise. We currently prune them after 14 days, but that’s a heuristic, not a rule. You’ll need to balance your storage costs against your debugging needs.
FAQ
Can I use Redis Streams to replace Kafka? For many mid-sized applications, yes. If you need simple persistence and consumer groups, Redis Streams is much easier to manage than a full Kafka cluster. If you need massive throughput and complex stream processing (like KSQL), stick with Kafka.
What happens if the Redis server restarts? Because Redis Streams are persistent (assuming you have AOF or RDB enabled), your messages will survive a restart. This is a major advantage over Pub/Sub, where a restart wipes the state of every subscriber.
How do I monitor my stream health?
Use XINFO STREAM <key> to check the length and XINFO GROUPS <key> to see how many messages are pending across your consumers. If the "pending" count is consistently high, you need more consumers or better error handling.