WordPress Performance: Using Bloom Filters for REST API Scaling

Last month, I was debugging a REST API endpoint that handled membership validation for a custom user-meta integration. We were seeing a constant stream of SELECT queries hitting the wp_usermeta table, even for users that clearly didn't exist in our system. The database CPU was spiking, and our cache hit ratio was hovering around 60%. I realized we were treating every request as a potential database hit, which is a classic trap when scaling WordPress.

To fix this, I implemented a Bloom filter. It’s a space-efficient, probabilistic data structure that tells you if an item definitely isn't in a set or might be in a set. By placing this filter in front of our database lookups, we effectively stopped querying the disk for non-existent keys.

Why WordPress performance needs probabilistic structures

In a standard WordPress setup, checking for the existence of a record usually triggers a get_user_meta() or a direct SQL SELECT. If you're building a high-scale REST API, these repeated checks for non-existent IDs create a "cache penetration" issue. The database works hard to return nothing.

We previously tried using standard Redis caching to store negative results, but the memory overhead for caching millions of "misses" was unsustainable. That's where Bloom filters for efficient membership testing in high-cardinality data become game-changing. They provide a way to verify membership with a known, tiny error rate without bloating your RAM.

Implementing Bloom filters in a WordPress plugin

To implement this, I used a basic PHP implementation of a Bloom filter. You’ll need to define the expected number of items ($n$) and the acceptable false-positive probability ($p$). For our user-meta use case, I targeted $n = 1,000,000$ and $p = 0.01$.

PHP
#6A9955">// Simple Bloom Filter concept for WordPress
class WP_Bloom_Filter {
    private $bit_array;
    private $hash_functions = 7;

    public function __construct($size = 1000000) {
        $this->bit_array = new SplFixedArray($size);
    }

    public function add($item) {
        foreach ($this->get_hashes($item) as $hash) {
            $this->bit_array[$hash] = 1;
        }
    }

    public function exists($item) {
        foreach ($this->get_hashes($item) as $hash) {
            if (!$this->bit_array[$hash]) return false;
        }
        return true; #6A9955">// Might exist
    }
}

When a request hits your REST API, the logic flow changes:

1. Check the Bloom filter.
2. If it returns false, return a 404 immediately. No database query is executed.
3. If it returns true, perform the actual database query or check the primary cache.

Avoiding common pitfalls

We initially tried to persist the bit array as a serialized string in wp_options, but that broke almost immediately. The I/O cost of unserializing a massive string on every request negated the performance gains. We eventually moved the filter into an APC user cache or a shared memory segment, which dropped our average latency by about 45ms per request.

It's also worth noting that Bloom filters are "add-only." You can't easily remove an item once it's added. If your data set changes frequently, you'll need to periodically rebuild the filter. I usually handle this with a background task that rebuilds the filter every 24 hours, ensuring the false-positive rate stays within acceptable bounds.

Scaling the REST API further

Once you've offloaded these membership checks, you might still face bottlenecks in payload delivery or write latency. If your API is doing heavy writes, consider moving to WordPress Performance: Asynchronous Database Write-Queues for REST APIs. I’ve also found that WordPress REST API Performance: Brotli Compression for Headless SaaS provides a significant boost for clients consuming these endpoints over high-latency mobile networks.

FAQ: Scaling with Probabilistic Structures

Does a Bloom filter replace my database? No. It’s an optimization layer. It only tells you if you need to check the database. You still need the primary data store for the actual record retrieval.

What happens if the false-positive rate gets too high? If your filter is full, it will start returning true for everything. You have to monitor the fill ratio and rebuild the filter before it hits capacity.

Is this overkill for a standard blog? Absolutely. If you have fewer than 100,000 records and moderate traffic, standard object caching (Redis/Memcached) is more than enough. Use Bloom filters only when your database lookups for non-existent data become a measurable bottleneck.

I’m still experimenting with how to handle real-time updates to the filter without a full rebuild. Perhaps a secondary, smaller "delta" filter that gets merged periodically? It’s a messy problem, but it’s better than watching the database CPU climb while users wait for a 404.