WordPress background processing: Implementing Local Message Queues

Last month, I was debugging a site that crashed every time a user initiated a bulk product import. The culprit wasn't the database or the server load; it was the sheer number of synchronous API calls firing during the request lifecycle. We were trying to process 5,000 external API integrations on a single thread. It’s a classic trap in plugin development: trying to force WordPress to be an event-driven system while treating it like a standard CRUD app.

If you’re building plugins that handle heavy data lifting, you need a way to offload that work. Standard WP-Cron is fine for clearing transient caches, but it’s notoriously unreliable for high-volume, time-sensitive jobs. When you need to scale, you need to look at dedicated WordPress background processing.

Why WP-Cron isn't enough

WP-Cron is not a real cron job. It triggers only when a page is loaded. If your site has low traffic, your jobs don't run. If your site has high traffic, you have dozens of concurrent processes fighting for the same resources.

We initially tried using a custom wp_remote_post approach inside the main execution thread. It failed because the PHP execution time limit hit before the third batch of requests finished. We then looked at WordPress Sidecar Architecture: Scaling Plugins for High Concurrency, but for this specific client, we needed a solution that lived entirely within the existing infrastructure. That’s when we pivoted to a local message queue implementation.

Architecting a local message queue

A robust queue needs three things: a storage layer, a producer, and a consumer. For a local WordPress implementation, we use a custom database table to act as our queue, allowing us to manage state and retries effectively.

1. Storage: Create a table wp_queue_tasks with columns for id, payload (JSON), status (pending, processing, failed), attempts, and next_run.
2. Producer: When a task is triggered, don't run it. Save it to the table and return a success message to the user immediately.
3. Consumer: This is a CLI-based script that runs as a system daemon (using systemd or Supervisor).

PHP
#6A9955">// The Producer: Simple push to queue
function enqueue_task($task_name, $data) {
    global $wpdb;
    $wpdb->insert("{$wpdb->prefix}queue_tasks", [
        'task_name' => $task_name,
        'payload'   => json_encode($data),
        'status'    => 'pending',
        'created'   => current_time('mysql')
    ]);
}

By decoupling your logic, you improve your WordPress performance: Asynchronous Database Write-Queues for REST APIs by ensuring the main request-response loop stays under 200ms.

Handling heavy-duty asynchronous tasks

Once the data is in the queue, you need a worker to pick it up. Since we're using a CLI worker, we bypass the web server entirely. This means we aren't bound by max_execution_time or memory limits defined in php.ini for the web process.

You should implement an exponential backoff strategy if your tasks involve external API calls. This is critical for WordPress plugin development: Implementing the Circuit Breaker Pattern, as it prevents your queue from hammering a failing external service.

When building your consumer, keep the worker loop tight:

PHP
#6A9955">// The Consumer: Basic loop
while (true) {
    $task = get_next_pending_task();
    if (!$task) {
        sleep(5); #6A9955">// Don't burn CPU cycles
        continue;
    }
    
    process_task($task);
}

Challenges and trade-offs

One of the biggest issues I ran into was race conditions. If two workers pick up the same task, data integrity goes out the window. We solved this by using UPDATE ... WHERE status = 'pending' LIMIT 1 with a FOR UPDATE lock, or simply relying on atomic transaction updates.

Another challenge is logging. When your processing is asynchronous, you lose the browser console. You must invest in a centralized logging system—even if it's just a dedicated wp-content/debug-queue.log file—to track why jobs fail.

Is it worth the complexity?

Implementing dedicated asynchronous tasks adds roughly two days of development time to a project. However, the payoff is immense. You gain the ability to handle spikes in traffic without the site timing out. You also gain better control over the execution environment.

If you’re working on high-scale systems, you'll eventually hit the limits of a single MySQL database. At that point, WordPress Performance: Implementing Database Partitioning for Scale becomes necessary, but don't optimize the database until you've successfully offloaded the task execution first.

FAQ

Can I use Redis for the queue instead of MySQL? Yes, and you should if your host supports it. Using Redis as a backend for your queue is significantly faster than querying MySQL, as it avoids disk I/O for every status check.

How do I monitor these background workers? I recommend using a simple heartbeat system. Have your worker update a last_heartbeat transient every 30 seconds. If the transient expires, trigger an alert via your monitoring service.

Does this replace Action Scheduler? Not necessarily. Action Scheduler is a great library for this, but building your own local message queue gives you total control over the architecture. If you're building a highly custom plugin, the DIY approach is often cleaner than forcing a third-party library to fit your specific data structures.

I’m still experimenting with how to handle dead-letter queues effectively. Currently, we just move failed tasks to a separate table, but I’d love to see a more automated approach to re-queuing after a manual review. If you're building for scale, remember: the best architecture is the one that stays out of the user's way.