WordPress Plugin Development: Asynchronous Webhooks with Backoff

Last month, a client’s integration started dropping about 15% of their outbound events because the third-party endpoint was flapping. My initial code was firing wp_remote_post() directly inside the save_post hook, which meant if the external API was slow, the user’s dashboard hung for seconds. When it failed, the data was simply lost.

We’ve all been there. Synchronous execution is the death of a stable WordPress site. To fix this, I had to move away from direct calls and build an asynchronous delivery engine that treats failures as expected behavior rather than exceptions.

The Problem with Synchronous Webhooks

In typical WordPress plugin development, we often hook into post_updated or woocommerce_order_status_changed and fire off a request. If the external server takes 800ms to respond, you've just added 800ms to your admin dashboard load time. If the server times out, the user gets no feedback, and your integration fails silently.

We first tried adding a simple try-catch block around the request. That helped with errors, but it didn't solve the retry logic. If the server was down, the data was gone. We needed a queue.

Architecting Asynchronous Processing

To handle asynchronous processing, I moved the payload into a custom database table. Instead of sending the request immediately, I insert the event into a webhooks_queue table and fire a WP-Cron task or an Action Scheduler job to process it later.

If you aren't using Action Scheduler yet, start now. It’s the same library WooCommerce uses, and it handles the heavy lifting of background task queues better than native WP-Cron.

Here is the basic schema for our queue table:

• id: Primary Key
• payload: JSON blob of the data to send
• target_url: The endpoint
• attempts: Integer (default 0)
• last_attempt: Timestamp
• next_retry: Timestamp

Implementing Exponential Backoff

When a REST API webhooks request fails, you shouldn't retry immediately. That’s how you get blacklisted for DDOS-ing the receiver. You need a strategy to space out the attempts.

Exponential backoff calculates the delay based on the number of attempts. A common formula is delay = 2^attempts * base_interval.

PHP
function get_next_retry_time($attempts) {
    $base = 60; #6A9955">// 1 minute
    $delay = pow(2, $attempts) * $base;
    return time() + $delay;
}

If the first attempt fails, it retries in 2 minutes. The second failure? 4 minutes. By the fifth attempt, it waits 32 minutes. This gives the external service breathing room to recover.

Building the Retry Engine

Your processing loop needs to check the next_retry column before attempting a delivery. If the current time is less than next_retry, skip the row.

PHP
public function process_queue() {
    global $wpdb;
    
    $jobs = $wpdb->get_results("SELECT * FROM webhooks_queue WHERE next_retry <= NOW() LIMIT 10");

    foreach ($jobs as $job) {
        $response = wp_remote_post($job->target_url, ['body' => $job->payload]);

        if (is_wp_error($response) || wp_remote_retrieve_response_code($response) >= 400) {
            $this->handle_failure($job);
        } else {
            $this->delete_job($job->id);
        }
    }
}

This ensures that your main thread remains untouched by external latency. If you find your queue growing too large, you might look into WordPress Performance: Asynchronous Database Write-Queues for REST APIs to ensure your database operations don't bottleneck the delivery process itself.

Lessons Learned

One thing I didn't account for initially was the "poison pill" payload. If a specific payload is malformed and causes a 400 Bad Request every time, the retry logic will just keep hammering the endpoint forever.

I eventually added a max_attempts cap of 5. After the 5th failure, the job moves to a failed_webhooks table for manual review. This is crucial for maintaining system integrity, much like how you would handle errors when WordPress plugin development: Implementing the Circuit Breaker Pattern is required to stop cascading failures.

FAQ

Why not just use a standard queue service like RabbitMQ? For WordPress, keeping the infrastructure native is usually easier for maintenance. Action Scheduler provides a persistent queue that doesn't require managing external server-side workers.

How do I monitor these background tasks? I recommend creating a simple admin page that queries your queue table. If you see the attempts count rising on multiple rows, you know your outbound integration is hitting a wall.

Does this increase database load? Yes, every webhook event is now a database row. Ensure your webhooks_queue table has an index on the next_retry column so your SELECT queries remain fast as the table grows.

I'm still tinkering with the ideal base interval for the backoff. For some APIs, 60 seconds is too aggressive; for others, it's too slow. In the next iteration, I'll probably make the interval configurable per-endpoint.