Scaling Laravel Queues on Kubernetes: A KEDA Implementation Guide

During our Q3 marketing push, we hit a wall. Our static deployment of 10 Laravel queue workers couldn't keep up with a sudden burst of 45,000 jobs hitting our Redis instance. While the CPU usage on our nodes hovered at a modest 30%, the job latency skyrocketed from a standard 120ms to over 14 seconds. The root cause wasn't lack of compute power; it was the fact that our Horizontal Pod Autoscaler (HPA) was watching CPU metrics, which are effectively useless for background job processing.

Scaling Laravel Queues with KEDA

If you’re running Laravel on Kubernetes, you’ve likely realized that scaling based on CPU or memory is a recipe for disaster. Background workers are often I/O bound, waiting on external APIs or database queries. By the time your CPU hits a threshold, your job queue is already a graveyard of delayed tasks. We needed to scale based on the actual depth of our Redis list, not the health of the container's processor.

Enter KEDA (Kubernetes Event-driven Autoscaling). KEDA allows you to define custom triggers that scale your deployments based on external events. When we shifted to KEDA and Prometheus: Mastering Event-Driven Autoscaling, we stopped treating our workers like web servers and started treating them like event consumers.

The Configuration Journey

We first tried scaling using the standard HPA metrics-server with a custom metric for queue depth. It broke because the polling interval was too slow and the metrics were often stale by the time the controller acted. We then moved to a ScaledObject configuration. Here is the YAML that finally gave us the granularity we needed:

YAML
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: laravel-worker-scaler
spec:
  scaleTargetRef:
    name: laravel-worker
  minReplicaCount: 2
  maxReplicaCount: 50
  triggers:
  - type: redis
    metadata:
      address: redis-master.default.svc.cluster.local:6379
      listName: queues:default
      listLength: "20"

In this setup, KEDA checks the Redis list queues:default every 30 seconds. If the length exceeds 20, it triggers a scale-out event. We configured our nodes to handle this burst using Implementing Kubernetes Node Auto-Provisioning: Karpenter and Bottlerocket to ensure that when KEDA requests 50 pods, the cluster actually has the capacity to spin them up without waiting for manual intervention.

Lessons from the Field

The transition wasn't seamless. We initially set the listLength threshold too low, which caused a "thundering herd" effect where our workers scaled up and down every two minutes. This churn killed our database connection pool. We stabilized it by introducing a cooldownPeriod of 300 seconds, ensuring that once we scaled up, the workers stayed active long enough to clear the backlog.

While this setup works, I’m still cautious about the "cold start" problem. Even with KEDA, there's a delay between the trigger and the pod being ready to process a job. We're currently experimenting with keeping a higher minReplicaCount during known high-traffic hours to mitigate this. I'm also still unsure if we should move our queue driver from Redis to SQS to better leverage KEDA's native AWS scalers, which might offer more consistent latency metrics.

FAQ

Why not just use the standard Horizontal Pod Autoscaler?

The standard HPA scales based on resource consumption (CPU/RAM). Laravel queue workers are typically idle while waiting for I/O, meaning they won't trigger an HPA scale-up until the queue is already backed up.

How does KEDA know how many workers to spin up?

KEDA calculates the number of replicas by dividing the current queue length by the listLength target. If your target is 20 and you have 100 jobs, KEDA will attempt to scale your deployment to 5 replicas.

What happens if the Redis connection drops?

KEDA enters a "failed" state for that trigger. Your deployment will remain at its last known scale or fall back to the minReplicaCount defined in your ScaledObject.

---

Scaling Laravel applications in a cloud-native environment requires shifting your mindset from resource-based monitoring to event-based scaling. By leveraging KEDA, you can ensure your infrastructure reacts to actual demand rather than arbitrary hardware metrics.