Advanced Queue Monitoring: Mastering Laravel Horizon

Previously in this course, we explored Asynchronous Processing with Queues, where we learned to offload slow tasks like email sending or report generation. While basic queue drivers work fine for development, production environments require deeper visibility and control, which brings us to Laravel Horizon.

Horizon is a powerful dashboard and configuration system for your Redis-backed queues. It provides a real-time view of your job throughput, runtime, and failure rates, allowing you to manage your infrastructure as code.

Monitoring with Laravel Horizon

Horizon gives you a "bird's eye view" of your entire queue system. Once installed via composer require laravel/horizon, it exposes a dashboard that tracks metrics in real-time.

The primary value of Horizon is the config/horizon.php file, which allows you to define your worker "environments" and "supervisors." Instead of manually running php artisan queue:work on every server, Horizon manages these processes for you.

Tuning Worker Configurations

Effective queue management depends on balancing resource consumption against job latency. In your config/horizon.php, you define how many processes should run for specific queues:

PHP
'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['high', 'default'],
            'balance' => 'auto', #6A9955">// Horizon will dynamically scale based on load
            'processes' => 10,
            'tries' => 3,
        ],
    ],
],

The balance option is critical here. Using auto tells Horizon to distribute worker processes across your queues based on which queue has the most pending work. This prevents a backlog in your high priority queue while default workers sit idle.

Handling Failed Jobs

Even with perfect code, external API outages or database lock conflicts occur. Horizon simplifies the lifecycle of failed jobs:

1. Visibility: The "Failed Jobs" tab shows exactly why a job failed, including the stack trace and the payload.
2. Retry Strategy: You can trigger a retry directly from the dashboard.
3. Pruning: To prevent your Redis database from growing indefinitely, configure trim settings in horizon.php to automatically remove old failed job records.

If you are dealing with complex failures, consider implementing a Dead Letter Queue pattern to isolate problematic tasks without blocking your main processing pipeline.

Advanced Workflow: The Project Board

In our project board application, we use queues to process heavy tasks like generating PDF exports for project reports. To ensure these jobs don't starve other tasks, we define them in a specific reports queue.

Update your horizon.php configuration to prioritize this:

PHP
'supervisor-1' => [
    'connection' => 'redis',
    'queue' => ['reports', 'default'],
    'balance' => 'simple', #6A9955">// 'simple' is more predictable for specific needs
    'processes' => 5,
],

Hands-on Exercise

1. Install Horizon in your local project board environment.
2. Navigate to /horizon in your browser.
3. Dispatch ten dummy jobs to a custom queue named heavy-lifting.
4. Observe the "Workload" tab in the Horizon dashboard.
5. Modify your horizon.php to set the processes count for heavy-lifting to 1, then run the jobs again and observe how the wait time increases compared to a higher process count.

Common Pitfalls

• Forgetting to deploy horizon:terminate: When you deploy new code, you must run php artisan horizon:terminate. This tells the master supervisor to restart and pick up the new code changes.
• Redis Memory Limits: Horizon stores metric data in Redis. If your Redis instance is small, ensure you are pruning old data to avoid OOM (Out of Memory) errors.
• Ignoring Job Timeouts: If a job runs longer than the timeout defined in your configuration, Horizon will kill the process. Ensure your timeout is always slightly higher than your longest expected job duration.

For those running highly complex architectures, remember that distributed tracing can be difficult with async queues. If you find your monitoring tools aren't showing the full picture, you may need to manually propagate trace headers into your job payloads.

Summary

By using Horizon, you move from "blind" queueing to a managed, observable system. You gain the ability to adjust capacity on the fly, visualize bottlenecks, and recover from failures with a single click.

Up next: Job Chaining and Batching — we'll learn how to coordinate multiple dependent jobs and handle failures for entire groups of tasks simultaneously.