Laravel OpenTelemetry Instrumentation: A Practical Guide

During our Q3 sprint, we had a customer ticket—#882—reporting that our checkout service was intermittently hanging for over 4 seconds. Our logs showed the database queries were fast, but the request lifecycle remained a black box. We were blind to the inter-service communication between our Laravel monolith and our secondary Go-based fulfillment microservice. To fix this, I decided it was time to implement Laravel OpenTelemetry instrumentation to gain full visibility into the request flow.

Implementing Laravel OpenTelemetry for Distributed Tracing

Before we could solve the latency issue, we had to get traces flowing. We started by installing the open-telemetry/opentelemetry-php SDK via Composer. It seemed straightforward, but we immediately hit a wall with how PHP handles process life cycles. Because PHP-FPM kills the process after every request, we couldn't rely on background exporter threads like you would in a Go or Java application.

We first tried using the default OTLP HTTP exporter. It added roughly 280ms to every request because the PHP process had to wait for the network round-trip to the collector before sending the response back to the user. That was a non-starter. We couldn't sacrifice that much performance for visibility.

Instead, we pivoted to using the OTLP gRPC exporter combined with a local OTel Collector agent running as a sidecar. By offloading the trace export to a Unix socket, we reduced the overhead to around 12ms. If you are running your services on Kubernetes, you should check out how to implement distributed tracing with Tempo to handle the trace storage backend efficiently.

Configuring the Middleware

To capture the traces, I wrote a custom Laravel middleware that hooks into the request lifecycle. Here is the simplified implementation we used:

PHP
use OpenTelemetry\API\Trace\SpanInterface;
use OpenTelemetry\API\Trace\TracerInterface;

public function handle($request, Closure $next)
{
    $span = $this->tracer->spanBuilder($request->path())->startSpan();
    $scope = $span->activate();

    try {
        return $next($request);
    } finally {
        $span->end();
        $scope->detach();
    }
}

This gave us the basic span, but it didn't capture the database queries or the outgoing HTTP requests to our fulfillment service. We had to manually instrument our Http facade and database drivers. It wasn't clean, but it was effective. For teams looking to scale this, I recommend reading our guide on end-to-end tracing and logging to understand how to consolidate these signals.

Trade-offs and Lessons Learned

The biggest hurdle wasn't the code; it was the sampling strategy. We initially tried to sample 100% of requests. We quickly realized our OTel collector was getting hammered, and our storage costs in Jaeger spiked by 40%. We dialed it back to a 10% head-based sampling rate, which was enough to catch the anomalies in our checkout flow without burning our infrastructure budget.

One thing I’m still not entirely happy with is the lack of automated instrumentation for some of our older Laravel packages. We’ve had to manually wrap several legacy controllers, which feels brittle. If we were starting from scratch, I’d probably look into using a dedicated service mesh to handle more of the heavy lifting. You can see how we manage that in our guide on mastering service mesh observability.

I'm still debating whether to move our trace collection to a purely asynchronous pattern using a Redis-backed queue. It would remove the overhead from the request path entirely, but it introduces a risk of losing traces if the queue worker crashes. For now, we're sticking with the sidecar collector, but I'm keeping a close eye on the performance metrics.

FAQ

Q: Does Laravel OpenTelemetry instrumentation slow down my app? A: It can. If you use an HTTP exporter directly in the request path, you'll see increased latency. Use an OTLP collector as a sidecar to minimize impact.

Q: How do I handle sampling in production? A: Don't sample 100%. Start with 5-10% and increase it only when you're actively debugging a specific issue to keep your observability costs in check.

Q: Is manual instrumentation required for every query? A: With the current state of PHP SDKs, yes, you often need to manually wrap critical paths, though some libraries provide auto-instrumentation hooks.

By focusing on efficient transport and conservative sampling, we finally managed to get the distributed tracing visibility we needed. It turned our "black box" into a readable timeline, allowing us to identify that the latency was actually coming from a misconfigured connection pool in our secondary service.