LLM Observability: Tracking Token Latency and Semantic Drift

When we pushed our first RAG-based support agent to production, we thought the logs were enough. We were wrong. A week later, users started complaining that the answers felt "off," but our standard error rates were green. It turns out, the model was experiencing a slow degradation in output quality, which is the hallmark of silent failure in AI.

To fix this, we had to build a robust approach to LLM observability that treated tokens and intent as first-class metrics. Here is how you can move from blind faith to data-driven monitoring.

Why Token Latency Tracking Matters

Most developers monitor request-response time, but that’s a vanity metric for LLMs. Because models stream tokens, the time to first token (TTFT) and the total tokens-per-second (TPS) are what actually define the user experience.

If your TTFT spikes, your app feels sluggish regardless of how fast the total generation is. We started by wrapping our OpenAI client calls in a simple decorator to capture the usage object returned by the API.

PYTHON
import time
from functools import wraps

def track_llm_metrics(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time()
        response = func(*args, **kwargs)
        duration = time.time() - start_time
        tokens = response.usage.total_tokens
        # Log these to your time-series DB (e.g., Prometheus/Grafana)
        print(f"Latency: {duration:.2f}s | TPS: {tokens/duration:.2f}")
        return response
    return wrapper

This gave us a baseline. We noticed that during peak hours, our latency for specific model versions fluctuated by roughly 350ms. By having this data, we could trigger alerts before the latency impacted user satisfaction scores.

Monitoring Semantic Drift in Production

Tracking latency is the easy part. Detecting semantic drift—where your model’s responses slowly diverge from the intended behavior—is where most teams get stuck. As discussed in LLM Observability: Detecting Semantic Drift in Production Pipelines, you need to compare incoming query embeddings against a "golden dataset."

We initially tried using simple keyword matching to detect drift, but it failed to catch nuanced changes in tone or context. We switched to cosine similarity scoring. By embedding both the user query and the model’s response, we could calculate the semantic distance from our expected output patterns.

Integrating RAG Evaluation

If you’re building a RAG pipeline, you cannot monitor the LLM in isolation. You have to monitor the retrieval process too. I recommend looking into Implementing Semantic Chunking for RAG Pipelines: A Practical Guide to ensure that the context you're feeding the model is actually high quality. If the retrieval is bad, the LLM will drift no matter how "perfect" your system prompt is.

When we implemented production AI monitoring, we set up a shadow evaluation loop. For every 10th request, we run a secondary, smaller model (like GPT-4o-mini) to grade the primary model's response against the retrieved context. It’s not cheap, but it’s the only way to catch semantic drift before it hits your support queue.

The Trade-offs of Real-time Monitoring

We first tried to evaluate every single request, but it ballooned our API costs by nearly 40%. We backed off to a sampling strategy. We now monitor 100% of latency metrics but only 5-10% of semantic quality. This balance keeps our infrastructure costs sustainable while still giving us a statistically significant view of how the model is behaving in the wild.

If you are just starting, don't try to build everything at once. Focus on:

1. Token Accounting: Track usage per feature to keep budgets in check, as detailed in LLM Cost Monitoring: A Guide to Granular Token Accounting.
2. Latency Baselines: Establish what "normal" looks like for your specific prompts.
3. Drift Detection: Start with a small, curated set of test cases that you run against every production deployment.

I’m still experimenting with how to automate the "golden dataset" updates. Right now, it’s a manual process of reviewing flagged logs every Friday. It’s not perfect, but it’s better than being blind.

FAQ

How do I know if my LLM is experiencing semantic drift? Look for a decrease in the cosine similarity scores between your production outputs and your evaluation test set. If the distance grows over time, your model is drifting.

Is token latency tracking enough to optimize costs? No. You need to combine latency data with granular token usage logs to understand if you're over-provisioning or using inefficient models for simple tasks.

What is the best way to start RAG evaluation? Start by measuring retrieval precision and recall using frameworks like RAGAS before you even look at the LLM's final response quality.