Semantic caching for RAG pipelines: Cut latency and costs

Last month, our team noticed a sharp spike in our OpenAI API bill. We were running a standard RAG pipeline to answer internal technical documentation questions, but our users kept asking the same five variations of "How do I reset my credentials?" over and over again. Every single one of those requests triggered a full embedding generation, a vector database search, and a costly call to GPT-4o.

We needed a way to intercept these redundant requests before they ever hit the LLM. That’s when we moved from simple exact-match caching to semantic caching for RAG pipelines.

Why exact-match isn't enough

When we first started controlling LLM cost and latency, we used a simple Redis key-value store. It worked for identical strings, but real human language is messy. "How do I reset my password?" and "I need to change my login credentials" are semantically identical to a user, but they don't match in a standard cache.

We tried tokenizing and normalizing strings, but that only solved about 15% of our redundancy issues. We needed something that understood intent. If you're currently relying on simple hash maps, you're leaving a lot of performance on the table.

Implementing semantic caching with vector search

The core idea behind semantic caching is simple: instead of checking for an exact key, you treat the user's prompt as a vector and check if a "close enough" query already exists in your cache.

Here is the basic flow:

1. Generate an embedding for the incoming user query.
2. Perform a similarity search against your cache (we used FAISS locally for this, but Pinecone or Milvus work too).
3. If the distance score is below a certain threshold (e.g., cosine similarity > 0.92), return the cached response.
4. If the threshold isn't met, proceed with the standard RAG flow and store the new result in the cache.

The technical setup

We used sentence-transformers for embedding generation and a lightweight Redis instance to hold both the vector and the associated LLM response.

PYTHON
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

model = SentenceTransformer(CE9178">'all-MiniLM-L6-v2')

def get_cached_response(query, threshold=0.92):
    query_vec = model.encode([query])
    # Search your local FAISS index
    dist, idx = index.search(query_vec, k=1)
    
    if dist[0][0] > threshold:
        return retrieve_from_redis(idx[0][0])
    return None

In our production environment, this simple check saves us roughly 30% of our daily token spend. The latency overhead of the vector search is around 12ms, which is negligible compared to the ~2.5 seconds we save by skipping the LLM entirely.

Lessons learned from production

Integrating this into our existing building a small RAG pipeline end to end in Python workflow wasn't without its headaches. The biggest trap is the threshold.

If you set your similarity threshold too high, your cache hit rate plummets. Set it too low, and you start returning answers that are "close" but technically incorrect—a disaster if you're dealing with strict data. We found that 0.92 was our sweet spot for technical documentation, but your mileage will vary based on your domain.

Also, don't forget to implement an eviction policy. We use a Time-To-Live (TTL) of 24 hours on our cache entries. Since our internal documentation changes frequently, stale answers are worse than no cache at all.

Integrating with existing patterns

Semantic caching pairs beautifully with hybrid search in RAG pipelines. While the cache handles the "easy" repetitive questions, the hybrid search handles the nuanced, long-tail queries that require fresh retrieval.

If you are just starting, don't overengineer it. Start with an in-memory vector store like FAISS or even ChromaDB. Once you have the logic flow down, you can move to a distributed store like Redis or Weaviate if you need to scale across multiple instances.

Frequently Asked Questions

Q: Does semantic caching replace the need for RAG? A: No. It complements RAG. It acts as a gatekeeper to prevent unnecessary computation for high-frequency, similar queries.

Q: How do you handle cache invalidation? A: We use a TTL approach. For a more robust solution, you could trigger a cache clear whenever your source documents are updated in your vector database.

Q: What is the biggest downside? A: Memory usage. Storing embeddings for every unique query takes up significantly more space than storing simple string keys. Keep an eye on your memory limits.

Final thoughts

Semantic caching for RAG pipelines is a high-leverage optimization. It’s one of those rare engineering tasks where you get a massive reduction in operational costs with relatively low complexity.

Next time, I want to experiment with multi-modal caching—specifically caching results for image-based queries in our RAG system. I’m still unsure how to handle the similarity threshold for multimodal embeddings, as the distance metrics seem much more sensitive to noise compared to text. If you've tackled that, I'd love to hear how you handled the sensitivity.