Few-shot prompting with vector search for better LLM context

Last month, our team spent three days debugging why our internal classification model was hallucinating on edge cases. We had hard-coded five static examples in the system prompt, thinking it was enough, but as the input variety grew, that fixed set became a liability. The model was trying to force-fit new, complex queries into templates that didn't actually match the intent.

That’s when we pivoted to a dynamic retrieval approach. Instead of static examples, we built a system that injects relevant history into the prompt at runtime.

Why static examples fail in production

Hard-coding your few-shot examples is fine for a prototype or a simple task with narrow input. But when your application handles diverse user intents, static prompts are a bottleneck. You’re essentially gambling that your top five examples cover 100% of the possible variance.

We found that when we moved away from static lists, our classification accuracy jumped by about 12% on our internal benchmark set. The magic wasn't just in the model; it was in providing the right "map" for the model to follow for a specific query.

Implementing dynamic few-shot prompting with vector search

The architecture for this is straightforward. You treat your bank of high-quality examples as a vector database. When a user sends a query, you perform a similarity search to find the most relevant examples, then inject those into your prompt template.

Here is how we implemented the retrieval flow:

1. Embedding: Store your gold-standard examples in a vector store like Implementing pgvector in Postgres for Semantic Search at Scale.
2. Retrieval: When a request arrives, generate an embedding for the user's input and query your store for the top k matches.
3. Prompt Injection: Format those k examples into the prompt dynamically.

The implementation code

Using Python and a standard vector search library, the logic looks roughly like this:

PYTHON
def get_dynamic_prompt(user_query, vector_store):
    # Retrieve the 3 most similar examples
    relevant_examples = vector_store.search(user_query, k=3)
    
    # Construct the prompt
    prompt = "Use the following examples to classify the request:\n"
    for ex in relevant_examples:
        prompt += f"Input: {ex.input}\nLabel: {ex.label}\n\n"
    
    prompt += f"Input: {user_query}\nLabel:"
    return prompt

This approach creates few-shot prompting that evolves with your data. If you have a Hybrid Search in RAG Pipelines: Boosting Retrieval Accuracy setup, you might even combine semantic similarity with keyword matches to ensure you're pulling the most technically accurate examples.

Managing the trade-offs

It isn't a silver bullet. The biggest risk is "context drift." If your retrieval logic pulls in noise or irrelevant examples, the LLM gets confused, and your performance might actually drop below the baseline.

We initially tried retrieving the top ten results, but the added token count caused our latency to spike by roughly 200ms per request. We settled on retrieving three high-confidence results. To keep things performant, we also integrated a Semantic caching for RAG pipelines: Cut latency and costs layer. This way, if someone asks a question similar to one we've already processed, we skip the vector search and LLM generation entirely.

Refining for stability

Even with dynamic context, you still need to ensure the output is usable. We found that pairing this with Structured output: Implementing Deterministic JSON Schema Validation was vital. Even if the retrieved examples help the model understand the task, you need that schema validation to ensure the output remains predictable for your application code.

Frequently Asked Questions

How many examples should I retrieve? Start with 3 to 5. Anything more often leads to diminishing returns and unnecessary token costs. Monitor your latency and trim if you exceed your budget.

What if my vector search retrieves irrelevant examples? This is a retrieval quality issue. You might need to refine your embeddings or use a cross-encoder to rerank your results. If you're struggling with accuracy, it's often a sign that your embedding model isn't capturing the nuance of your specific domain.

Does this increase cost significantly? Yes, every few-shot example consumes tokens. You're trading API costs for higher model reliability. If the cost is too high, consider using a smaller, faster model (like GPT-4o-mini or Haiku) for the task since the few-shot context makes the model's job much easier anyway.

I’m still not 100% sold on the "set it and forget it" nature of these systems. We’re currently exploring how to automatically prune our example database when we see the model failing on specific clusters of inputs. It’s a constant game of maintenance, but the reliability gains make the effort worth it.