LLM Function Calling for Efficient Structured Data Extraction

Last month, our team noticed a 40% spike in our monthly OpenAI bill after scaling our document processing pipeline. We were relying on long, descriptive system prompts to force the model into returning specific JSON objects for entity extraction, which is an expensive way to handle data.

If you’re still writing "You are a helpful assistant that returns JSON" prompts, you're wasting tokens. By switching to LLM function calling for structured data extraction, we managed to cut our input token count by roughly 30% per request while simultaneously improving the reliability of our downstream parsers.

Why Prompt Engineering Optimization Isn't Enough

We initially tried to optimize our costs by shortening our system prompts. We went from a 500-token instruction block down to 150 tokens. While it helped slightly, the model occasionally "hallucinated" extra fields or failed to follow the formatting rules, forcing us to implement Structured output: Implementing Deterministic JSON Schema Validation just to keep the application from crashing.

The problem with pure prompt engineering is that the model spends a significant amount of its attention (and your money) parsing your natural language instructions. When you use an API-supported tool definition, you offload that logic to the model's native training for function execution. It’s cleaner, faster, and significantly cheaper.

Implementing Function Calling for Data Extraction

Instead of asking the model to "extract the user's name, email, and subscription tier," you define a tool. The model is then forced to return a JSON object that perfectly matches your schema.

Here is a simplified example of how we define a tool in Python using the OpenAI SDK:

PYTHON
tools = [{
    "type": "function",
    "function": {
        "name": "extract_user_info",
        "description": "Extracts user data from the provided text.",
        "parameters": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "email": {"type": "string", "format": "email"},
                "tier": {"type": "string", "enum": ["free", "pro", "enterprise"]}
            },
            "required": ["name", "email", "tier"]
        }
    }
}]

By providing this schema, you replace paragraphs of instructions with a concise, machine-readable format. This is the core of API cost reduction; you aren't paying the model to "understand" how to format JSON, because the API handles the enforcement on the backend.

Comparing Extraction Approaches

When deciding how to handle your data, consider the trade-offs between standard prompting and function-based approaches:

As we discussed in LLM Function Calling: A Guide to Dynamic Tool Selection, the real power comes when you combine these schemas with strict validation.

Best Practices for Lowering Costs

If you are serious about reducing your token usage, follow these three rules:

1. Keep schemas lean: Don't include fields you don't need. Every key in your JSON schema adds to the input token count.
2. Reuse definitions: Create a library of shared schemas for your common extraction tasks.
3. Use JSON mode vs. Tool calling: If the model doesn't need to "select" a tool, use JSON mode. However, for structured data extraction where you need to parse specific entities, function calling usually provides better structural guarantees.

If you find that your extraction tasks are still too heavy, you might want to look into LLM Caching Strategies to Slash Latency and API Costs to avoid hitting the API for redundant requests entirely.

FAQ

Does function calling work with all models? Most modern providers (OpenAI, Anthropic, Gemini) support function calling, but the syntax varies. Stick to the SDK provided by your model vendor to ensure the best performance.

Is JSON schema validation still necessary? Yes. While function calling guarantees the structure is valid JSON, your application logic should still validate the values (like ensuring an email is actually an email) using a library like Zod or Pydantic.

How much can I realistically save? In our production environment, we saw a reduction of about 25-35% in input tokens. Your mileage will vary based on how verbose your original system prompts were.

I’m still experimenting with caching partial tool outputs to see if we can drive the costs down even further. For now, moving our extraction logic into schema-based tools has been the single most effective change we’ve made to our infrastructure costs this quarter.