LLM Agents Conflict Resolution: Merging Divergent Workflow Outputs

Last month, I spent about three days debugging a pipeline where two agents were supposed to extract project milestones from a transcript. One agent insisted the deadline was "next Friday," while the other pegged it to a specific ISO date. My system didn't know which one to trust, and the downstream database entry ended up completely corrupted.

When you scale to multi-agent systems, you quickly realize that divergence is the default state, not an edge case. If you're building systems where different LLM agents handle specific sub-tasks, you need a strategy for conflict resolution that doesn't involve just picking the first output.

The Failure of Naive Merging

My first attempt at a fix was simple: I picked the output from the agent with the higher "confidence score" (which, let's be honest, is just a hallucinated probability value). That failed immediately because agent A was confident about the wrong date, and agent B was confident about a different, also wrong, date.

I learned the hard way that you can't just merge JSON blobs. You need a semantic layer that understands the context of the disagreement. If you're currently struggling with messy data, you should revisit how you handle structured output: implementing deterministic json schema validation to ensure your agents are at least speaking the same language before you try to resolve their differences.

Designing a Conflict Resolution Layer

To solve this, I moved to a three-tier architecture:

1. Extraction Tier: Agents output raw data using strict schemas.
2. Validation Tier: A deterministic check ensures the outputs are schema-compliant.
3. Arbitration Tier: A "Judge" agent (or a simple code-based heuristic) reconciles semantic gaps.

If you don't have a solid foundation for your data, your arbitration will be garbage-in, garbage-out. I highly recommend mastering structured output with pydantic: a guide to reliable llm parsing to ensure your agents aren't just throwing strings at each other.

Implementing the Judge Pattern

When two agents disagree, don't ask them to "fix it." Instead, provide the Judge agent with the context, the two conflicting outputs, and the specific goal.

PYTHON
# A simple example of an arbitration prompt
system_prompt = CE9178">"""
You are a senior project manager. You are presented with two conflicting 
milestone extractions. Your task is to select the most accurate date 
based on the provided transcript. If neither is correct, return CE9178">'NULL'.
"""

# The Judge receives the conflicting data as structured input
# and returns a Pydantic model with the resolved value.

By forcing the Judge to justify its choice, you gain a massive advantage in debugging. It’s similar to how we implement llm agents: implementing reflection patterns for better reasoning to let agents critique their own work before it hits production.

When Code Beats AI

Sometimes, you don't need an LLM to resolve a conflict. If Agent A says {"date": "2023-12-01"} and Agent B says {"date": "2023-12-05"}, don't prompt the LLM to "decide." Use a simple heuristic: take the most recent date mentioned in the source document, or use a library like dateutil to check which date falls on a valid business day.

I've found that about 70% of "semantic" conflicts are actually just formatting inconsistencies that can be solved with standard Python logic. Only escalate to an LLM-based arbitrator when the conflict is truly ambiguous, like when two agents interpret a vague phrase like "the end of the quarter" differently.

Why Multi-Agent Systems Need Evals

If you aren't running evals on your arbitration logic, you're flying blind. I've started building small test suites that inject synthetic conflicts into my pipeline to see how the Judge handles them. It’s a bit like the logic in llm evaluation strategies: building multi-model verification systems, where you compare consensus against a ground truth.

Final Thoughts

The biggest trap is trying to build a "perfect" resolver. You will never eliminate all errors. Instead, focus on observability. When an arbitration fails, make sure you're logging the raw outputs of all involved agents alongside the Judge's final decision.

I’m still experimenting with whether it’s better to have a single monolithic "Judge" or a decentralized "Peer-Review" pattern where agents audit each other. For now, the centralized Judge is easier to maintain, but it adds roughly 400ms of latency per resolution. It’s a trade-off I’m willing to make for data integrity, but I’m keeping a close eye on those response times.

---

Frequently Asked Questions

Q: Should I use a separate LLM model for the Judge agent? A: Usually, yes. Use a smaller, faster, and cheaper model for the extraction agents, and a more capable model (like GPT-4o or Claude 3.5 Sonnet) for the Judge. The Judge needs better reasoning capabilities to handle the conflict.

Q: What if the Judge agent also hallucinates? A: This is the danger of "turtles all the way down." Always include a "None of the above" or "Manual Review Required" option in your arbitration schema. If the Judge can't reach a high-confidence conclusion, flag it for human intervention.

Q: How do I handle conflicts in real-time? A: If latency is critical, move your conflict resolution to an asynchronous background task. Return a "Pending" state to the user while your arbitration logic works, then push an update to the UI via WebSockets once the conflict is resolved.