Why I keep collapsing my multi-agent systems back into one

June 20269 min read

Multi-agent systems sound advanced. The bill they charge in production is paid in context loss, and that bill is usually larger than the problem they were supposed to solve.

The pattern I keep running into goes like this. Someone reads an article about agentic AI. They decide that the right architecture is one specialist agent per subtask — a planner, a SQL agent, an analyst, a critic. They wire them up. The demo works. Three weeks later, the system is failing in ways that are hard to debug: confident, fluent answers that are subtly wrong, and nobody can quite say why.

I have built several of these. I have also, several times, replaced them with a single agent and watched the failure rate drop. The reason is almost always the same, and it isn’t obvious until you’ve seen it twice.

The hidden cost is context loss

When you split a task across two agents, you also split the context. Agent A holds everything: the original question, the schema, the conversation history, the tool outputs, the half-formed reasoning. Agent A then produces some output and hands it to Agent B. But Agent B does not inherit Agent A’s context — it inherits only Agent A’s output. Everything else is gone.

The whole architecture is built on the assumption that Agent A’s output is a complete and faithful summary of what Agent B needs to know. In practice it almost never is.

Agent A reasons over a rich, messy context. Only its final output crosses to Agent B. Every filter choice, schema assumption, and edge case Agent A weighed disappears at the boundary — and Agent B must now interpret the output as if those choices never had to be made.

A concrete example

The user asks: “Why did patient enrollment drop in Q3 for the myasthenia gravis cohort?”

In the multi-agent setup, Agent A is the SQL specialist. It reads the question, looks at the schema, writes a query, runs it, returns a result. Agent B is the analyst. It takes Agent A’s table and writes a narrative explanation.

Agent A does what it’s supposed to do. It interprets “enrollment” as “currently active enrolled patients” — the conventional reading in this kind of dashboard — and adds WHERE is_active = TRUE to its query. The result table shows Q3 down 23%. Agent A is satisfied. Agent B receives the table.

Agent B doesn’t see the WHERE clause. It doesn’t see the schema. It doesn’t know that is_active exists as a field, or that Agent A chose to filter on it. It sees a table with a 23% drop and confidently writes: “Patient enrollment in the myasthenia gravis cohort dropped 23% in Q3, likely due to seasonal slowdowns in clinical recruitment.”

The answer is fluent. It is professional. It is wrong.

What actually happened: a batch of patients churned out of the program in late Q2 — not new enrollment dropping, but existing patients leaving. Raw enrollment was flat. The active count dropped because the denominator changed. An analyst looking at this directly would have caught the distinction in thirty seconds. Agent B couldn’t catch it, because Agent B never knew the filter existed.

Both architectures use the same model with the same tools. The difference is whether the query writer and the interpreter share context. When they do, the model catches the filter’s implication. When they don’t, the interpreter writes a fluent, confident, wrong answer.

The failure isn’t that the agents were too dumb. Both are the same underlying model. The failure is that the architecture put a context wall between query writing and interpretation, and the wall destroyed the information that would have caught the error.

The counter-intuitive bit: raw beats compressed

The defense people usually give for multi-agent setups is that long contexts confuse models, and that giving each agent a focused, tidy view of just its piece will make reasoning sharper. This was a defensible argument three years ago. It is no longer defensible.

Modern reasoning models — the Claude 4.x family, the GPT-5 series, recent Geminis — perform better when given the raw information to reason through than when given a pre-compressed summary. Their reasoning passes are explicitly designed to ingest, attend over, and re-derive from large unstructured contexts. Compressing the context before they see it is doing work that the model would do better on its own — and you are doing it without the model’s judgment about what to keep.

Compression strips the fine details that the model would have used to think correctly. The Q3 example above is exactly this: the WHERE is_active = TRUE clause is a fine detail, the kind a summarizer would naturally omit (“Agent A ran a query and got these counts”), and it’s also the only piece of information that lets the interpreter avoid the mistake. Compression and the mistake are the same act.

But context windows are finite

The honest version: prefer raw until you can’t. Long agent runs hit context limits even with million-token windows. Tool outputs balloon. Retrieved documents pile up. Conversation history accumulates. At some point the budget runs out and compression isn’t a choice anymore.

When you get there, the question is not “how do I summarize this?” It’s “how do I keep the parts the next step actually needs?” Different question. Different answer. The failure in the SQL example wasn’t that information was lost — it was that the wrong information was lost. The result table survived. The filter choice didn’t.

A few patterns from people running this in production:

Structured handoff, not narrative summary. Factory.ai ran an evaluation on 36,000 real engineering session messages and found that maintaining a persistent document with explicit sections — session intent, decisions, file modifications, pending work — and merging new content into it outperforms regenerating a summary from scratch. They call it anchored iterative summarization. Structure beats narrative because the downstream agent can query a structured document. It can’t easily query a paragraph.
Filter tool outputs at ingestion. Verbose tool outputs are the single biggest token sink in long sessions. Truncate or transform them as they come in, not after you’ve hit the limit. The Agent-Skills context-optimization playbook frames this as observation masking: replace verbose outputs with compact references once their purpose has been served, keep the original retrievable if needed downstream.
Decisions and failed approaches, not deliberations. “I filtered on is_active = TRUE because that’s the conventional reading” is one sentence. The 200 tokens of reasoning that produced that one sentence can be dropped. What you shouldn’t drop, and what most pipelines do drop, is the list of things that were tried and rejected. The Claude-Handover plugin makes this an explicit section in its HANDOVER.md template — without it, the next agent walks straight back into the same dead ends.
Pass tool inputs alongside outputs. If Agent A ran a query, the query text travels with the result. Otherwise Agent B is reading a table without knowing what was asked of it — which is exactly the Q3 enrollment failure above. XTrace calls the opposite mistake the context dump fallacy: assuming that more raw data always helps. It doesn’t. Unstructured volume just adds noise.
Keep the original question verbatim. The user’s exact wording is cheap to preserve and surprisingly hard to reconstruct from a paraphrase. LiveKit’s voice-agent handoff pattern is explicit about this: pass the caller’s stated goal as-is, alongside extracted intent and actions taken — not a clean summary of what you think they meant.
Compress for the consumer, not for tidiness. MindStudio’s framing of the agent handoff pattern is the right one: design the output for what the next step does with it, not for how it reads in a chat window. The downstream task is the spec. Generic summarization throws away the wrong things because it doesn’t know what matters next.

The corollary, restated honestly. If you must split, push the boundary as late as you can. Keep the contract as wide as your context budget allows. When the budget forces you to compress, structure the handoff for what comes next, not for what looks neat. The boundary between agents is where information goes to die unless you design against it.

When to actually split

I’m not arguing that multi-agent is never right. I’m arguing that the bar is much higher than “this feels like a multi-step task.” A useful test is the following decision tree, in roughly the order you should ask the questions.

Four questions to ask in order before splitting. Three of them have escape hatches back to a single agent, and the fourth requires a measurable benefit, not a vibe. Most architectures that should be multi-agent fail at least one of these checks; the rest were better off as one.

The questions matter in this order. Q1 kills most candidates immediately — if you have not actually tried building one capable agent with the right tools, you don’t know whether you need more than one. Q2 kills the next batch: if your subtasks share state, share context, or refer to each other’s outputs, splitting them creates exactly the information loss this whole article is about. Q3 is the technical filter: if you can’t write a precise spec for what crosses the boundary, you don’t have a multi-agent design, you have two agents and a prayer.

Q4 is the one most teams skip, and it’s the most important. “Multi-agent felt right” is not a measurable win. Latency, cost, accuracy, reliability — pick one and show that splitting actually improves it on real workloads before you accept the operational complexity that splitting brings.

The cases where splitting is right

There are real ones. They tend to share three properties: the subtasks are genuinely independent (no shared state, no cross-references); the contract between them is small and precise (a typed object, a fixed schema, an enum response — not natural-language narrative); and the win is real (parallel execution that’s 10x faster, or a specialized subagent that’s measurably more reliable on a narrow task than the generalist).

A retrieval-rerank-generate pipeline fits all three. A planner that emits a structured action list consumed by an executor fits. A fan-out across independent searches that get merged at the end fits. What doesn’t fit: anything where one agent’s reasoning informs the next agent’s interpretation, anything where the boundary carries judgment, anything where the downstream agent might second-guess the upstream agent if it had access to the upstream context.

That last category — where the downstream agent would reach a different conclusion if it could see the upstream context — is exactly where most multi-agent systems break. The downstream agent doesn’t know what it’s missing. It produces an answer with the same confidence it would have produced from full information. And confident wrong answers are the worst failure mode an AI system can have, because they look exactly like correct ones until someone with the full context happens to check.

The take

Build the simplest version first. Start with one agent that has the tools, the context, and the ability to reason across the whole problem. If it works, you’re done. If it doesn’t, find out specifically why it doesn’t before adding another agent — because every agent you add is a context wall, and context walls do not get cheaper as your system gets bigger.

The model is smart. Trust the model with the raw information. Most of the work you would have done splitting the system into specialists is work the model would have done better on its own.