LLM Agent Failure Taxonomy: Six Clusters Engineers Must Track
An agent scores 92% on tool-use sub-tasks and 34% end-to-end. Why? A July 2026 synthesis of 27 papers across 19 benchmarks argues the gap is structural, not tuning-solvable (arXiv:2607.05775).
LLM Agent Failure Taxonomy: Six Clusters Engineers Must Track
An agent scores 92% on tool-use sub-tasks and 34% end-to-end. Why? A July 2026 synthesis of 27 papers across 19 benchmarks argues the gap is structural, not tuning-solvable (arXiv:2607.05775). Failures compound nonlinearly with task length, and stacking more scaffolding does not reliably improve reliability.
That last claim should stop every team building on ReAct-style loops.
Six failure clusters, not one
The synthesis proposes six clusters: tool invocation errors, planning and constraint-satisfaction failures, long-horizon degradation, multi-agent coordination failures, safety and security failures, and measurement validity problems. The first four sink pilots. The last two get pilots deployed anyway.
Measurement validity is the underrated one. Benchmarks reward completion signals that correlate weakly with production reliability — a point echoed by NIST’s AI Risk Management Framework, which asks operators to distinguish task-level accuracy from mission-level assurance. Benchmark contamination during pretraining inflates numbers that do not survive out-of-distribution traffic. For teams building on open-source model catalogs, this matters more, not less: shared benchmarks converge on shared failure modes and hide production gaps. Call it the leaderboard illusion.
2026-07-08T04:12:03Z eval.agent run_id=8f2e
step=1 tool=search status=ok latency_ms=812
step=2 tool=extract status=ok latency_ms=340
step=3 tool=schema_check status=ok latency_ms=90
step=4 tool=submit_form status=FAIL reason=constraint_violation
final_answer_match=true end_to_end_success=falseSub-tasks green, mission red. That is the reliability lie.
Why scaffolding stops helping
More tools, more retries, more reflection prompts — the paper finds these do not consistently improve end-to-end success. Nonlinear compounding is the reason. At 10 steps a 95% per-step success rate delivers 60% completion. At 25 steps it delivers 28%. Adding a critic agent that is itself 90% reliable multiplies error surfaces rather than dampening them. Seven-step planning tasks in AgentBench and GAIA already exhibit this pattern: sub-task green rates above 90% coexist with mission green rates below 40%.
The right response is not another retry loop. Shorten the horizon, commit smaller steps with hard checkpoints, and instrument the coordination boundary between agents. This matches the ENISA 2024 threat landscape guidance on cascading failures in orchestrated AI systems.
Our position on agentic AI safety at the topology layer treats the graph of agent handoffs as the primary risk surface, not the weights of any single model.
What we ship instead of leaderboard scores
For EU mid-market deployments under Directive (EU) 2022/2555 and the AI Act, leaderboard scores are not defensible evidence. The European Commission’s AI Act implementation guidance requires providers of high-risk systems to document post-market monitoring; a per-step audit log is the minimum viable answer. We build eval harnesses that record per-step tool status, constraint checks, and end-to-end mission success separately — then alert on the gap between them.
The CAI working position on production monitoring for autonomous agents treats sub-task/end-to-end divergence as a leading indicator, not a lagging metric. If your dashboards show only aggregate accuracy, you will hear about failures from customers first.
Bring us your agent’s eval logs. We will show you the six clusters inside them.