Observability for AI Agents: Metrics Beyond Latency
Nine agents ran concurrently against the same task queue last Thursday. Latency looked green across the board.
Observability for AI Agents: Metrics Beyond Latency
Nine agents ran concurrently against the same task queue last Thursday. Latency looked green across the board. Two agents were silently retrying the same tool call in a loop, burning $340 in tokens before a human noticed. The dashboard lied because it measured the wrong thing.
Observability for AI agents is not APM with a chatbot skin. When you orchestrate 9 concurrent agents sharing tools, memory, and a budget, latency is table stakes — the signals that catch real failures live one layer deeper.
The four metric families that matter
Traditional OpenTelemetry semantic conventions for GenAI give you spans and token counts. Useful, insufficient. A production agent fleet needs four families instrumented from day one:
Trajectory metrics. Steps per task, tool-call depth, replan count, loop detection (same tool + same args within N steps). The NIST AI Risk Management Framework treats trajectory divergence as a first-class risk signal — most teams still don’t emit it.
Tool-call health. Success rate per tool, argument-schema validation failures, retries per tool, and mean cost per successful tool invocation. If one MCP server has a 12% argument-validation failure rate, that’s a broken prompt contract, not a network issue.
Budget metrics. Tokens per task, dollars per task, tokens per useful step (steps that changed state vs. steps that reflected). Concurrency amplifies waste: 9 agents each doing one extra reasoning loop is 9× the bleed. Cost per resolved ticket beats cost per API call every time.
Contention metrics. Queue depth per shared resource (RAG index, vector store, code executor), lock wait time, and cross-agent conflict rate when two agents write to the same memory key. This is where multi-agent systems die quietly.
What a real trace looks like
2026-09-01T14:22:07Z agent=research-3 trace=8f2a step=4
tool=web_search status=ok tokens_in=1842 tokens_out=203
cost_usd=0.0231 loop_score=0.87 budget_remaining=0.61
2026-09-01T14:22:09Z agent=research-3 trace=8f2a step=5
tool=web_search status=ok args_hash=SAME_AS_STEP_4
loop_detected=true action=halt_and_replanloop_score is a rolling cosine similarity across the last N tool-argument vectors. args_hash=SAME_AS_STEP_4 triggers a hard halt. Both signals live in the trace, not in a separate log stream — you correlate them with a single trace ID.
Wiring it into the fleet
The kill switch is the point. Observability that only surfaces problems after the budget is spent is a postmortem tool, not an operational one. ENISA’s 2024 threat landscape for AI frames runaway agent behavior as an availability threat — treat it that way.
Where we’ve landed
At CAI Technology we run our Iris agent orchestration layer with all four metric families emitted per step, correlated by trace ID, and gated on hard budget ceilings. The interesting work is not building more dashboards — it’s picking the three signals that let a human trust the fleet at 3 a.m. See how we structure multi-agent reliability in Demeter for the operational pattern.
If your fleet grew from 2 to 9 agents this quarter and your dashboard didn’t change, that’s the conversation worth having.