AI Agent Observability: Metrics That Matter Beyond Latency

Last Tuesday a colleague paged me at 03:14: nine agents running a claims-triage flow, p99 latency clean, dashboards green, yet the escalation queue kept filling. The trace told the truth the SLO hid. AI agent observability, done right, catches that gap before the customer does.

Latency is a comfort metric. It survived from web-app monitoring because it was easy to plot. For multi-agent systems it hides the failures that matter: silent tool errors, semantic drift across handoffs, and cost that compounds per retry. The OpenTelemetry GenAI semantic conventions define spans for gen_ai.operation.name, gen_ai.tool.call.id, and token-usage attributes precisely because span duration alone is uncorrelated with task success.

The four signals that predict agent failure

Track these four before you touch p99 again:

1. Tool-call failure ratio per agent role. A retrieval agent failing at low single digits while a planner stays clean is a routing bug, not a model bug.
2. Handoff-to-thinking ratio. If more than half of wall-clock is inter-agent messaging, the topology is wrong — not the LLM.
3. Context-token drift. Watch input tokens per turn. A monotonic climb means your memory strategy leaks.
4. Cost per closed task, not per call. A cheap model that retries five times is more expensive than Opus that succeeds once — the same logic behind our note on cost-aware LLM routing.

2026-09-15T22:41:03Z trace_id=a3f2 agent=planner span=gen_ai.chat
  gen_ai.request.model=claude-sonnet-4.5
  gen_ai.usage.input_tokens=8412
  gen_ai.usage.output_tokens=311
  gen_ai.tool.call.count=3 tool.failures=1
  handoff.next=retriever cost_usd=0.041 task.status=partial

Instrumenting nine concurrent agents

Traces must join. If an orchestrator span cannot correlate to every child agent span through trace_id, you cannot compute the handoff ratio. The NIST AI Risk Management Framework categorises this under GOVERN-1.5 (traceability) — a requirement, not a suggestion, for systems that fall under the EU AI Act high-risk classification.

The failure surface grows non-linearly with agent count. Nine agents is not three times three agents — it is a graph where every edge is a place a semantic contract can break. ENISA’s Multilayer Framework for Good Cybersecurity Practices for AI recommends per-component telemetry because component-level metrics catch what aggregate SLOs bury. The same discipline underpins the failure taxonomy we track for agent systems.

What we run at CAI

We keep three dashboards per production agent graph: trace completeness, cost per closed task, and semantic-check pass rate. Latency is the fourth panel, not the first. If a customer stack reports green on latency alone, we treat it as unmonitored. Walk through how we wire OpenTelemetry GenAI conventions into a real orchestrator in our MetaRoute-Bench write-up.

Read further

• How agents choose between tools under load
• Six failure clusters every agent engineer should track
• Routing between Opus, Sonnet and open-source models by cost