Cost-Aware LLM Routing: A 2026 Decision Matrix
At current list prices, running one million reasoning tokens on Claude Opus 4 costs roughly 40× what the same job costs on Claude Haiku, and about 15× a hosted Llama 3.3 70B endpoint (Anthropic pricing, Together AI pr…
Cost-Aware LLM Routing: A 2026 Decision Matrix
At current list prices, running one million reasoning tokens on Claude Opus 4 costs roughly 40× what the same job costs on Claude Haiku, and about 15× a hosted Llama 3.3 70B endpoint (Anthropic pricing, Together AI pricing). Yet most production teams still send every request to the biggest model they trust. That is a procurement problem dressed up as an engineering one.
The Three-Tier Split
We route by the shape of the task, not by the seniority of the caller. Three tiers cover the vast majority of enterprise workloads:
Tier 1 — Frontier reasoning (Opus 4, GPT-5, Gemini 2.5 Ultra). Multi-step legal analysis, code refactors touching more than five files, novel synthesis with no retrieval anchor. High cost, high latency, but the only tier where accuracy on out-of-distribution problems justifies the invoice.
Tier 2 — Balanced workhorses (Sonnet 4.5, GPT-4.1, Gemini 2.5 Pro). RAG answering, structured extraction, code review on scoped diffs, tool-calling agents with three to seven hops. This is where the bulk of your tokens should live.
Tier 3 — Open-source and small proprietary (Llama 3.3 70B, Qwen 2.5, Haiku, Mistral Small). Classification, summarisation of clean input, embeddings, translation, PII redaction, cache-warmers. Self-hosted where data residency matters — the EU AI Act’s Article 10 data-governance obligations bite harder when logs cross borders (EUR-Lex, Regulation (EU) 2024/1689).
The Decision Matrix
The router itself is boring on purpose. A rules engine plus a small classifier beats a learned meta-model in every audit we have run — see our notes on why deterministic routing wins in agentic systems and how per-tenant accounting keeps the bill honest in governed RAG deployments.
A Router Config That Survives Contact With Production
router:
default_tier: 2
escalation_rules:
- name: legal_reasoning
match: {domain: [contracts, litigation], min_chain: 3}
route: tier1
max_cost_eur: 0.85
- name: pii_redaction
match: {task: redact, sensitivity: high}
route: tier3_local # self-hosted Llama, no egress
- name: rag_answer
match: {retrieval_hits: ">=2"}
route: tier2
fallback: {on_error: tier1, budget_cap_eur_day: 400}Two guardrails matter more than the model choice: a hard daily budget cap per tenant, and a confidence probe that re-routes low-score Tier-3 outputs to Tier 2 before the user sees them. NIST’s Generative AI Profile calls this out under MG-2.2 for good reason (NIST AI 600-1).
Our position at CAI Technology: routing is a governance surface, not a savings hack. The team that owns the routing table owns the model risk register — and that is the same team that will answer to auditors under the ENISA AI threat landscape guidance (ENISA, 2023). Start there. Walk through the reference implementation on our iris engineering pillar when you are ready.