Cost-Aware LLM Routing: Opus, Sonnet, or Open-Source in 2026
A single Claude Opus call for a customer-support classification burns roughly 15× the tokens of a Mistral 7B call producing the same label.
Cost-Aware LLM Routing: Opus, Sonnet, or Open-Source in 2026
A single Claude Opus call for a customer-support classification burns roughly 15× the tokens of a Mistral 7B call producing the same label. Teams that route every request to the flagship model pay for reasoning they never used. The routing question in 2026 is no longer “which model is best” — it’s “which model is enough for this specific request.”
The three-tier decision matrix
Route by task shape, not by intuition. We group live production traffic into three tiers:
- Tier A — flagship reasoning (Claude Opus, GPT-5, Gemini 2.5 Pro): multi-hop synthesis, legal drafting, adversarial red-teaming, code review with security implications.
- Tier B — workhorse (Claude Sonnet, GPT-5 mini, Gemini Flash): summarization over 20-100 pages, structured extraction, RAG answering with citations, tool-calling agents.
- Tier C — open-source local (Llama 3.3 70B, Mistral Small 3, Qwen 2.5): classification, entity extraction, boilerplate rewriting, embedding pre-processing, PII redaction where data cannot leave the VPC.
Rebuild the tier list quarterly — pricing shifts and new open-weight releases invalidate last quarter’s matrix within weeks. The EU AI Act general-purpose model regime further constrains Tier A choices: models trained above the 10^25 FLOPs threshold carry systemic-risk obligations that flow through to deployers under Art. 55. Read those deployer clauses before you standardize on a single frontier vendor.
What the router actually measures
A production router in our Iris orchestration stack samples every candidate model on a per-tenant eval set and stores three signals: task success rate, p95 latency, and cost per 1,000 completed tasks. The signals feed a dashboard the CFO reads on Monday and the platform lead reads on Tuesday. Anthropic publishes canonical pricing in the Claude API reference, and the NIST AI Risk Management Framework recommends documenting every substitution decision when a cheaper model replaces a certified one.
Sample router config from a live deployment:
router:
default_tier: B
escalation_rules:
- if: token_estimate > 40000
route: A
- if: contains_pii and jurisdiction == "EU"
route: C_local
- if: task_class in [classify, extract, redact]
route: C_local
fallback_chain: [sonnet-4.5, llama-3.3-70b, mistral-small-3]
budget_ceiling_eur_per_day: 180Escalation is one-way: Tier C to B on low confidence, B to A on repeated tool-call failure. The reverse — starting expensive and downgrading — wastes the flagship call you already paid for.
The position we hold
Cost-aware routing is not a cost story. It is a governance story. Every routing decision is a documented substitution under ENISA’s guidance on AI supply-chain cybersecurity, and every fallback to open-source is a data-residency choice under GDPR Art. 6(1)(f). We have watched teams cut inference spend meaningfully without measurable quality drop, once escalation rules move from ad-hoc chat threads into a written policy the auditor can read.
Our Demeter workload-economics practice treats the router as an auditable control plane, not a load balancer. Talk to us before you standardize on any single frontier vendor — see how Iris routes production traffic.