Smart Routing¶

The most expensive part of an agent is usually the LLM call. Frontier models can cost 30–100× more per token than small models, yet many systems send every request to the most capable model by default. Smart routing reverses that default: use the cheapest model that can handle the request, and escalate only when the request proves complex. It assumes a portfolio of models has already been chosen; for how to decide between API, self-host, and hybrid deployments, see Model Selection.

Kroki Kroki

Why routing is not a minor optimization¶

A simple lookup — "How much is a latte?" — does not need a frontier model. A multi-factor planning request — "Plan the team retro order for five people with allergies, a budget, and weather in mind" — might. When both go to the same expensive endpoint, the cheap request subsidizes the expensive one.

Berkeley's RouteLLM formalized this as a principled routing problem. On public benchmarks, learned routers cut costs by 35–85% while retaining roughly 95% of the strong model's quality. Anyscale's follow-up work shows similar results: a fine-tuned causal classifier routing between GPT-4 and an open-source model can reduce cost by 30–70% at a fixed quality floor.

As of 2026, the frontier-to-small price spread is roughly 20–50× per token [CalibreOS, 2026]. Empirical observations from FrugalGPT and RouteLLM suggest that 40–70% of production queries can be served by the cheap model with no detectable quality loss, 20–30% require frontier capability, and 10–20% fall into an ambiguous middle that benefits from cascade verification [CalibreOS, 2026]. Routing is now the highest-impact cost lever in LLM serving.

The cost-quality Pareto frontier¶

Routing is a constrained optimization: minimize cost subject to a quality floor. Without an explicit floor, "save money" silently becomes quality regression [CalibreOS, 2026]. Define the floor per intent category:

Intent	Example quality floor
Extraction	≥ 92% schema accuracy
Reasoning	≥ 88% correct on verified answers
Summarization	≥ 4.0 / 5 on judge eval
Chat	Human-equivalent response rate ≥ threshold

The Pareto frontier is the set of (cost, quality) points no model dominates. A router picks the point on that frontier that satisfies the floor with the lowest expected cost.

A three-step router¶

A production router is rarely a single model call. It is a small pipeline:

Input-type filter. Reject or short-circuit by modality. If the input is an image, only models with vision capability are candidates. If it is a banned topic, route straight to a guardrail response.
Classifier. A small model or fine-tuned classifier reads the request and predicts the cheapest adequate target model. The classifier is the only non-deterministic stage; its prompt contains concise capability descriptions of every candidate model.
Fallback rules. If the classifier is uncertain, slow, or returns an invalid choice, deterministic rules take over: default to a safe model, or route by keyword/regex.

Stage	Deterministic?	Cost	Purpose
Input-type filter	Yes	Negligible	Eliminate impossible candidates early
Classifier	No	Cheap LLM	Choose the cheapest capable model
Fallback rules	Yes	Negligible	Guarantee a valid route

Model descriptions matter¶

The classifier's accuracy depends on the quality of the candidate descriptions it is given. A good description does not list every benchmark; it states what the model is for in your domain:

Small chat model: fast, cheap, good for factual lookups, short answers, and structured extraction from clear context.
Coding model: strong at reasoning over code, diffs, and architecture decisions; slower and more expensive.
Vision model: required for image, diagram, or PDF understanding.

When you add fine-tuned or domain-specialized models, the router can also increase quality by matching the right specialist to the right request.

Cascades and confidence gating¶

A cascade tries the cheap model first and escalates only when a verifier is unsure [CalibreOS, 2026]. This avoids classifier latency at the cost of occasional extra upstream calls. Cascades work best when there is a cheap, reliable verification signal:

Code generation: unit tests.
Math: answer checkers.
Extraction: schema validators.
Open-ended chat: expensive judge or reward model; often not worth routing unless volume is huge.

Confidence gating means the classifier's probability should be calibrated: a score of 0.9 should mean roughly 10% error on that route [CalibreOS, 2026]. Uncalibrated classifiers produce either over-confident misroutes or under-confident waste. Shadow traffic is the safe way to recalibrate: send 1–5% of queries to both the routed and gold-route models and compare outcomes.

The hidden cost: KV-cache switching¶

Routing gains can be erased by cache mechanics. Most providers cache the prefix of a long conversation; switching the model in the middle of a session often breaks that cache. The cheaper model may now cost more than continuing the cached expensive model.

The rule is: measure the full cost, not just the per-call price. Track:

Cost per routed request by model.
KV-cache hit rate per session.
Cost delta caused by switching mid-dialog.
Prompt-cache hit rate and effective discounted price.

Routing without these metrics is hope, not engineering.

Failure modes¶

Failure mode	Symptom	Mitigation
Router drift	Quality gap grows over time	Shadow traffic, periodic retraining on recent traffic
Uncalibrated confidence	Classifier score does not match error rate	Platt scaling or isotonic regression on held-out set
Cascade tail latency	Worst-case latency is sum of all hops	Set a per-hop timeout and fallback to frontier
Cache-breaking switch	Mid-session model switch costs more than saved	Cache-aware routing, full-session cost accounting
Training-serving skew	Router trained on old model pool or old traffic	Regenerate routing dataset when models or traffic shift
Overly aggressive cost saving	Frequent escalation or user-visible quality drops	Raise quality floor, retrain with class weights

Observability¶

Every routing decision should be observable:

Which model was chosen and why.
Confidence score from the classifier.
Latency and cost of the chosen route.
Whether the fallback rule triggered.
Routed-vs-gold quality gap on shadow traffic.

This data feeds back into the classifier: retrain or adjust descriptions when the router consistently misroutes a class of requests.

Design principles¶

Fail cheap, then escalate. Start with the cheapest viable model; only re-route if quality checks fail.
Cache-aware routing. Do not switch models mid-session unless the saved tokens outweigh the lost cache.
Observable decisions. Record choice, confidence, cost, and outcome for every request.
Calibrated confidence. A classifier score is a probability, not a rank.
Explicit quality floor. Constrain cost minimization by intent category.
Shadow traffic for changes. Router changes are silent quality changes; verify them before full rollout [CalibreOS, 2026].

Deeper dive: routing requests to the right model¶

As of 2026, the dominant design pattern is to treat the router as a cost-quality gate rather than a single classifier. Production teams report 40–85% bill reductions after moving from a one-size-fits-all frontier endpoint to a routed portfolio, because the capability spread between the cheapest usable model and a frontier model has grown to roughly 100× [Digital Applied, 2026]. The decision is not only about which model is smarter; it is about which model is smart enough for this specific request. Most enterprise traffic falls into commodity categories — intent classification, entity extraction, structured JSON, FAQ lookup, template summarization — that do not need frontier reasoning [LeanLM, 2026]. Routing makes the price paid track the actual difficulty of the task.

Three architectural patterns have emerged, each with different data and latency requirements. Confidence-based cascading tries the cheapest model first and escalates only if a verifier or confidence score is below threshold; it needs no labeled data and captures savings immediately. Pre-inference classification trains a lightweight model to predict the right target before any LLM call; it is cheaper per request than a cascade but requires representative labeled examples. Embedding-based routing matches a query to historical queries with known best-model assignments; it improves as production data accumulates but needs a fallback for cold-start shapes [LeanLM, 2026]. In latency terms, rule-based routing adds under 1 ms, embedding lookup about 5 ms, and learned classifiers 50–100 ms — all small against typical inference latencies of 500–2,000 ms [Digital Applied, 2026].

The real production risk is silent quality regression, not router overhead. Cheaper models can produce plausible but subtly degraded answers that surface as customer tickets days later, so a pre-merge eval gate of 50–500 representative cases is the safeguard that earns the savings safely [Digital Applied, 2026]. Mature routers also fold in policy constraints — per-customer budgets, latency SLAs, data-residency rules, and customer tier ceilings [LeanLM, 2026]. Policy sets the envelope; the difficulty classifier decides where inside that envelope each query lands.

Approach	When to use	Latency / cost note
Cascade	No labeled data, mixed query stream	May call a second model; easiest to deploy
Pre-inference classifier	High volume, stable task mix	Single extra call; needs labeled examples
Embedding-based	Diverse queries with rich history	~5 ms lookup; needs fallback for cold starts

References¶

CalibreOS. "LLM Router and Model Cascade: Cost-Aware Query Routing at Production Scale." 2026. https://www.calibreos.com/learn/genai-llm-router
Chen, B., Zaharia, M., Zou, J. "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance." 2023. https://arxiv.org/abs/2305.05176
Digital Applied. "LLM Model Routing in 2026: Cost-Quality Optimization." 2026. https://www.digitalapplied.com/blog/llm-model-routing-2026-cost-quality-optimization-engineering-guide
LeanLM. "LLM Model Routing: Cheapest Capable Model Per Query." 2026. https://leanlm.ai/blog/llm-model-routing
Ong, I., et al. "RouteLLM: Learning to Route LLMs with Preference Data." UC Berkeley / Anyscale / Canva. https://sky.cs.berkeley.edu/project/routellm/
Anyscale. "LLM Router: Train and Deploy State-of-the-Art LLM Routers." https://github.com/anyscale/llm-router
Xue, R., et al. "R2-Router: A New Paradigm for LLM Routing with Reasoning." arXiv:2602.02823, 2026. https://arxiv.org/abs/2602.02823
Li, Z., et al. "LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing." arXiv:2601.07206, 2026. https://arxiv.org/abs/2601.07206