Model Selection¶

Choosing a model for an agentic system is no longer a one-time decision between a few API providers. The landscape has split into two large ecosystems — closed frontier APIs and open-source weights — and most production agents end up using several models at once: a frontier model for complex generation, a small model for routing, embedding and reranking models for RAG, vision models for images, and a fallback model for resilience. The right answer is rarely "one model for everything." It is a system for choosing, routing, and replacing models as the landscape changes.

The two landscapes¶

On one side are closed frontier providers — Anthropic, OpenAI, Google, and others — each running five or six tiers: Opus, Sonnet, Haiku, GPT-4o, o3, Gemini Pro, Flash, Flash-Lite, and so on. Releases happen weekly; the major providers shipped more than 250 releases in Q1 2026 alone. On the other side is open source: over two million models on Hugging Face, though the count includes fine-tunes, embeddings, and duplicates. The foundation models that matter come from Meta (Llama), Alibaba (Qwen), Mistral, DeepSeek, Microsoft (Phi), Google (Gemma), and active Chinese labs such as Moonshot, Zhipu, and MiniMax, with new releases every few weeks.

A production agent is usually a stack, not a single call:

Role	Typical model type
Primary generation	Frontier API or strong open-weight model
Routing / classification	Small open model, 2–7 B parameters
Embeddings	Lightweight embedding model
Reranking	Cross-encoder reranker
Vision	Multimodal model or dedicated vision API
Fallback	Secondary API provider or local model
Security / guardrails	Small classifier or rule layer

Each of these choices must be validated on your data, not on a leaderboard. And every choice is layered with economics: token prices fall, but agents consume one or two orders of magnitude more tokens than a simple chat, so bills often grow faster than expected.

Five families that matter in 2026¶

For enterprise shortlists, the 2026 landscape condenses to five families [Wolyra, 2026]:

Anthropic Claude 4.7 — strongest agent and tool-use behavior, a clear constitutional AI safety story for regulated industries, and a one-million-token context window that makes long-document reasoning practical.
OpenAI GPT-5 — broadest ecosystem, the most mature fine-tuning and assistants tooling, and the deepest integration into Microsoft enterprise stacks.
Google Gemini 2.5 Pro — native multimodal leadership on video and large-PDF reasoning, plus the strongest data-residency story for organizations already on Google Cloud.
DeepSeek-V3 — closed the frontier gap on reasoning benchmarks at a fraction of the cost and a credible self-hosted option when the security posture allows.
Qwen-3 — the strongest non-English-first model family and the right choice for enterprises with substantial Chinese, Japanese, or Korean operations.

Meta Llama remains a serious option for fine-tuning and on-premises deployment but has lost ground at the frontier. Smaller specialty vendors — Mistral, Cohere, AI21 — are still relevant for data residency in Europe, retrieval-tuned models, and smaller-context efficient deployments [Wolyra, 2026].

Evaluation axes that survive contact with reality¶

Public benchmarks are nearly useless for final enterprise selection in 2026; frontier models cluster within a few percentage points, and your specific workload will reorder them. The practical axes are [Wolyra, 2026]:

Capability on your workload — build an internal eval set of 200–1,000 real production examples with human-graded ground truth, track pass rate, latency, and cost per example.
Latency under realistic load — measure p50, p95, and p99 at expected concurrent request rate, not single-request vendor quotes. Time-to-first-token matters for streaming UIs; total response time matters for batch and agent workflows.
Cost per resolved task — the blended cost including retries, not the list price per million tokens. A cheaper model that takes three attempts to succeed is more expensive than a premium model that succeeds on the first call.
Data residency and processing geography — get the contractual data flow in writing, including zero-retention guarantees, training-data policies, and incident-disclosure obligations. Cloud platforms (AWS Bedrock, Azure OpenAI Service, Google Vertex AI) often have explicit regional guarantees; direct API access from the model vendor often does not.
Fine-tuning and customization support — decide the customization strategy first. Not every vendor offers full-parameter fine-tuning; some only offer LoRA-style adapters.
Compliance and contract terms — SOC 2 Type II is table-stakes; HIPAA requires a BAA; ISO 27001, FedRAMP, and PCI DSS matter by vertical. Look for IP indemnification and clear breach-notification clauses.

Decision tree: API, self-host, or hybrid?¶

The first question is not "which model?" but "who runs it?" The cleanest mental model is a decision tree.

Kroki

1. Data sensitivity and residency¶

If the product handles PII, financial, medical, or government data, most jurisdictions restrict where that data can be processed. The first check is not "must we self-host?" but "can the provider deploy in our region under a data-protection agreement?" Many frontier providers offer regional endpoints and DPAs. If that satisfies the requirement, self-hosting is unnecessary. If no compliant provider exists in the region, self-hosting becomes mandatory.

2. Scale and unit economics¶

There is no universal break-even point. It depends on model size, sequence length, traffic pattern, and whether the team can operate GPU infrastructure. The practical calculation:

Estimate the monthly API bill at current load.
Estimate GPU or managed-inference cost plus DevOps time for self-host.
Re-run both numbers at 3× load.

If the difference is small at current scale, API is usually the rational default. If the API bill is already painful and keeps growing, self-hosting or hybrid deserves a serious spreadsheet.

3. Fallback and routing¶

Two needs often push a system toward self-host or hybrid:

Fallback: if the primary provider is down, a second provider or a local model can keep the agent alive. A fallback does not have to be local, but if the reliability requirement demands a reserve outside cloud APIs, self-hosting is a strong argument.
Cost-aware routing: if traffic is mixed and most requests are simple, running the cheap ones on a local small model while reserving frontier APIs for hard requests cuts spend.

Decision summary¶

Question	If yes	If no
Can a compliant provider host in your region?	Use API	Self-host
Is the API bill small and scale uncertain?	Use API	Calculate further
Does 3× load make self-host cheaper?	Consider self-host	Stay on API
Is a second-level fallback required?	Add self-host or second API	Single API may suffice
Is mixed traffic with many simple requests?	Add a router and small model	Send everything to one model

Procurement red flags¶

The following items in a vendor proposal should slow the process and trigger additional review [Wolyra, 2026]:

No commitment to model version stability or a deprecation notice.
Indemnification limited to direct damages with low caps.
Mandatory training-data clauses requiring enterprise data to be usable for model improvement.
SLA targets below 99.5 percent for production endpoints, or no SLA on the specific models you intend to use.
Incident notification obligations longer than 72 hours for security incidents, or no obligation to notify on model behavior changes.
Pricing structures that incentivize the vendor to maximize tokens consumed (per-token-only with no efficiency credits, no committed-use discounts, no rate-limit transparency).

The outcome is usually one of three patterns:

API-first: early products, modest or bursty traffic, fast iteration, no strict boundary constraints.
Hybrid: mixed sensitivity, cost-aware routing, fallback across providers and internal models, gradual migration.
Self-host: regulated data, steady high volume, strong need for runtime control, mature platform team.

The mistake is to treat self-hosting as a prestige move. It is an operating commitment that pays off only when a hard constraint or a clear economic advantage forces the choice.

Economics: open source is not free¶

Open-weight models are often called "free" because there is no per-token bill. The real cost structure has four parts:

API dollars: usage-based spend with frontier or managed providers.
Infrastructure dollars: GPUs, reserved capacity, spot instances, networking, storage.
DevOps dollars: stack setup, upgrades, debugging, on-call incident response.
Complexity dollars: engineering time spent understanding failure modes, versioning, and performance tuning.

On small volumes the sum of the last three often exceeds any API savings.

GPU purchasing models¶

GPU infrastructure is now sold like taxi rides:

On-demand / per-minute: pay for actual use; good for experiments and bursty workloads.
Spot: discounted, interruptible instances; fine for non-critical inference or batch jobs with fallback, but not for latency-sensitive production routes.
Reserved / annual contract: discounted if load is predictable and steady.
Serverless managed inference: a catalog model exposed as an API endpoint. No hardware or driver work, predictable pricing, but less control and another vendor dependency. Price usually sits between self-host and frontier API.
Consumer cards (RTX 3090 / 4090): much cheaper than H100 per card and can run medium models with quantization. The hidden cost is multi-GPU scaling.

The real comparison¶

Do not compare "GPU cost vs API cost." Compare:

monthly API spend,
monthly infrastructure spend,
monthly DevOps hours at your team's loaded cost,
the operational complexity of each path.

A $250 GPU can easily turn into several thousand dollars once human hours are counted. The crossover point is unique to each team, workload, and engineering capability.

Why public benchmarks fail on your domain¶

Public benchmarks — MMLU, HumanEval, GSM8K, SWE-bench — are useful for one thing: filtering out obviously weak models. They are poor tools for choosing the model that will perform best on your product.

The reasons are structural:

Different distribution: benchmarks measure general capability on clean, standardized tasks. Production data is messy, domain-specific, and shaped by your users.
Domain and language: a model may claim multilingual support but perform much worse on your target language. Chinese models sometimes inject ideographs into Russian or English outputs even when labeled multilingual.
Tool descriptions: your tool schemas, edge cases, and failure modes do not appear in public tests. A high-scoring model can misunderstand your function calls; a mid-tier model can outperform it because its training data was closer to your domain.
Benchmark contamination: training corpora often include benchmark questions, inflating scores beyond genuine capability. Perturbing the wording of benchmark-style questions can reveal memorization rather than reasoning.
Ceiling effects: when three models all score 89–92% on MMLU, the difference is inside the noise and tells you nothing about your workload.

How to evaluate on your own data¶

Build a short evaluation set from real traffic. You do not need thousands of cases at the start; a few dozen covering the main traffic patterns plus a handful of hard edge cases is enough.

Run candidates through three layers:

Capability probes: can the model do the task on your documents, in your language, with your terminology?
Robustness tests: paraphrase stability, adversarial inputs, prompt-injection attempts, context poisoning, empty documents, multi-language queries.
Production simulation: replay anonymized traffic through the full pipeline — retrieval, reranking, generation, guardrails — at expected concurrency. Measure latency percentiles, error rates, and resource use, not just output quality.

A fallback model must be tested the same way. A model that works as the primary generator may fail when used as a fallback, because the fallback is called precisely when the primary path is under stress.

The rule applies equally to GPT, Gemini, Claude, Qwen, DeepSeek, and Llama: the same tool description can behave very differently across families.

Hybrid architecture: router and fallback¶

At scale, a single model is rarely optimal. Different tasks have different zones of strength. A hybrid design routes each request to the cheapest model that can handle it, with a fallback path when the primary is unavailable.

The request flow:

Router: a small classifier — often 2–7 B parameters — categorizes the request in milliseconds. It does not write the answer; it decides whether the request is sales, accounting, simple, or complex.
Branch selection: the request goes to the matched model or model group.
Fallback branch: if the primary provider returns an error or times out, the request is retried on the fallback model.

A router is calibrated to minimize expensive errors, not to maximize raw accuracy:

A simple request sent to a frontier model wastes money.
A complex request sent to a small model produces a bad answer and can lose the user.

The second error is more expensive. A good router is therefore biased toward escalation: when uncertain, send the request to the stronger model. Cost savings come from confidently classifying the large volume of truly simple requests.

Router placement and operations¶

The router needs the same operational discipline as the models it controls:

tested on real traffic,
observed for choice, confidence, latency, and cost,
maintained as the traffic distribution shifts,
deployed either self-hosted or through a managed inference provider, depending on team size and reliability needs.

Routing is not failover. Failover is about availability; routing is about cost and quality optimization. They share machinery — a candidate list and a policy — but conflating the two causes incidents.

For the internal mechanics of a production router — classifier pipeline, model descriptions, KV-cache awareness, and observability — see Smart Routing.

The NVIDIA consumer-GPU P2P trap¶

If you plan to self-host on consumer cards such as the RTX 3090 or 4090, one hardware detail changes the economics: NVIDIA disables peer-to-peer (P2P) direct GPU-to-GPU communication on GeForce cards in the driver, even though the hardware can support it.

When a model does not fit on one card and is split across several with tensor parallelism, the GPUs constantly exchange data. Without P2P, that traffic goes through the CPU and system memory, adding latency and consuming PCIe bandwidth. The result is often 2–3× slower real-world throughput than a naive calculation based on card count would suggest.

Community patches and modified drivers can re-enable P2P, but running patched kernel drivers in production introduces security, update, and stability risks. For dev/test, experimentation, or non-real-time batch work, consumer multi-GPU is fine. For serious production latency targets, budget for datacenter-class GPUs with working P2P, or accept the real performance penalty in the calculation.

When hybrid is not needed¶

Hybrid architecture is not a universal upgrade. If the workload is uniform and small — for example, 1,000 complex requests per day — send everything to a frontier API. Adding a router, a local model, and fallback logic adds cost and failure modes without meaningful savings.

Hybrid starts to make sense when:

traffic is mixed: many simple requests and a smaller share of complex ones,
paying frontier prices for every request is no longer justified,
the system needs fallback independence from a single provider,
the team can operate the additional infrastructure.

Multi-vendor strategy¶

For any enterprise spending more than about $2 million per year on inference, a primary-secondary architecture is the safer default [Wolyra, 2026]:

Primary vendor handles 70–80% of traffic with deep integration and full feature use.
Secondary vendor handles the remainder through the same abstraction layer.

The cost premium of the second integration is usually under 10% of the inference budget; the negotiating leverage and failure resilience often exceed that. Avoid a thin, lowest-common-denominator wrapper, because it hides each vendor's distinguishing features. The better pattern is a capability-based router: each route declares what it needs (tool use, structured output, long context, low latency, low cost) and the router selects the best vendor for that capability mix [Wolyra, 2026]. Revisit the decision every 12 months; the 2026 model landscape changes quarterly.

Business-oriented metrics for model evaluation¶

Public benchmarks saturate quickly; by 2026 frontier models cluster above 90% on MMLU and HumanEval and show documented training-set contamination. Stabilarity's BOME (Business-Oriented Model Evaluation) framework replaces leaderboard chasing with four operational metrics that map directly to enterprise outcomes [Stabilarity Hub, 2026]:

Metric	Formula	What it captures
Cost-per-Correct-Output (CpCO)	`(Input tokens × input price + Output tokens × output price) ÷ Task accuracy rate`	The full cost to produce one verified correct answer, including retries
Latency-Adjusted Accuracy (LAA)	`Accuracy × (Latency threshold ÷ max(Actual latency, Latency threshold))`	Quality discounted when it comes at unacceptable latency
Context-Window Utilisation Efficiency (CWUE)	`Accuracy at 80% context fill ÷ Accuracy at 20% context fill`	Lost-in-the-middle degradation on long documents
Failure Recovery Cost (FRC)	`(1 − Accuracy) × (Retry cost + Human review cost + Downstream correction cost)`	Total remediation cost when the model is wrong

A model with 95% accuracy at $10/M output tokens can have a lower CpCO than a model with 80% accuracy at $5/M output tokens once retry and review costs are included. For high-stakes domains such as financial reporting or compliance, FRC often dominates total cost of ownership even when per-token prices are low [Stabilarity Hub, 2026].

Real self-hosting economics: break-even and anti-patterns¶

Open-weight models are not free. The headline GPU bill is typically only 60–75% of true total cost of ownership; the remaining 25–40% is engineering time, eval suites, cold-start replicas, multi-AZ standby, egress, and storage [Ghalme, 2026].

AWS instance (Apr 2026, us-east-1)	GPUs	Memory	1-yr reserved $/hr	Best for
g6.4xlarge	1× L4	24 GB	$0.79	Qwen 2.5 Coder 32B in fp8
g6.12xlarge	4× L4	96 GB	$2.75	Llama 3.3 70B in fp8
p4d.24xlarge	8× A100	320 GB	$19.20	200B models, multi-GPU TP
p5.48xlarge	8× H100	640 GB	$35.10	DeepSeek V3 fp8, frontier inference

Self-hosting beats the API only when three conditions are simultaneously true: sustained monthly token volume above 100–500 million, 60–80% GPU utilization, and at least half an FTE with GPU-ops and vLLM experience. At 20% utilization a g6.12xlarge running Llama 3.3 70B costs ~$15.94 per million output tokens, which saves nothing against a Sonnet API. At 80% utilization the same hardware drops to ~$3.99/M — a 3× saving [Ghalme, 2026].

Anti-pattern	Why it costs more	Fix
Idle GPUs at 5% utilization	Fixed hourly cost swamps per-token savings	Size the fleet to average+peak or use spot + API fallback
One replica in one AZ	First GPU fault is an outage	Run N+1 with multi-AZ warm standby
No eval suite on every quant/engine change	Silent 3% quality regression ships	Automate evals before any model/engine/quant change
Over-quantizing to int4	Saves memory but tanks reasoning quality	fp8 is the safe default; skip int4 without eval
No API fallback	GPU OOM or driver crash takes the feature down	Configure gateway-level fallback to a hosted API

The engine choice also matters: vLLM remains the default for production (PagedAttention, continuous batching, prefix cache, broad model support), SGLang can win on structured-output workloads, TensorRT-LLM is a last-mile NVIDIA optimization, and llama.cpp is for edge or dev workstations [Ghalme, 2026].

Case study: the build-vs-buy trap¶

A FinTech chatbot on GPT-4o went from $15K in month one to $35K in month two to $60K in month three, projecting $700K annually. The team panicked and self-hosted everything. Six months later they were spending $85K per month on infrastructure, a part-time DevOps engineer, and three CUDA production incidents. They eventually reached $8K per month — not by self-hosting everything, but by routing intelligently [Tianpan, 2026].

Lessons from that arc:

The per-token rate is the minimum you will pay, not the total cost. Hosted APIs hide rate-limit engineering, egress, and committed-throughput lock-in.
The GPU quote is also the minimum. Staff, maintenance, security, and cold-start replicas dominate the true self-hosting bill; hardware and electricity together account for only 20–30% of TCO in peer-reviewed analyses.
Hybrid routing is usually the answer: self-host the high-volume cheap/middle tier and route the hard reasoning tail to a frontier API. One real production pattern runs ~70% of traffic on self-hosted Llama 3.3 70B, ~25% on Sonnet API, and ~5% on Opus API for a blended ~$8/M output tokens, 47% cheaper than pure Sonnet and 89% cheaper than pure Opus [Ghalme, 2026; Tianpan, 2026].

The single most expensive mistake is making the decision before running the math at your actual scale and utilization.

Model selection is not a single procurement decision. It is a continuously updated system:

Start with a decision tree: data sensitivity, scale economics, fallback needs, routing needs.
Count all four cost categories: API, infrastructure, DevOps, and complexity.
Ignore public benchmarks for final decisions; test on your own domain with capability, robustness, and production-simulation layers.
Use a hybrid router only when traffic is mixed and the team can operate it.
Watch the consumer-GPU P2P trap if self-hosting on RTX cards.
Plan for a primary-secondary vendor architecture and revisit it at least annually.

Even after the model is chosen and tested, production surfaces problems that demos hide: provider outages, prompt injections through RAG documents, cost spikes, model-version drift, and duplicated answers. Those are the concerns of the next layer — security, resilience, observability, and deployment.

Deeper dive: choosing and evaluating models for production¶

In production, the API-versus-self-host choice is better framed as a design constraint than as a vendor preference. It shapes data governance, latency targets, unit economics, and how quickly the team can ship. API-based models win on speed, support, and lower upfront risk; they remove infrastructure, model-version management, and on-call burden. Self-hosted models give full data custody, domain-specific customization, and fixed-cost predictability, but only if the team has the talent and observability to manage drift, upgrades, and security [Bhairav, 2026; Abstract Algorithms, 2026]. The most common successful pattern is to start API-first, prove value, then selectively migrate privacy-sensitive, high-stakes, or cost-heavy workloads to a self-hosted layer while keeping the API as a fallback.

That migration is only safe if the evaluation layer is built first. Build a versioned golden set of real production examples, run automated regression in CI on every model, prompt, or engine change, and add online or shadow scoring against production traffic before scaling self-hosted capacity [Abstract Algorithms, 2026]. Cost and latency are first-class quality dimensions, not operational afterthoughts: a model that is slightly more accurate but several times slower or more expensive is usually the wrong choice for a user-facing surface.

A practical seven-factor scorecard maps the hosting decision to three tiers. Score each factor as 0 (strong API signal), 1 (neutral / hybrid), or 2 (strong self-host signal) and sum the result: daily token volume, latency SLA, data privacy, customization need, team ML capability, budget predictability, and model freshness [Abstract Algorithms, 2026]. The totals translate to a deployment mode:

API-first (0–4 points) — under ~5M tokens/day, no regulated PII/PHI, latency above 500 ms TTFT acceptable, prompt engineering reaches target quality, and the team has no dedicated MLOps function.
Hybrid routing (5–9 points) — 5M–50M tokens/day, some privacy or cost pressure, 200–500 ms TTFT, LoRA-style adapters are sufficient, and a small ML/SRE team can operate a self-hosted tier.
Self-host first (10–14 points) — over ~50M tokens/day, strict residency requirements, sub-200 ms TTFT, full fine-tuning or a 12-month stable model version required, and a dedicated ML platform team is in place.

References¶

Abstract Algorithms. "Build vs Buy: Deploying Your Own LLM vs Using ChatGPT, Gemini, and Claude APIs." Apr 2026. https://www.abstractalgorithms.dev/build-vs-buy-llm-self-host-vs-api
AWS. "Multi-LLM routing strategies for generative AI applications." Apr 2025. https://aws.amazon.com/blogs/machine-learning/multi-llm-routing-strategies-for-generative-ai-applications-on-aws/
Clarity. "How to Evaluate LLMs for Enterprise Use Without Getting Fooled by Benchmarks." Mar 2026. https://heyclarity.dev/blog/how-to-evaluate-llm-enterprise-use-without-getting-fooled-by-benchmarks/
McLeod, Sam. "Patching NVIDIA's driver and vLLM to enable P2P on consumer GPUs." Feb 2026. https://smcleod.net/2026/02/patching-nvidias-driver-and-vllm-to-enable-p2p-on-consumer-gpus/
Puget Systems. "Problems With RTX 4090 Multi-GPU and AMD vs Intel vs RTX 6000 Ada." https://www.pugetsystems.com/labs/hpc/problems-with-rtx4090-multigpu-and-amd-vs-intel-vs-rtx6000ada-or-rtx3090/
Resilio Tech. "When to Self-Host AI Models vs. Use API Providers: A Decision Framework." Apr 2026. https://resiliotech.com/blog/when-to-self-host-ai-models-vs-use-api-providers-decision-framework
TrueFoundry. "Intelligent LLM Routing: Cost & Quality-Aware Selection." Jun 2026. https://www.truefoundry.com/blog/llm-routing-cost-quality-aware-model-selection
TrueFoundry. "LLM Benchmarking for Enterprise Production." May 2026. https://www.truefoundry.com/blog/llm-benchmarking-enterprise-production
Wolyra. "Enterprise LLM Selection Criteria: 2026 Framework." May 2026. https://wolyra.ai/enterprise-llm-selection-framework-2026/
Bhairav, Suhas. "API-Based LLMs vs Self-Hosted LLMs: Production tradeoffs and deployment patterns." Jun 2026. https://suhasbhairav.com/blog/api-based-llms-vs-self-hosted-llms-fast-product-launch-vs-long-term-cost-control