Resilience¶
Production agents depend on providers that fail. When they fail, users blame your product, not the provider. Resilience is the answer to the question: what does the system do when a dependency dies?
Providers fail in the wrong region at the wrong time¶
On 19–20 October 2025 AWS us-east-1 suffered a multi-hour disruption. A latent race condition in DynamoDB's DNS automation produced an empty DNS record for the regional endpoint. Within minutes other AWS services that depended on DynamoDB could not establish new connections. The cascade continued: EC2 launches failed, Network Load Balancers dropped healthy nodes, and services from Roblox and McDonald's to United Airlines and Fortnite went down. AWS's own postmortem describes three distinct impact windows and a recovery that required manual operator intervention [AWS, 2025].
Less than seven months later a cooling failure in one us-east-1 zone produced another broad outage. Coinbase was offline for roughly seven hours, and dozens of services experienced liquidity or availability issues. Multi-zone and multi-region designs did not help when the affected region was the one their architecture treated as primary.
A later analysis of the October 2025 incident notes that recovery took more than 15 hours, not because the DNS race condition took that long to fix, but because each recovery phase depended on the previous one: DNS correction, DynamoDB restoration, dependent systems rebuilding state, then customer applications clearing backlogs and resetting circuit breakers [ThousandEyes, 2025]. The October 2025 outage also exposed that regional failures can have global impact when coordination services, replication, or API calls touch the affected region.
These are not AWS-specific problems; they are reminders that even the most expensive cloud on the planet falls over. Any agent that assumes its provider is always reachable is an agent that will fail in production.
Three failure-mode strategies¶
When a dependency fails, a system has three coarse strategies. The choice is not an engineering default — it is a product and risk decision that should be made feature by feature.
Fail closed¶
Block the request and return an error. No autonomous action, no graceful fallback. Use this when the downside of a wrong action is worse than doing nothing.
- Payment checkout when the fraud classifier is down.
- Medical triage when the safety checker cannot run.
- Any irreversible financial transaction when risk scoring is unavailable.
Principle: better nothing than wrong. A false-positive block is annoying; a false-negative pass can be catastrophic.
Fail open¶
Let the request through even if a guard or model is unavailable. Use this when blocking a good user is more expensive than accepting a small risk.
- Read-only search when the query-enrichment model is down.
- Content recommendations when the personalization model is timing out.
- Internal dashboards when a classification service is slow.
Principle: better fast than right. Apply only where the blast radius is small and human review is easy.
Graceful degradation¶
Keep the product working in a reduced form. This is the right answer for most customer-facing features.
- Primary LLM provider down → route to a secondary provider.
- Secondary provider down → serve a cached or precomputed response.
- No cache hit → return a safe static template or hand off to a human.
Principle: better bad than none. The user should not see a 500 or an empty screen unless there is no other honest choice.
The decision is per feature¶
A single product usually needs all three strategies in different places.
| Feature | Failure strategy | Why |
|---|---|---|
| Checkout / payment | Fail closed | A fraudulent or double-billed transaction is worse than a delayed one. |
| Product search | Degrade to raw ES | Empty results lose users; raw results keep them. |
| Personalized recommendations | Degrade to top sellers | Users see relevant items even if the AI model is down. |
| Customer-support triage | Degrade to human queue | A human answers more slowly, but does not hallucinate a solution. |
Make these decisions with product, security, and legal during design, not by the engineer writing the code alone.
Provider failure taxonomy¶
Not every failure calls for the same response. Treat provider failures by class [TrueFoundry, 2026]:
| Failure class | What it looks like | Response |
|---|---|---|
| Hard 5xx | The provider returns a server error. | Fail over or fail closed, depending on the feature. |
| 429 / rate-limit | Headers like Retry-After or x-ratelimit-* indicate quota exhaustion. |
Honor headers, use exponential backoff + jitter; do not retry blindly. |
| Latency degradation | Calls are slow but eventually succeed. | Treat as a signal for circuit breaker or hedged request. |
| Partial / streaming failure | Tokens or chunks stop mid-stream. | Harder to recover from; streaming failover needs per-token state. |
| Content-filter rejection | The provider refuses the request for policy reasons. | Not a transient failure; route to policy remediation, not another provider. |
Engineering patterns for resilience¶
The patterns are the same ones backend engineers have used for years; only the payload has changed.
Circuit breakers¶
A circuit breaker around an LLM provider call stops the agent from hammering a failing endpoint. After a threshold of errors or latencies, the breaker opens and traffic is routed to a fallback. It closes again only after a probe succeeds. This is the same pattern used for microservices; it works because LLM providers are just another downstream service.
Retries with backoff and jitter¶
Retry 429 or 529 errors, but not blindly. Use exponential backoff with full jitter, respect Retry-After headers, and distinguish between retryable provider errors and terminal request errors. Anthropic's own SRE blog popularized jittered backoff for provider calls.
Naive immediate retries turn a 429 into a thundering-herd storm that sustains the overload you are trying to escape [TrueFoundry, 2026]. Jitter de-synchronizes clients; backoff reduces pressure over time.
Fallback chains and model variance¶
When retries against the primary are exhausted, a fallback chain keeps the request alive: primary, then secondary, then tertiary. Two axes matter:
- Across providers (e.g. Claude → GPT → Gemini) gives independent failure domains and independent quota pools.
- Within a provider (alternate region or deployment) is cheaper but shares the same failure domain.
The same prompt can behave differently on a fallback model, so failover is not free. Validate the fallback's output, especially for tool-calling agents [TrueFoundry, 2026].
Load balancing and hedging¶
Load balancing across providers, deployments, and independent quota pools spreads load and adds real rate-limit headroom. Extra API keys under the same organization usually share a quota, so they do not multiply it [TrueFoundry, 2026].
Hedging cuts the latency tail: after a short delay near the p95, send the same request to a second provider and take whichever returns first. It costs two calls when it fires, and cancellation may still bill for partial tokens. Use it only where the call is idempotent and the latency tail is the dominant problem [TrueFoundry, 2026].
Idempotency keys¶
Payment and mutation tools must be idempotent. If a tool call times out, the harness can retry the same idempotency key without risking a double charge or double write. Treat every tool that changes state the way you would treat a payment API.
Timeouts¶
Set explicit timeouts per stage and per tool. A slow tool that blocks the loop turns a fast request into an expensive crawl. Timeouts are also a signal: if a provider is slow, the circuit breaker should notice.
Queues for slow work¶
Not every agent task needs an immediate response. Long-running tasks should leave the synchronous loop and run in a queue. The agent returns a job ID; the queue handles retries, progress tracking, and final delivery.
Provider status monitoring¶
Watch provider status pages, but do not trust them blindly. Instrument your own error-rate and latency metrics per provider. When your own metrics show a problem, start routing around it before the provider's status page turns red.
Resilience checklist¶
| Component | Decision | Examples |
|---|---|---|
| Provider | Is there a tested fallback chain with an independent quota? | Primary Claude → secondary GPT → cached static response. |
| Retry policy | Is backoff jittered and does it honor rate-limit headers? | 2–3 retries with exponential backoff + full jitter. |
| Circuit breaker | Does the breaker open on error-rate or consecutive failures, then probe before closing? | Open after 5 consecutive 5xx; half-open with 1 probe. |
| Timeouts | Are timeouts set per stage and per tool? | Tool p99 + headroom; abort before the user timeout. |
| Idempotency | Does every write tool accept an idempotency key? | Payment, order, volume create. |
| Queue | Are long-running tasks moved out of the synchronous loop? | Job ID returned; queue handles retries and delivery. |
| Observability | Are provider error rate and latency instrumented before the status page turns red? | Per-provider p99, error rate, token burn. |
Summary¶
- Providers fail. The user will blame your product.
- Fail closed, fail open, and graceful degradation are product decisions, not engineering defaults.
- Most real systems mix all three strategies across different features.
- Circuit breakers, retries, idempotency, timeouts, queues, and provider monitoring are the backend skills that make agents survive outages.
- The code that saves you is not the system prompt; it is the infrastructure around the model.
Deeper dive: surviving provider and tool failures¶
By 2026, provider-level outages have become a measurable risk rather than a theoretical one. In April 2026 an OpenAI routing-layer out-of-memory condition took ChatGPT, Codex, and the API down for more than two and a half hours; the traffic collapse was global, not regional [VeriSwarm, 2026]. Independent monitoring of representative hosted LLM APIs showed about 99.3% uptime for the year — roughly five hours of downtime per month — with median time-to-repair of 1.23 hours for OpenAI and 0.77 hours for Anthropic [VeriSwarm, 2026]. Those numbers are an order of magnitude worse than the four-nines expectations most backend operators assume for databases or object stores, and they mean a single-provider agent will experience provider failure as a routine operating condition.
A breaker per provider or per provider/model pair is the first line of defense: it opens after a small number of consecutive failures, rejects new requests fast while the provider recovers, then probes in a half-open state before it closes again [VeriSwarm, 2026]. Breaker state should be shared across workers and exposed on a monitoring endpoint so the fleet acts as one unit and operators can force-trip or reset it manually. A circuit breaker alone only answers “what do we do this second”; the longer-horizon control is an error budget derived from the SLO. For a 99.9% availability target, only 0.1% of requests in the rolling window may fail — about ten failures out of ten thousand requests in a day. When the budget exhausts, the default posture should be to restrict: pause long-running multi-step plans, send high-confidence tool calls to human review, and shift the tenant’s trust scoring toward the conservative end of its profile [VeriSwarm, 2026]. The alternative alert only mode is appropriate when a human reviewer is already in the loop.
Provider outages are not the only failure surface. A typical agent makes several tool calls per turn; with five calls each at 99% reliability, roughly one in twenty requests already hits at least one tool failure [CallSphere, 2026]. A tool-fallback chain treats each capability as a list of implementations — for example tavily -> brave -> cached for web search — and moves to the next option rather than failing the whole turn. On top of that, a service-level ladder keeps the user experience honest when multiple dependencies are unhealthy: full service when the LLM, tools, and database are up; degraded when the LLM is available but tools are down; minimal when only cached or rule-based responses are possible; and offline when nothing critical is available [CallSphere, 2026].
| Signal or layer | What to record | Why it matters |
|---|---|---|
| LLM call site | Latency, timeout, 5xx, malformed response, content-policy rejection | Drives provider/model circuit breaker and error-budget burn. |
| Tool execution | Per-tool success/failure and fallback activation | Reveals which capabilities need a longer fallback chain. |
| Service level | full -> degraded -> minimal -> offline transitions |
Keeps the user experience explicit instead of silently failing. |
| Autonomy policy | Breaker trips, budget exhaustion, restrict vs alert actions | Links infrastructure health to trust scoring and human review. |
References¶
- AWS. "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region." Oct 2025. https://aws.amazon.com/message/101925/ (mirrored at https://gist.github.com/cosimo/53ee003ea00e4d6caa050f59d2a00a85)
- ThousandEyes. "AWS Outage Analysis: October 20, 2025." Oct 2025. https://www.thousandeyes.com/blog/aws-outage-analysis-october-20-2025
- TrueFoundry. "LLM Failover & Load Balancing for Provider Outages." Jun 2026. https://www.truefoundry.com/blog/llm-failover-load-balancing-provider-outages
- CallSphere. "Building Resilient AI Agents: Circuit Breakers, Retries, and Graceful Degradation." Mar 2026. https://callsphere.ai/blog/building-resilient-ai-agents-circuit-breakers-retries-graceful-degradation
- VeriSwarm. "Your LLM Provider Will Go Down. The Question Is Whether Your Agent Goes With It." May 2026. https://veriswarm.ai/blog/llm-provider-circuit-breakers