A team running agents in production can end up paying list price for every token, a percentage fee on top through its gateway, and frontier-model rates for steps like classification and formatting that a smaller model could handle as well.
An LLM router can cut into both costs. The gateway side decides what you pay per token and in fees, and model routing decides which model each request goes to, although a cheaper model only saves money if it still gets the answer right.
Below are the best LLM routers and AI model routers, compared on what they charge, how they route, and who each one suits.
TL;DR: Best LLM routers
| Router | Pricing | Routes on | Fallback |
|---|---|---|---|
| Respan | Up to 55% off, 0% fee | Price, speed, task (Span Router) | Ordered chains, retries, load balancing |
| OpenRouter | 5.5% credit fee | Price, throughput, task (Auto Router) | Provider and model fallbacks |
| LiteLLM | Free OSS, Enterprise custom | Rules, budgets, classifier (beta) | Ordered chains, retries |
| Portkey | Free OSS, hosted via sales | Request metadata conditions | Fallbacks, retries, timeouts |
| Bifrost | Free OSS, Enterprise via sales | Weighted keys, routing rules | Ordered chain after retries |
| Vercel AI Gateway | No markup, paid add-ons | Provider health and order | Provider and model fallbacks |
| Microsoft Foundry model router | Routing fee + model tokens | Prompt demand, routing mode | Next-best model on errors |
| Amazon Bedrock Intelligent Prompt Routing | $1/1K requests + tokens | Predicted quality vs. cost | Fallback model as baseline |
| Cloudflare AI Gateway | Free core, 5% credit fee | Metadata, splits, budget limits | Fallback path at limits |
Respan is the pick specifically for teams that want one API key to 1,000+ models at up to 55% off per token with a 0% fee, providers ordered by price by default, the span-router model for choosing a model per task, and a trace on every request.
What is an LLM router?
An LLM router sits between your application and provider APIs and selects the destination for each request. Your code calls one OpenAI-compatible endpoint, and the router picks OpenAI, Anthropic, Bedrock, or an open-weight model behind it, based on rules, price, health signals, or a read of the request itself.
Because the selection happens in the routing layer, changing models becomes a config change rather than an application deploy.
Routers are not the same as destinations. Groq and Together AI are places a request can land, and a router is the thing choosing among them.
An LLM gateway is the related layer that handles the shared endpoint, keys, caching, and logging, and many products ship both. If the interface matters more to you than the routing decision, the best LLM gateways comparison sorts the same vendors on that basis instead.
An AI model router is the subset of routers that decides which model should answer each request, rather than only which provider should serve a model you already named.
How AI model routers pick a model
Routers differ in what information they use to choose a destination, and each step adds something the one before it can't see:
- Failover on errors - The router watches for 5xx responses, timeouts, and 429s, then sends the request to the next model in an ordered list. Load balancing spreads traffic across keys or regions so one account's quota doesn't become the ceiling.
- Price and speed across providers - When several providers serve the same model, the router orders them by price or by measured output speed, which changes what you pay or how fast tokens arrive without changing the model.
- Rules on the request - Metadata, prompt length, or an explicit task tag decides the model, so classification and formatting go to a small model and multi-step reasoning goes to a frontier one.
- A classifier on the prompt - AI model routers read the prompt, system message, and tool definitions before the model call and pick the model per request, which adds a hop worth measuring.
- Scores from production traffic - LLM-as-a-judge results, rule-based checks, human review, or online evals show whether a switch to a cheaper model actually held quality. They need to sit on the same request IDs as the routing decision, and keeping them in a separate product means stitching identifiers across two systems.
Failover only catches failures that announce themselves, so a fluent, wrong answer with a 200 status code passes every resilience check. Cheaper routing carries the opposite risk, because the savings only count net of the retries, escalations, or human cleanup a weaker answer causes, which is why model selection needs scores on the same traffic.
9 best LLM routers and AI model routers compared
How we picked
To make the cut, a tool had to route live production traffic through one endpoint and pass four checks against its own docs and pricing pages:
- It makes a routing decision - It chooses where each request goes, by provider, price, rules, or a read of the prompt, instead of forwarding everything to one fixed model.
- Fallback is documented - The docs say what happens when a provider errors or rate-limits.
- Pricing is clear - You can tell what you'll pay on top of tokens, or the entry says plainly when that takes a sales call.
- You can use it today - It's a product you can deploy or sign up for, not a research framework like RouteLLM, and not a tool that only recommends a model while another gateway serves the request.
1. Respan

- Pricing: Up to 55% off per token on Respan credits with a 0% fee; Free tier, Team at $199/month billed yearly ($249 monthly), Enterprise custom.
- Routes on: Price by default (cheapest provider first),
:fastestfor output speed, and thespan-routermodel for choosing a model per task. - Fallback: Ordered model chains set at the org level or per request, with retries and load balancing.
- Best fit: Teams that want lower token prices, provider and model routing, failover, and tracing in one platform.
- Not ideal if: You plan to use your own provider keys on the free plan, since the key vault starts on Team.
Respan's AI gateway is one API key to 1,000+ models at up to 55% off per token with a 0% fee, because bulk rates are passed on instead of marked up. It's also built to stay out of the request path: in Respan-run benchmarks it handles 56,734 requests per second with 0.07 ms median added latency.
Routing starts with price. Most models are served by more than one provider, and by default the gateway tries them from lowest to highest price, so the cheapest route is the one you get without configuring anything. Add :fastest to a model name to order providers by measured output speed instead, or set the model to span-router, which classifies each request with Span-02 and sends it to the model best suited to that task.
- Pay less per token - Up to 55% off per token on Respan credits, with a 0% fee, so moving traffic through the gateway lowers the bill before any routing decision is made.
- Stay up when a provider fails - Ordered model fallback moves an upstream error to the next model in your list, with automatic retries and load balancing across keys and providers.
- See every routing decision - Each call becomes a trace showing the model attempted, whether a fallback fired, cache hit or miss, tokens, latency, and cost.
- Reach 1,000+ models through one endpoint - Send OpenAI-style calls across every major provider, or keep a provider's native SDK on a passthrough endpoint and still get unified logs.
- Run routing on your own infrastructure - Respan's open-source gateway runs as a single binary under MIT or Apache-2.0, and any OpenAI, Anthropic, or Gemini SDK can call any model through it. Its routing runs on rules you configure: named routes, ordered fallback, retries, hedged races between providers, and picking the cheapest or fastest candidate. The Respan plugin sends each request into the platform as a trace.
- Know where the money goes - Soft warnings or hard caps per API key, Slack and email alerts when a threshold crosses, and response caching on exact repeat requests.
- Stop a request before it routes anywhere - Respan Guardrails check each request inside the gateway and block categories like prompt injection or system prompt leak before they reach any model, so a blocked request spends no provider tokens.
- Score quality on production traffic - Rule-based checks and LLM-as-a-judge evaluators run on sampled production traces, so a switch to a cheaper route comes with numbers behind it.
Pricing starts at $0 with 100k logs, 1k scores, and 4,200 requests per minute of proxy throughput. Team is $199 per month billed yearly and raises throughput to 8,400 requests per minute with unlimited prompts, datasets, and evaluators, while the open-source gateway stays free to run yourself.
2. OpenRouter
- Pricing: 5.5% fee on credit purchases ($0.80 minimum), 8% on Business, no markup on inference.
- Routes on: Price and throughput sorting across providers, plus Auto Router for choosing a model.
- Fallback: Automatic fallback to the next provider, plus model fallback lists.
- Best fit: Reaching a large model catalog through one account and one balance.
- Not ideal if: Your spend is high enough that a 5.5% fee on every credit purchase is a line item worth removing.
OpenRouter exposes 600+ models behind one OpenAI-compatible endpoint as a managed service. Stripe agreed to acquire it in August 2026, and OpenRouter says its product and roadmap stay the same. Provider routing sorts by price or throughput, and when a provider returns an error, OpenRouter falls back to the next one.
For choosing the model itself, Auto Router narrows the pool to models that fit your constraints, has a decision model rate each request on aspects like task type and difficulty, and weighs that against live prices, cache status, and benchmark results. Auto Code applies the same engine to coding agents, and either way you pay the selected model's standard rate with no extra fee. Those signals describe the request and the market rather than your own outputs, so checking whether a routed choice held up on your traffic is still work you'd do elsewhere, which is the tradeoff to weigh when comparing OpenRouter alternatives.
The fee sits on credit purchases at 5.5% with a $0.80 minimum on Standard and 8% on Business, so a $10 top-up effectively pays 8% while larger purchases pay the headline rate. Bring-your-own-key traffic is free up to $25,000 of list-price inference a month, then 5%.
3. LiteLLM
- Pricing: Free open source, Enterprise priced on request.
- Routes on: Rule-based fallback chains, budget thresholds, and load balancing, plus classifier-based Auto Routing in beta.
- Fallback: Ordered fallback chains with retries.
- Best fit: Platform teams that want the routing layer on their own infrastructure.
- Not ideal if: You need SSO, RBAC, and audit logs without buying an Enterprise license.
LiteLLM gives you one OpenAI-format interface to 140+ providers, as a Python SDK or an MIT-licensed proxy server you host. Fallback chains are declarative, so an ordered list across Azure OpenAI, OpenAI direct, and Anthropic is config rather than application code, and virtual keys, per-project budgets, and rate limits all live in the proxy.
Auto Routing, still in beta, adds per-request model selection through a heuristic scorer by default, with an LLM classifier, keyword rules, or a custom plugin as options, although defining your own tiers relies on an Enterprise capability. Observability arrives through callbacks into an external tool, which puts the trace and the routing decision in different systems by default, a split worth weighing against LiteLLM alternatives that keep both in one place.
Self-hosting also means owning the supply chain. Datadog Security Labs reported that malicious LiteLLM releases were published to PyPI in March 2026 as part of a broader supply-chain campaign, and because the proxy holds every provider key you own, it's worth pinning versions and reviewing dependency updates the way you would for any credential-bearing service.
4. Portkey
- Pricing: Open-source gateway free; the hosted product is sold as Prisma AIRS AI Gateway through sales.
- Routes on: Conditional rules on request metadata, plus weighted load balancing.
- Fallback: Fallbacks to another provider or model, retries, and request timeouts.
- Best fit: Rule-driven routing with guardrails, self-hosted or under an enterprise security contract.
- Not ideal if: Published, self-serve pricing for a hosted gateway is a requirement.
Portkey's gateway is open source under MIT and runs locally with one command, covering conditional routing, fallbacks, retries with exponential backoff, weighted load balancing, caching, and pre-built guardrails. Portkey says its enterprise gateway features are merging into the open-source project with its 2.0 release.
The hosted product is now Prisma AIRS AI Gateway from Palo Alto Networks, generally available since July 16, 2026, and Portkey's old pricing page redirects there with a demo request in place of published tiers. For teams that chose Portkey for its self-serve plans, that's the change to weigh against Portkey alternatives. Conditional routing is also only as current as the rules someone last wrote, so selection that adapts per request is something you'd build on top.
5. Bifrost
- Pricing: Free and open source under Apache 2.0; Enterprise through sales, with a 14-day trial.
- Routes on: Weighted keys, routing rules, and load balancing across virtual keys.
- Fallback: An ordered chain, tried once the primary's retries run out.
- Best fit: Teams that want a self-hosted gateway running inside their own VPC.
- Not ideal if: A managed service is what you're after, since Bifrost's site describes in-VPC deployment rather than a hosted option.
Bifrost, from Maxim AI, is an open-source gateway you deploy in your own VPC, with access to 1,000+ models and drop-in compatibility with OpenAI, Anthropic, and LiteLLM clients through a base URL change. Fallbacks run as an ordered chain: if the primary fails after its retries, Bifrost moves to the first fallback whatever the error, and if every provider fails, you get the primary's error back.
Routing is configuration, with weighted keys, routing rules, and virtual key load balancing deciding the chain, while adaptive routing is listed under Bifrost Enterprise. Enterprise pricing isn't published, so beyond the free open-source build, cost is a sales conversation, and running the gateway yourself means owning its uptime and upgrades.
6. Vercel AI Gateway
- Pricing: Provider list price with no markup, including on BYOK; add-ons billed per use.
- Routes on: Provider health and the provider order you set.
- Fallback: Ordered provider and model fallbacks.
- Best fit: Teams already on Vercel that want failover and spend controls without running a proxy.
- Not ideal if: You want the gateway to choose the model, since Vercel's own router guide keeps that decision in your code.
Vercel AI Gateway is a hosted endpoint you can call from any cloud, not only from Vercel deployments, with one key in front of models across providers. It routes a model across healthy providers, lets you set a provider order, and supports ordered model fallbacks, while Vercel's guide to building your own model router keeps the choice of model in your application code.
The no-markup headline is accurate and incomplete. Tokens pass through at provider list price, including on BYOK, but add-ons bill separately once you turn them on:
- Custom Reporting - $0.075 per 1,000 tag or user ID writes and $5 per 1,000 reporting queries.
- Team-wide provider allowlist - $0.10 per 1,000 billed requests, on Pro and Enterprise.
- Team-wide zero data retention - $0.10 per 1,000 requests.
- Regional hosting (beta) - 5% of inference cost on requests sent to a regional endpoint.
- Trace Drains - $0.05 per 1,000 traces plus $0.50 per GB, on Pro and Enterprise.
At volume, those per-thousand lines can become a real share of the gateway bill. The free tier covers a subset of models at lower rate limits and stops once you buy credits, BYOK needs purchased credits, and a failed BYOK call retries on Vercel's keys against your balance by default.
7. Microsoft Foundry model router
- Pricing: A routing fee per 1M input tokens, plus the tokens of whichever model answers.
- Routes on: How demanding each prompt is, weighed by the routing mode (Balanced, Quality, or Cost).
- Fallback: Fails over to the next-best model on a retryable error.
- Best fit: Teams already deploying models in Microsoft Foundry on Azure.
- Not ideal if: Your traffic needs long context across the whole pool, since the smallest model's window sets the limit.
Microsoft Foundry model router deploys like any other model in Foundry and works as an AI model router inside Azure. It's a routing model trained by Microsoft that judges how demanding each request is from its text and sends it to the model best suited to answer, from a pool Microsoft lists at 45 models across OpenAI, Anthropic, Microsoft AI, xAI, DeepSeek, Meta, and Fireworks. Balanced mode picks the most cost-effective model within a small quality band of the best one, Cost widens that band, and Quality ignores cost.
Handing over the choice changes how individual requests behave. The effective context window is the smallest one in your pool, optional parameters like temperature are dropped when the chosen model doesn't support them, and Claude models only join once they're deployed in the same Foundry resource. Microsoft's docs recommend comparing the router with your current model on your own prompts before moving production traffic.
Billing has two parts: a routing fee per million input tokens and the selected model's usage at its own rates. Each response's model field names the model that answered, so you can see where traffic went.
8. Amazon Bedrock Intelligent Prompt Routing
- Pricing: $1 per 1,000 routed requests, plus model tokens.
- Routes on: Predicted response quality against cost, between two models in one family.
- Fallback: A fallback model you name acts as the baseline.
- Best fit: AWS-resident teams already using Amazon Nova or Meta Llama models on Bedrock.
- Not ideal if: You need routing across providers or model families, or your prompts aren't mostly in English.
Amazon Bedrock Intelligent Prompt Routing predicts how well each model in a family would answer a prompt and routes to the one with the best mix of response quality and cost. You configure a router with exactly two models from the same family, one of them the fallback that serves as the baseline, plus a response quality difference that sets how much better the other model has to be before the router switches.
Default routers use select Amazon Nova and Meta Llama models. AWS also notes that the router is optimized for English prompts and can't adjust its decisions based on your application's own performance data, so the routing reflects what the router learned in training rather than what your traffic shows.
AWS charges $1 per 1,000 routed requests on top of the token costs of the model that answers. That charge applies whether or not the router moves a request off the fallback model, so at high volume it's worth comparing against the savings the switches actually produce.
9. Cloudflare AI Gateway
- Pricing: Core features free on all plans; 5% fee on Unified Billing credit purchases.
- Routes on: Conditions on metadata, headers, or the request body, percentage splits, and budget or rate limits.
- Fallback: A fallback path when a rate or budget limit is hit.
- Best fit: Teams on Cloudflare that want routing and spend controls at no platform cost.
- Not ideal if: Routing needs to react to output quality, since Dynamic Routing branches on request data and limits.
Cloudflare AI Gateway runs on Cloudflare's network, with analytics, caching, and rate limiting included at no cost, and DLP scanning is free on all plans, with two predefined profiles available without a Zero Trust subscription.
Dynamic Routing adds conditional flows: branch on request body, headers, or metadata, split traffic by percentage for A/B tests and gradual rollouts, and switch to a fallback path when a rate or budget limit is hit. Unified Billing, which lets you pay for inference through Cloudflare, passes provider rates through with no markup and adds 5% to credit purchases.
For accounts that created their first gateway before September 24, 2026, Workers Free stores 100,000 logs total across all gateways and Workers Paid raises that to 10,000,000 per gateway, while newer accounts follow Workers Logs pricing, so log volume is worth sizing before you rely on it.
Best LLM router for each use case
The right router depends mostly on what you're optimizing for and where your traffic already runs:
- Cutting cost per token - Respan, with up to 55% off per token, a 0% fee, and providers ordered by price by default.
- Choosing a model per request - Respan's
span-routermodel, with OpenRouter's Auto Router and LiteLLM's Auto Routing (beta) as alternatives. - Self-hosting the router - Respan's open-source gateway, with LiteLLM, Bifrost, and Portkey's open-source gateway as other options.
- Staying inside Azure or AWS - Foundry model router or Bedrock Intelligent Prompt Routing, which keep routing inside the cloud your models already run in.
- Already on Vercel or Cloudflare - Their own gateways, which add failover and spend controls to the platform you deploy on.
When more than one of these applies, cost per token tends to matter most at scale, because it applies to every request no matter how the request is routed.
How to evaluate an LLM router
Vendors compete on model counts because model counts are easy to print on a page. What separates routers is narrower and harder to market, and most of it comes down to the routing decision and what it costs you:
- What you pay on top of tokens - A fee on credit purchases, a per-request routing charge, or add-ons billed per thousand requests all scale with volume, so price them at your real traffic.
- What the router routes on - Provider price or health, request metadata, or a classification of the request itself. Each answers a different question, so match it to the decision you actually need made.
- Quality signals behind the routing policy - Whether the policy can act on scored outputs from real traffic, through evals on production traffic, or whether it only sees status codes and token prices.
- Routing observability - Whether a single trace shows the model attempted, whether a fallback fired, cache hit or miss, latency per span, tokens, and cost. Without this, a routing regression is invisible until a customer reports it.
- Added latency - The router's own overhead at P95, and whether content-aware routing adds a classification hop before the model call. Measure it against your own latency budget rather than the vendor's benchmark.
Weight these against where you are. A team at 50k requests a day needs failover and budgets. A team at 50M is choosing between a frontier model and a small one thousands of times an hour and needs evidence for that choice.
FAQ
Which AI model router is the best?
Respan fits teams that want lower token prices and routing in one place, with up to 55% off per token and a 0% fee, providers ordered by price by default, the span-router model for choosing a model per task, and a trace on every request. If you're committed to Azure or AWS, Foundry or Bedrock keep routing inside your cloud, and if you'd rather self-host, LiteLLM and Bifrost are open-source options alongside Respan's own gateway.
What is the best LLM router with fallback support?
The best LLM router for fallback lets you define the chain once and shows you, on each request, whether it fired. Respan lets you set an ordered fallback chain at the org level or per request, with retries and fallback that move to the next model on rate limits, server errors, and dropped connections, and every fallback appears on the request's trace.
OpenRouter, LiteLLM, Portkey, Bifrost, and Vercel also support model fallbacks, Cloudflare switches to a fallback path when a limit is hit, and Foundry fails over to the next-best model on retryable errors.
How much does an LLM router cost?
It depends on what the router charges for: a fee on credit purchases, a per-request or per-token routing charge, platform tiers, or the infrastructure to self-host it. Respan charges a 0% fee with up to 55% off per token on Respan credits, the free tier includes 100k logs and 4,200 requests per minute, Team is $199 per month billed yearly, and Respan's open-source gateway is free to self-host.
OpenRouter charges 5.5% on credit purchases, Bedrock adds $1 per 1,000 routed requests, Foundry adds a routing fee per million input tokens, and Portkey's hosted product is now sold through Palo Alto Networks sales. LiteLLM and Bifrost are free software you host and pay for in infrastructure.
Is there an open-source LLM router?
Yes. Respan's open-source gateway is MIT or Apache-2.0 licensed and runs as a single binary on your own infrastructure. It routes on rules you configure, including named routes, ordered fallback, retries, hedged races between providers, and picking the cheapest or fastest model. Any OpenAI, Anthropic, or Gemini SDK can call any model through it, and the Respan plugin sends each request to the hosted platform as a trace when you want to trace and evaluate that traffic. LiteLLM, Bifrost, and Portkey's gateway are also open source.
What you take on is the operations: deployment, scaling, patching, and dependency review on a component holding every provider key you own, which the March 2026 LiteLLM PyPI incident made concrete. If what you want from open source is a free starting point rather than infrastructure to run, Respan's hosted free tier covers 100k logs and 4,200 requests per minute with nothing to deploy.
Can an LLM router send simple requests to a cheaper model?
Yes, and that's the job AI model routers are built for. Respan's span-router model classifies each request with Span-02 and sends it to the model best suited to that task, and OpenRouter's Auto Router and LiteLLM's Auto Routing (beta) do similar per-request selection.
Foundry and Bedrock do per-prompt selection inside Azure and AWS. Score the cheaper model on real requests before shifting traffic to it, since a downgrade that triggers retries or escalations can cost more than it saves.


