Most AI applications are manual cars. Somewhere in the code (or in the interface), the developer (or the user) selects the gear:
model="gpt-5.6-sol"
reasoning={"effort": "medium"}The choice fixes a capability, latency profile, price and provider before the journey starts. For ordinary AI-processing jobs that can be characterised in advance (a classification endpoint, a nightly batch, a summarisation pipeline, …), this can be good engineering.
Conversational and agentic applications are much less predictable. The driver knows the destination, but not the road or the exact sequence of turns required to reach it. An agent may finish in three calls or continue for hundreds; it may invoke tools, recover from failures, accumulate a long context or discover a difficult subtask halfway through. Choosing a specific model up front means you’ll drive in third gear for hours: speeding through town streets and moving sluggishly on the highway.
An automatic transmission does not choose the destination or take control away from the driver. It handles a lower-level decision as the conditions change. A model-routed application does the same by asking for a logical model:
model="principia/agent"But in a long-running agent, the analogy begins to break down: every shift can strand the prompt cache that made the run affordable.
The router resolves that name to a physical model for this request, turn or task. Like a gearbox responding to speed, load and terrain, it can respond to difficulty, tool requirements, latency, budget and the state of the run. Its job is not to find the universally best model—there is no such thing—but the best eligible model for the application’s objective at that point in the journey.
A router is not just a gateway
First, a quick disambiguation: the terminology can be confusing because several useful pieces of infrastructure sit close to each other on the network, including routers and gateways.
An LLM gateway is an access and control layer. It can translate provider APIs into one format, hold credentials, enforce budgets and rate limits, collect traces, retry failures, and balance traffic between deployments. LiteLLM is a good example: its proxy exposes a common API over many providers, while its own Router handles retries, fallbacks and application-level load balancing.
A model router makes a policy decision: which model should attempt this work? It can inspect the request, predicted difficulty, required tools, user preference, budget, latency target, data boundary, previous turns and live capacity.
The layers are complementary—and often technically very close to each other: a semantic router might sit immediately before, after or inside a gateway. Some gateways make policy decisions as well. For instance, a call from a particular application might be classified as sensitive, so it is sent to a model in the EU data zone.
In this article, however, we will focus on model routing with a particular emphasis on cost-based policies: how do we pick the right model at the right time to minimise cost and maximise utility?
Why route at all?
Cost is the obvious reason. The frontier model is an expensive default for turns that are mostly classification, formatting, retrieval or mechanical tool use. Sending the easy fraction to a smaller model can move the blended cost dramatically.
Quality is the converse of the cost problem. A team may have pinned a workflow to an inexpensive, underpowered model and then forgotten about the choice. As the workload evolves, that stale default can silently cap quality. A router can recognise when a request needs more capability and escalate it instead of forcing every task through yesterday’s compromise.
Both users and developers are poor model selectors. On our platform, we have observed that 93% of users pick a model and stick with it—regardless of task complexity or model recency. Of those users, the vast majority choose the most expensive model available. Large-scale providers (OpenAI with ChatGPT, Microsoft with GitHub Copilot, …) have long recognised this, offering an “auto” mode in their UI. For fixed-price subscription services in particular, being able to serve the cheapest model that still satisfies users’ needs directly affects your bottom line.
How to route, then?
Here are a few strategies and decision factors, from the simplest to the most elaborate.
Rules and capability filters
Make simple algorithmic choices based on input metadata and heuristics: modality, language, context length, tool availability, … Rules provide hard guarantees and make an excellent first stage, with the advantage of being trivially explainable. The router may even expose a “requested level” field if the caller is confident in the capabilities it requires.
Semantic and complexity classifiers
Predict a task class or difficulty from the input, using embeddings or a small classifier or SLM. For conversational use, the input can be either the immediate message or the full conversation.
Cascades and escalation
Evaluate the quality of the model outputs a posteriori, and downgrade or upgrade accordingly. For instance, a cheap model that produces a malformed tool call or attempts an illegal CLI command gets upgraded to a more expensive one on the next turn. This has been framed as a “cascade of models” in FrugalGPT.
All of the above (and more)
It’s important to treat these elements as potentially cumulative rather than exclusive. Note, however, that the more elaborate your routing system gets, the more drawbacks pile on: latency, evaluation and calibration complexity, and reduced robustness to new uses, …
How is routing impacted by agentic usage?
An atomic request used to be a useful conceptual unit of work: a chat message, a text-processing job, … Nowadays, the same atomic request sent to an agent may trigger a run that lasts for hours and uses tools hundreds of times. Routing on the initial user message is simple, but makes little sense, as it would lock the whole interaction and its numerous LLM invocations to a single model. Should such a naive router initially pick a model that is not smart enough, the end result may very well produce no value at all, and millions of tokens may be spent for nothing—nullifying all possible cost gains. Routing has to remain revisable at every turn during the agent run.
But doing so creates more challenges:
Route per turn, but don’t flip-flop at every step.
A weak decision compounds. Saving on one call is irrelevant if the chosen model takes five extra turns, makes an unsafe tool call or sends the agent down the wrong branch. Evaluate task success, total trajectory cost and wall-clock time.
Some states are not portable. Provider-managed reasoning state may not transfer between models, and tokens that fall outside the next model’s context window cannot be carried over.
The router can now be a serious latency bottleneck. Agent turns may be as close as a couple of seconds apart. If hundreds of agents (including sub-agents) are running, the router might need to make hundreds of decisions per second.
Prompt caching rewards continuity. Providers can reuse the processed state (KV cache) of an identical prompt prefix. In an agent history that stays on one model, the input of turn 67 requires processing only the new turn, while the preceding 66 turns can be reused. A new model introduced by the router will need to reprocess the full history, causing a large cost and latency penalty. A model with a lower input-token price can therefore cost more for the next call than a more expensive model with a warm cache. Moreover, caches have a short TTL: typically on the order of minutes to tens of minutes (this feels long, but remember that an agent might run for tens of hours). When the router sends the agent back to the first model, a purged cache will cause another price and latency penalty. OpenAI’s current GPT-5.6 documentation makes the economics explicit: cache writes cost 1.25× the uncached input rate, reads cost 0.1×, and entries have a 30-minute minimum lifetime that refreshes on reuse. Anthropic documents the same 1.25×/0.1× multipliers for its default five-minute cache. Exact mechanics vary by provider, model and deployment, so a router must observe actual usage fields rather than assume a hit.
Simulate routing and cache locality
We wanted to get a sense of the savings (or lack thereof) associated with routing, so we built a small simulation app.
The simulator generates a sequence of calls across Sol, Terra and Luna. “Random” chooses a model at random on every turn; “correlated” gives the next turn a configurable chance of remaining on the same model. These are switching patterns, not claims about how a real router decides.
The default prices use the promotional GPT-5.6 Sol, Terra and Luna API rates published on 3 September 2026. The model is intentionally editable and simplified: it is designed to expose the interaction rather than serve as a billing calculator.
What does switching do to the bill?
75% chance that the next turn stays on the same model.
Assumptions and extra parameters
Cache writes cost 1.25× input and reads cost 0.1×. The prompt grows by new input plus the previous output on every turn.
$15.35 saved (83%) · 96% routed-input cache hit rate
This is a cost model, not a quality benchmark. Model selection is synthetic; router inference, retries, tool fees, reasoning-token differences, compaction and quality-driven extra turns are omitted. Each model has its own cache, and a returning model can reuse its older prefix only before it expires.
Try three moves:
- Turn the cache off and on with a few different configurations. You’ll notice that its effect on cost can be at least as large as that of model switching.
- Select Sol and Luna with the correlated strategy. Move stickiness from 10% towards 90% and watch the number of switches decrease. Observe how the savings change as longer runs allow an inactive model’s cache to expire.
- Under “Assumptions and extra parameters”, move the number of turns between 20 and 240. Notice how the gains can increase or decrease with the run length, depending on the configuration.
The simulation is still generous to routing. It assumes identical tokenisation and output length across models, with no routing calls, retries, quality losses or extra turns.
Rethinking routing in the agentic age
Agentic usage doesn’t only come with challenges to routing, but also with opportunities. Long running sessions are usually made of rounds of compaction, sub-agents exploring well scoped sub-tasks, and generally taking decision about how the task at hand ought to be solved. All of these open new doors to optimize model choice : compaction could be delegated to a cheaper model, provided it preserves task-critical information, sub-tasks complexity can often be estimated in advance by the main agent, … They can also be interesting boundaries to evaluate routing choices. In the age of long-running agents, agent harness, routing and caching cannot be addressed separately. To obtain the best savings, the whole system must be optimized as one.
Open-source projects worth reading
Principia’s editorial bias is towards open-source and self-hosting, and we’ll regularly highlights such projects readers can explore and try themselves. On this particular matter, it is particularly important for a routing layer that might see a sizeable chunk of your AI usage. This is why we feature these four repositories in particular, although performant managed and closed implementations exist as well.
NeMo Switchyard
OpenAI Chat Completions, Responses and Anthropic Messages surfaces; random, classifier, stage, composite, escalation, advisor and sub-agent-aware routes.
Read it as: a young, pragmatic agent-routing server with policies worth studying.
RouteLLM
Four trained router families, threshold calibration, benchmarks and an OpenAI-compatible server.
Read it as: the compact research baseline and easiest mental model.
vLLM Semantic Router
Combines request signals, user preferences and application policies across heterogeneous compute and providers.
Read it as: a broad systems project, not just a prompt classifier.
LLMRouter
16+ methods across single-turn, multi-turn, multimodal, agentic and personalised routing, plus data generation and evaluation tooling.
Read it as: a catalogue for experimenting across router families.
Want to go further?
This article is a deliberate overview of model routing. We conducted a comprehensive benchmark of our solution, LaPlace, which you can find here. TODO: add a link here.
Sources and further reading
- Isaac Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data”, 2024.
- Lingjiao Chen, Matei Zaharia and James Zou, “FrugalGPT”, 2023.
- Martian, “RouterBench: A Benchmark for Multi-LLM Routing System”, 2024.
- Elias Lumer et al., “Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks”, 2026.
- OpenAI prompt-caching guide and GPT-5.6 Sol price card, checked 3 September 2026.
- Anthropic prompt-caching documentation and pricing, checked 3 September 2026.