Background. Prefix caching lets an instance skip prefill work for a prompt sharing a prefix with one it served earlier. On a single instance that is a cache replacement problem.

Problem. Across a cluster the reuse depends on where the request lands. A load-balanced router scatters requests that share a prefix over many instances, so each one recomputes the same prefix and the cache that would have paid for itself is never hit.

Key idea. Schedule and place requests with prefix reuse as an explicit objective, so a request carrying a long shared prefix reaches the instance that already holds it. Cache reuse becomes a scheduling and placement question rather than a per-instance cache policy.

Findings. Across five workloads on clusters of two to eight GPUs, Preble lowers average request latency by 1.5-14.5× and p99 latency by 2-10× against distributed SGLang.

Why it matters. It makes prefix locality a first-class input to routing, and it sets up the tension the rest of this meeting works on. Affinity concentrates traffic exactly where the popular prefixes live.

Background. A serving cluster picks an instance for each request when the request arrives, and the request stays on that instance until it finishes generating.

Problem. Output lengths are unknown at arrival, so a placement that was balanced when it was made goes stale. Memory fills unevenly, requests get preempted on one instance while another has room, and a one-shot scheduler can no longer touch either.

Key idea. Migrate running requests between instances, KV cache included, so a placement can be revised while the request is still generating. Scheduling becomes a continuous decision rather than a single admission-time choice.

Findings. Against the evaluated serving systems, Llumnix improves tail latency by an order of magnitude, accelerates high-priority requests by up to 1.5×, and reduces cost by up to 36% at similar tail latency.

Why it matters. It puts LLM serving on the same ground as virtual-machine live migration, and makes the cost of moving state the quantity to reason about. Every later routing paper either pays that cost or explains why it only reroutes queued requests.

Background. Prefix-aware routers send each request to the instance that already holds its prefix, which is what makes distributed prefix caching pay for itself.

Problem. Affinity and balance pull in opposite directions. Pinning a prefix to one instance sends every request carrying that prefix there, and the popular prefixes are the ones carrying the most traffic, so the best cache hit rate produces the worst load skew.

Key idea. Hash each request to two candidate instances using independent prompt hashes, then choose between them by live load. Power-of-two-choices adapted to prefix locality, with a fallback to load-aware placement once TTFT breaches the SLO.

Findings. On real workloads, DualMap supports up to 2.25× more requests than prior schedulers under the same time-to-first-token SLO.

Why it matters. It quantifies the cache-affinity versus load-balance conflict rather than asserting it, and shows what a second candidate buys: most of the balance back, at the cost of one more place each prefix has to be cached.

Background. Prefix-aware routing tries to send a request to an instance that already caches its prefix, while load-aware routing tries to keep work balanced. Existing schedulers combine the two signals.

Problem. A linear combination requires workload-specific weights, and a simulator requires an accurate model of the hardware and traffic. Both add tuning cost and can still choose a poor operating point.

Key idea. Multiply two indicators: the number of new prefill tokens if the request is routed to an instance, and that instance's current batch size. The comparison cancels the hyperparameters that a linear combination would expose, so the score needs no workload-specific tuning.

Findings. On real chatbot and coding-agent workloads, LMetric reduces time to first token by 92% against vLLM-v1 and 39% against a production scheduler. Time per output token falls by 24% and 51%, respectively. The paper also derives detectable conditions under which multiplication can fail.

Why it matters. LMetric replaces a tuned tradeoff with a score whose two factors retain clear units. The derived failure conditions make its simplicity auditable rather than heuristic.

Background. Every router above decides one request at a time. DualMap chooses between two candidates per request and CacheRoute plans per prefix key, but the unit being placed is still the individual request.

Problem. An agent does not send individual requests, it sends a session, and the requests within a session share almost all of their prefix. A per-request rule therefore re-derives the same affinity decision on every turn, and inherits the same load skew each time it does.

Key idea. Move the decision to the session. Route a session's first request purely for load balance, then route its follow-ups cache-aware to whichever instance the first one landed on. Balance is chosen once, at the moment nothing is cached yet and the choice costs no reuse, and reuse is then collected on every turn after.

Findings. KV reuse exceeds 80% of request tokens in a production trace from BAILIAN, against 54-62% in chat, which is what makes a single placement per session sufficient. Measured against state-of-the-art schedulers, cluster throughput rises 10-16% under prefill-decode colocation with a global store, and prefill throughput rises 2-34% under disaggregation.

Why it matters. Routing granularity can remove the affinity-balance tradeoff: choose balance before state exists, then preserve affinity. The result depends on sessions being long and prefix-stable.

Background. Instance routers choose where one model runs. A deployment with cheap and expensive models must instead choose which model answers.

Problem. Most requests do not need the strong model, but their quality gain is unknown before either model answers. A fixed traffic split cannot follow a changing budget.

Key idea. Learn the gain from human preference data. Train router models on pairwise preferences between model outputs to predict whether the weak model's answer would be preferred for a given prompt, then expose one threshold that trades quality against cost, so the router traces a frontier rather than sitting at a single operating point.

Findings. On the evaluated benchmarks, learned routers sometimes cut cost by more than 2× against always using the strong model with no measured quality loss. Performance largely transfers to replacement model pairs.

Why it matters. The quality-cost frontier becomes the design object, and existing preference data becomes a training signal for allocating model cost.

Background. A small model that fits on a low-cost device, an edge device included, is far cheaper to run than a large model behind a cloud API, and it answers many requests about as well.

Problem. It does not answer all of them as well, and the deployment cannot tell in advance which ones those are. Routing on the prompt alone therefore means predicting how hard a request is. Moreover, how much difficulty is tolerable is not a fixed quantity: it depends on how much quality the deployment is willing to give up, which differs between scenarios and can change after the router is trained.

Key idea. Train a router to predict query difficulty and send the easy requests to the small model, with the desired quality level as an explicit second input to the routing rule. That level is tunable at test time, so one trained router covers the whole range of quality-cost operating points instead of one.

Findings. The approach makes up to 40% fewer calls to the large model with no drop in response quality, measured against routing every request to the large model.

Why it matters. It separates the two quantities RouteLLM folds into a single threshold: a difficulty estimate, which is a property of the request, and a quality target, which is a property of the deployment. Separating them is what lets the same router serve a strict deployment and a thrifty one.

Background. Both routers above commit before any model runs, using only the prompt as evidence.

Problem. The prompt is weak evidence. Whether the small model will get a request right depends on the answer it would actually produce, and that evidence is nearly free, because the small model is the cheap one and running it first costs little.

Key idea. Let the small model answer, have it verify its own answer with a few-shot self-verification prompt, and escalate to a larger model only when the verification is unconvincing. Self-verification is noisy, so the escalation decision is posed as a POMDP whose hidden state is the correctness of the answer in hand and whose observation is the confidence score, which is the natural formulation once routing is a sequence of decisions rather than one.

Findings. Across five language models and five hard datasets, AutoMix consistently beats strong cascading baselines and reduces computational cost by more than 50% relative to the baselines whose performance it matches.

Why it matters. It moves routing from prediction to observation, making the cheap model's own attempt the feature the decision reads. The cost is that every escalated request pays for the small model's answer as well as the large model's, so the saving holds only while the escalation rate stays low, which is the quantity to watch in any cascade.

Background. Each router above reports a quality-cost frontier, and each reports it on its own model pair, its own benchmark mix, and its own cost accounting.

Problem. Those frontiers are not comparable to one another, so a new routing policy cannot be placed against the published ones. Evaluating one honestly also means knowing what every candidate model would have answered on every query, which is expensive enough to discourage measuring it at all.

Key idea. Precompute the outcomes once and share them. RouterBench releases more than 405,000 inference outcomes from representative LLMs over a benchmark mix, so a routing policy is evaluated by looking up what each model would have produced rather than by calling it. The paper adds a theoretical framework for routing and a comparative analysis of existing approaches under one metric.

Why it matters. Read it before designing a router. It fixes what a frontier means, supplies the non-routing reference points a router has to beat, and reduces an evaluation that would otherwise cost inference on every model to a table lookup, which is what makes iterating on a routing policy affordable.