Background. Systems work on agents has been designed against an assumption about the workload: long prompts, growing context, prefill-heavy execution. That assumption came from reasoning about how agents work, not from traces of them running.

Problem. Designing a scheduler or a cache policy for the wrong workload shape wastes the effort. Whether agent serving is prefill-bound or decode-bound, and whether state is per-request or per-session, changes which mechanism pays off, and neither question can be settled by inspection.

Key idea. Instrument ReAct-style agents end to end across five benchmarks, collecting traces from both the model-serving and tool-execution sides rather than measuring isolated model calls.

Findings. With effective context caching, most input tokens are reused across turns, making execution decode-dominated and dependent on long-lived KV-cache state. Tool use also follows a temporal pattern, shifting from read and explore operations early in a trajectory to execute and write operations later.

Why it matters. Each finding retargets a design decision. Decode-dominated execution moves the bottleneck to memory bandwidth, long-lived session state makes cache retention a first-class scheduling concern, and the phase structure of tool use is something a scheduler can anticipate.

Background. Serving research measures one model call at a time: prefill a prompt, decode a response, report time to first token and throughput. Agentic applications interleave those calls with tools, sandboxes, and retrieval.

Problem. The assumptions that hold for single-call inference do not survive that interleaving, and until an agent is instrumented end to end you cannot say which of them break, or by how much.

Key idea. AgentSysBench combines ten representative agentic applications with unified end-to-end instrumentation and production traces, then follows each measured bottleneck to a serving-system intervention.

Findings. Non-LLM components dominate latency in 5 of 10 applications, sandbox working sets peak at 28 GB per session, and component latencies diverge by as much as 32×. Four design explorations reduce task latency by 29-40%, improve communication-aware placement by as much as 4.5×, reduce state memory by 4.6×, and eliminate 35.2% of redundant search calls.

Why it matters. Each property invalidates an assumption you would otherwise carry over from request serving, so it tells you which parts of the stack are worth redesigning. Read it for the four design explorations at the end, which are the agenda this meeting works through.

Background. Serving research has mostly measured chat traffic: short prompts, long outputs, and requests independent of one another. The cache and scheduling policies in production stacks were tuned against that shape.

Problem. A coding agent emits none of it. It runs long autonomous loops, sends long contexts for short outputs, and issues heavy-tailed tool calls, so a policy tuned on chat traffic is tuned on the wrong distribution.

Key idea. Release a trace of roughly 4,300 Claude Code and Codex sessions containing about 350,000 model steps and 430,000 tool calls, together with the collection pipeline and analysis code.

Findings. Coding-agent sessions have long autonomous loops, long contexts paired with short outputs, diverse heavy-tailed tool calls, and prefix-cache hit rates that are high but imperfect.

Why it matters. It is an artifact rather than an argument. You can drive your own simulation and cache-policy experiments against real agent sessions, and the imperfect part of that hit rate is exactly where a cache policy still has room to work.

Background. A serving engine caches KV blocks by prefix and evicts them under memory pressure. vLLM's automatic prefix caching evicts in least-recently-used order, which assumes a request's blocks stop being useful once the request goes quiet.

Problem. A coding-agent session goes quiet on every tool call and returns needing the same prefix, so recency-ordered eviction discards blocks that are about to be reused. Long-running sessions hold sustained KV pressure that conventional policies mishandle.

Key idea. Measure coding-agent KV reuse first, then design around the measurement: prefix-aware scheduling plus reuse-aware eviction, driven by lightweight predictions from tool-call metadata, implemented in vLLM.

Findings. On the collected coding-agent traces, CacheWise reduces KV-cache evictions by 2-2.6× and improves total agent-session completion time by as much as 3.5× over conventional cache management.

Why it matters. It is the concrete case that eviction order should follow predicted reuse rather than recency, and it shows the prediction can come from metadata already attached to the request instead of from the model.

Background. A GPU server is provisioned as a uniform unit, with a fixed ratio of CPU to accelerator sized for a workload that keeps the accelerator busy. Inference deployments have been analyzed on that assumption.

Problem. An agentic request fragments into LLM calls, tools, and orchestration that repeatedly cross the CPU-GPU boundary. That puts the CPU on the critical path and strands both CPU and GPU capacity under bursty load.

Key idea. Characterize agentic workflows from Microsoft Azure production data and open-source frameworks, attributing time and stranded capacity to workflow fragments rather than to the model call alone.

Findings. The study finds that host-side orchestration and tools put the CPU on the critical path, bursty execution strands both CPU and GPU capacity, homogeneous CPU provisioning mismatches heterogeneous software roles, and core sharing degrades locality. Agora improves utilization and server throughput while preserving agent tail latency.

Why it matters. It gives you the vocabulary to separate inefficiency a scheduler can fix from inefficiency built into a uniform server. The second kind is a provisioning question, and no amount of batch tuning answers it.

Background. Agent applications are nondeterministic programs whose later model calls depend on earlier generations and tool results. A conventional serving benchmark can replay a fixed list of prompts, but that list no longer describes the application that produced it.

Problem. Comparing serving systems requires a repeatable workload without erasing the dependencies, branching, and timing that make agent traffic different from chat. Re-running the live application changes the trace, while replaying requests independently changes the workload.

Key idea. XPerf collects, synthesizes, and replays traces of nondeterministic agent applications while preserving the structure needed to exercise a serving stack. The release includes eight applications and supports controlled experiments from recorded executions.

Findings. The evaluation shows that its replay is accurate enough to support comparative profiling, and demonstrates the framework as a tool for debugging serving behavior and studying scaling under agentic workloads.

Why it matters. It connects characterization to experimentation. The traces in this group tell you what production-like agent traffic looks like; XPerf supplies a way to reproduce that traffic closely enough to compare systems and explain where their performance diverges.

Background. Disaggregated serving puts prefill and decode on separate machines so each phase gets hardware matched to its bottleneck, prefill compute and decode memory bandwidth. A further split, attention apart from FFN, has been proposed on the same reasoning.

Problem. Whether a split pays depends on the traffic, and finding out normally means building each configuration and measuring it. Agentic requests are not chat requests: context accumulates across turns, so the ratio of prefill work to decode work moves.

Key idea. Simulate prefill-decode and four-way prefill-decode-attention-FFN specialization across quantization, parallelization, and heterogeneous accelerator choices, then compare them with unified execution.

Findings. The simulations project up to 75% higher throughput from prefill-decode disaggregation than unified serving on current GPUs. With custom NPUs, the four-way split is the most consistently beneficial configuration across the evaluated models.

Why it matters. The result makes hardware specialization conditional on both workload and device assumptions. It shows why an architecture cannot be evaluated apart from its traffic, but also why a projection for custom NPUs should not be read as measured production performance.

Background. By 2024 the strongest results on many tasks came not from one model call but from an assembly: a retriever, several model calls, tool invocations, and control logic holding them together. The practice had run ahead of any vocabulary for it.

Problem. Treating the model as the system leaves the interesting engineering invisible. If the only unit you measure and tune is a single forward pass, you cannot reason about a pipeline whose quality, latency, and cost are set by how the calls are composed.

Key idea. Name the unit of design. A compound AI system has models as components, and includes retrieval, tools, and control flow; the argument is that recent gains come from that composition, and that design, optimization, and operation therefore belong at the system level rather than the model level.

Why it matters. It supplies the system boundary that Parrot and SGLang then make explicit and optimizable. Once the unit is the program rather than the call, a scheduler can be shown the dependencies between calls, and DSPy, LangChain, and LangGraph are all frameworks built at that boundary.

Background. An LLM application is rarely one call. A chat agent, a map-reduce summarizer, and a multi-agent pipeline all issue many calls whose inputs depend on earlier outputs, and each call reaches the serving system as an independent request.

Problem. The public completion API hides that dependency structure. A scheduler seeing only isolated requests can optimize only per-request latency, which is the wrong objective: it will finish an intermediate call quickly while the application waits on a chain the scheduler cannot see.

Key idea. Semantic Variables annotate a request's inputs and outputs, so the runtime can reconstruct the dependency graph across the calls of one application and schedule against it, optimizing end-to-end latency instead of per-request latency.

Findings. Across the evaluated application patterns, Parrot improves end-to-end performance by as much as an order of magnitude over request-level serving baselines.

Why it matters. It names the interface question the whole agent-serving block turns on: how much structure must an application declare before a serving system can do anything useful with it? Once the graph is visible, prefix sharing, batching, and priority all have a target other than the individual request.

Background. Serving engines built for chat treat each request as independent and try to recover reuse after the fact, through a prefix cache that hopes the next arrival shares a prefix with something still resident.

Problem. Treating structured agent traffic as generic requests leaves the engine unable to anticipate reuse, long-context contention, or changes in demand across workflow stages.

Key idea. Agent workflows are predictable, so let the application say what it is about to do. It exposes workflow semantics through a small interface at the serving layer, and the scheduler exploits that declaration instead of inferring structure from the request stream.

Findings. Production traces from an agent-serving platform and an internal coding assistant show low prefix-cache hit rates, severe contention from long-context requests, and substantial queuing from suboptimal scaling. Pythia improves throughput and job completion time over the evaluated serving baselines.

Why it matters. Read it against Parrot for how thin a declaration interface can get and still pay for itself. It also supplies the measurement the rest of this block assumes: the cross-call reuse a chat-era prefix cache counts on is not there in agent traces.

Background. An agentic application runs between two layers it does not control: a framework above that expresses the workflow, and one or more serving engines below that execute individual calls. Which model and which backend handles each step is normally left to static configuration.

Problem. Those choices interact across the workflow and change while it runs. The right model for one node depends on what later nodes will need, the right backend depends on load that shifts between steps, and re-solving the assignment on every call puts an optimizer in the request path.

Key idea. Dyserve inserts a layer between frameworks and engines and solves one integer linear program per workflow, picking each DAG node's model, verifier, and backend over a heterogeneous pool. That program is pre-solved at several pressure levels at admission, so a load shift can redirect the workflow's uncommitted suffix without re-running the solver.

Findings. Across LiveCodeBench, GAIA, ComplexFuncBench, and SWE-bench, Dyserve scores 3-10 accuracy points above the strongest compared baseline while reducing latency by 1.1-6.8×. Its admission compilation takes less than 60 ms per request at p95.

Why it matters. Read it for how much structure the application must expose to make that solve possible. It also shows what pre-computation buys: the expensive decision happens once at admission, and the runtime only chooses among answers already computed.

Background. Software-defined networking separated a network's control plane from its data plane, so routing policy became a program running against a global view instead of a configuration file on each switch. Agentic serving is arranged the other way, through static parameters fixed per deployment.

Problem. A fixed configuration cannot answer questions whose answers change continuously: where to send this call, which instance already holds the state it needs, how much of the interconnect its transfers deserve. The system has the runtime state to decide and no place to put the policy.

Key idea. The proposal is that agentic serving be programmable and system-aware rather than statically parameterized. This short position paper sketches an SDN-inspired framework whose communication attributes are adjusted from runtime state instead of set at deployment time.

Why it matters. It is worth reading for whether the SDN analogy holds when the “packets” carry gigabytes of KV state. Read it as an argument about where serving policy belongs rather than as a system to evaluate.

Background. Emerging LLM applications need behaviors such as custom control flow, state handling, and coordination around inference. A fixed completion interface keeps those behaviors outside the serving engine, where they cannot participate in its scheduling and batching decisions.

Problem. Adding every new application pattern directly to a serving system does not scale, but exposing the engine's internals to application code would compromise isolation and portability. The interface needs to be programmable without turning each feature into an engine fork.

Key idea. Pie exposes fine-grained event handlers controlled by WebAssembly inferlets. Application-specific serving logic runs through those handlers while the engine retains control over execution and resources.

Findings. Pie reports 3-12% latency overhead for ordinary completion requests and 1.3-3.4× higher throughput on the evaluated agentic workflows.

Why it matters. It offers a concrete answer to how much programmability the serving layer should expose. Read it against declarative workflow interfaces: Pie lets applications bring behavior into the engine, and WebAssembly is the boundary intended to keep that behavior manageable.

Background. Agent code in Python naturally expresses dependencies by assigning the result of one step and passing it to the next. A serving runtime instead needs those dependencies before values exist if it is to schedule the workflow globally.

Problem. Requiring developers to rewrite an application as a separate DAG duplicates its control logic, while intercepting isolated model calls reveals dependencies too late. Context and mutable state also have to survive across the calls the runtime reorders.

Key idea. Nalar uses Python stubs that return dependency- and context-carrying futures, allowing ordinary-looking application code to reveal work before it completes. The runtime manages state and combines global control with local decisions as the workflow unfolds.

Findings. Across the tested request rates, Nalar reports 34-74% reductions in tail latency and up to 2.9× speedup on the evaluated workflows.

Why it matters. It is an interface design for partial disclosure: the program exposes enough structure to schedule ahead without becoming a static graph. That makes it a useful comparison point for both declarative workflow systems and lower-level programmable serving hooks.

Background. A cloud agent workflow combines model calls, tools, and other services whose resource needs differ by stage and shift with load. Application code commonly mixes the workflow's logical structure with deployment and resource configuration.

Problem. A configuration chosen before deployment cannot remain efficient as stage costs and queueing change, while a purely reactive runtime lacks the profile needed to choose good placements. Coupling configuration to code also makes retuning the workflow invasive.

Key idea. Murakkab separates a declarative workflow specification from its configuration, then combines a profile-guided optimizer with an adaptive runtime. The optimizer establishes an efficient plan and the runtime adjusts execution as conditions change.

Findings. Across the evaluated workflows, Murakkab reports reductions of up to 2.8× in GPU use, 3.7× in energy, and 4.3× in cost while maintaining the tested SLOs.

Why it matters. It spans the boundary between offline planning and online control. The separation of specification from configuration makes the workflow an object the platform can optimize, while adaptation acknowledges that no profile remains exact once the system is running.

Background. A serving engine queues individual requests. An agent program is not one request: it is a sequence of LLM calls interleaved with tool calls, and the engine sees only the calls, one at a time, with no record that they belong together.

Problem. Scheduling each call on its own arrival lets a newly submitted request cut ahead of the next call of a program that has already run for minutes. The program waits behind work that started later, and head-of-line blocking is paid at program granularity, where it is far more expensive than at request granularity.

Key idea. Treat the agent program, not the request, as the scheduling unit. Track how much work each program has already done and order calls by what that program still needs, so the queue reflects program progress rather than request arrival order.

Findings. Across the evaluated models and agent workloads, Agentix sustains 4-15× more completed programs than vLLM at the same end-to-end latency.

Why it matters. It changes the objective a serving engine is optimizing. Once the unit is a program, per-request latency stops being the right target and program completion time takes over, which is the metric a user of an agent actually experiences.

Background. An agent workflow spans many LLM calls, and on a GPU cluster those calls land on whatever instance the router picks. The cluster scheduler places calls; nothing above it knows the calls form one workflow.

Problem. Placing calls independently spreads a single workflow across instances and interleaves it with others, so cached state is scattered and the workflow finishes only when its slowest fragment does. Optimizing each placement locally does not optimize the workflow.

Key idea. Workflow-atomic scheduling: admit and place the whole agent workflow as one unit on the cluster, so the scheduler commits resources to a workflow rather than to its individual calls.

Findings. On 64 GPUs running SWE-bench and WebArena agents, SAGA reduces task completion time by 1.64× over vLLM with prefix caching and affinity routing, improves GPU-memory utilization by 1.22×, and reaches 99.2% SLO attainment. Its latency gains cost about 30% of peak throughput relative to throughput-optimal batching.

Why it matters. It lifts the program-as-scheduling-unit argument from a single engine to a cluster, where the scarce resource is instances and KV capacity rather than a single GPU's batch slots. That is where the gang-scheduling and admission-control tradeoffs become visible.

Background. Serving metrics were defined for humans reading streamed output, so time to first token and inter-token latency drive the scheduler. Agents consume responses programmatically, and they issue many requests per session.

Problem. Under that objective, cache-aware routing looks wrong: sending a session's requests to the instance holding its prefix overloads a few instances while others idle. Load balance and KV reuse pull against each other, and per-token latency does not say which should win.

Key idea. Take cluster throughput as the objective, then balance only each session's first request and route the follow-ups cache-aware. The tradeoff being managed is reuse against load balance, not latency against fairness.

Findings. Relative to state-of-the-art schedulers, SMetric improves cluster throughput by 10-16% for colocated prefill and decode with a global cache, and improves prefill throughput by 2-34% under disaggregation while also lowering per-token latency.

Why it matters. It shows that a metric chosen for one consumer misprices a mechanism for another. Once the reader is an agent, deliberate imbalance is the right call, and that is a general lesson about which objective a scheduler is allowed to optimize.

Background. Most scheduling proposals for agent serving need to know something about the future: how many tokens a response will take, how many turns remain, whether a session is finished. They obtain it by prediction.

Problem. A prediction that is wrong costs more than having no prediction at all, because the scheduler commits placement and memory on it. Agent behavior depends on tool outputs and on the model's own choices, so the quantities being predicted are exactly the ones that are hard to predict.

Key idea. Refuse to predict. Schedule on observed conversation state instead, at conversation granularity across disaggregated instances, so decisions rest on what has already happened rather than on an estimate of what comes next.

Findings. Against a per-turn prediction baseline, ConServe reduces p95 time to first effective token by 51.08% and improves energy efficiency by 7.51% while preserving last-turn time between tokens and SLO attainment. Heterogeneous GPU tiers improve energy efficiency by a further 22.75%.

Why it matters. It is the robustness argument against the prediction-based approaches that dominate this section, and it makes the cost of a mispredict explicit. Reading it alongside the predictive schedulers tells you what each one is actually betting on.

Background. Multi-agent applications send every agent's call to one shared model endpoint. The serving stack underneath admits and batches those calls one request at a time, and a public cloud deployment is sized for average load rather than peak.

Problem. Under overload the queue decides end-to-end latency, and per-request scheduling is blind to the workflow that issued the request. One queued call gates an entire workflow, because the agents downstream of it cannot start until it returns.

Key idea. Schedule with the workflow in view. Kairos orders requests by each agent's latency characteristics and dispatches by memory demand, so calls a workflow is waiting on are not left behind calls that nothing depends on.

Findings. In the paper's experiments, Kairos reduces end-to-end latency by 17.8-28.4% against the evaluated multi-agent serving baselines.

Why it matters. It argues the scheduling unit should be the workflow, not the request. Once one model serves many agents under load, queue discipline becomes the dominant latency term, and the information needed to choose a good one lives above the serving layer.

Background. A multi-agent workflow is a sequence of stages, and those stages call different models spread over more than one cluster. The scheduler sees each call at admission, with no knowledge of what the call is about to do.

Problem. Without a stage's output length or memory footprint you cannot place it well. Two costs follow: loading a model's weights on demand adds a cold start, and a long stage in front of a short one blocks the interactive tasks behind it.

Key idea. Predict each stage's output length and memory footprint, then spend the predictions at three levels: co-locate several models on a node by caching weights, route across clusters to dodge cold starts, and prioritize globally by workflow to limit head-of-line blocking.

Findings. Across prototype experiments and trace-driven simulations, Maestro reduces reserved KV-cache HBM by 67.2% and improves high-contention SLO attainment over earliest-deadline-first scheduling by 23.6 percentage points.

Why it matters. The useful scheduling input turns out to be a prediction about the stage rather than a measurement of the request. Read it for what workload-awareness costs as well, because all three levels inherit whatever the predictor gets wrong.

Background. A database does not execute queries one at a time as written. It compiles each into a plan and then optimizes across concurrent plans so shared work runs once. An agent workflow is also a graph of operations, and agent frameworks execute it one call at a time.

Problem. Concurrent workflows overlap heavily, sharing prompts, retrieved context, and whole subgraphs, and a stack that sees only independent calls repeats that shared computation once per workflow.

Key idea. Halo compiles each agent workflow into a query-plan DAG and consolidates concurrent workflows into one graph, so shared computation runs once. A cost model over prefill and decode cost, cache reuse, and GPU placement drives optimization at the level of the plan rather than the individual call.

Findings. Across six benchmarks, Halo speeds up batch inference by as much as 3.6× and improves online-serving throughput by as much as 2.6× while preserving output quality, including workloads with thousands of queries and complex graphs.

Why it matters. It is the cleanest statement of the database view of agent serving. If a workflow is a query, then serving many of them is multi-query optimization, and the vocabulary of plans, cost models, and shared subexpressions transfers to a problem the serving literature has been solving without it.

Background. An LLM application is a pipeline: embed the query, search an index, rerank, prompt the model, parse the output, call a tool, prompt again. Each stage is tuned by whoever owns it, and the serving engine sees only the model calls.

Problem. Optimizing the model call alone leaves end-to-end latency roughly where it was. The engine cannot see what runs before or after it and the orchestration layer cannot see inside the engine, so stages that could overlap run in sequence, and work that could be shared across requests is not.

Key idea. Express the whole application as a dataflow graph of primitive operations rather than opaque service calls, then optimize across that graph: run independent branches in parallel, batch equivalent primitives from different requests together, and let a downstream stage start on partial output from the stage above it.

Findings. Across several representative LLM applications, Ayo achieves as much as a 2.09× end-to-end speedup over existing orchestration systems.

Why it matters. It defines the object the rest of this group operates on. Once the application is a graph, serving becomes query planning and the question shifts from how fast one call runs to what the planner is allowed to move. It also relocates the target: what a user waits for is the application rather than the model call, and the largest remaining latency is often in the parts that are not the model at all.

Background. Agentic workflows arrive at a serving engine as a stream of independent calls. Data systems have long handled a related shape, a declarative program compiled and planned before execution.

Problem. A serving engine plans one call at a time, so it cannot reuse intermediate results across a workflow, cannot reorder stages, and cannot choose an execution strategy from the workflow's structure. The information a planner would need never reaches it.

Key idea. Read agentic serving as a data systems problem: describe the workflow, then plan and execute it the way a query engine would, with the workflow rather than the call as the unit of optimization.

Findings. Across the evaluated workloads, Helium's proactive caching and cache-aware scheduling deliver as much as a 1.56× speedup over the compared agent-serving systems.

Why it matters. The framing imports decades of planning, caching, and cost-model machinery into serving, and it makes explicit what today's engines give up by scheduling greedily on arrival.

Background. Production clusters now split prefill and decode onto separate, often unlike, hardware, and transfer KV cache between them. An agent workflow is a DAG whose next node is revealed only after the current call returns.

Problem. A scheduler that sees one call at a time cannot tell which ready call is on the critical path of its workflow, and on heterogeneous prefill-decode disaggregated hardware it must also choose where each half runs under KV capacity limits and cross-stage transfer cost. Local choices miss the workflow's deadline.

Key idea. Schedule the online-revealed DAG directly: rank ready calls by projected risk of missing the workflow's completion horizon, then jointly pick prefill placement, decode placement, and queue priority. The objective is workflow-level latency, not per-call latency.

Findings. Across agent workloads on heterogeneous A100, H100, and H200 clusters, HexAGenT lowers the SLO scale needed for timely completion by 20.1% on average at 95% attainment and by 33.0% at 99% attainment.

Why it matters. It joins two decisions the field usually treats separately, what to run next and where to run it, and shows that heterogeneity and workflow structure have to be resolved together rather than in sequence.

Background. Multi-agent workflows repeatedly invoke specialized agents whose fixed role prompts create reusable prefixes. Prefix caching can preserve the corresponding KV tensors instead of recomputing them on every invocation.

Problem. Least-recently-used eviction sees only past accesses. It may discard an agent's cached prefix immediately before the workflow invokes that agent again, causing recomputation or a host-to-device reload even when the workflow exposes the future execution order.

Key idea. Represent the workflow as an Agent Step Graph and assign each agent a steps-to-execution value that estimates when it will run again. Use that value for fine-grained KV-node eviction, and prefetch the next step's tensors from CPU memory while the current step executes.

Findings. Relative to SGLang with its hierarchical radix cache, KVFlow speeds up a single workflow with large prompts by as much as 1.83× and workloads with many concurrent workflows by as much as 2.19×.

Why it matters. KVFlow shows why agent structure belongs in the cache manager as well as the scheduler. When the workflow reveals near-future reuse, recency is an avoidable approximation, and prefetching can turn the same structural signal into overlap rather than a generation stall.

Background. Reinforcement learning on coding agents needs rollouts, and every rollout runs the agent against a real repository inside an isolated execution environment. That environment is drawn from four substrates in practice: single containers, hosted sandboxes, Kubernetes, and cloud VMs.

Problem. The substrate is usually picked for convenience, on the assumption that its cost is small next to the GPUs. Nobody had measured what the four actually cost on the same rollout workload.

Key idea. Compare single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines on the same coding-agent rollout workload, then project each substrate's overhead at large training scale.

Findings. Cold-start latency differs by as much as 110× across the four substrates, and projected worker-hours differ by 1.8× for one million 150-step trajectories. The projection covers rollout workers only and excludes GPU inference and optimizer steps.

Why it matters. It puts a number on infrastructure that the agentic-RL papers next to it treat as free. A 1.8× spread in worker-hours makes the substrate a first-order term in an RL training budget rather than a deployment detail.

Background. An agentic RL step generates a batch of rollouts and then trains on them. Each rollout drives an agent through tool calls and environment steps, so the batch is finished only when its slowest trajectory is.

Problem. Trajectory length is data dependent and long tailed, so trainer utilization is set by the tail rather than the mean. Waiting for the straggler idles the trainer; the trainer has work available, just not the work its batch is waiting on.

Key idea. Schedule rollouts by readiness, so the trainer consumes trajectories as they become available instead of waiting for a batch boundary. On-policy staleness is the constraint that bounds it: reordering may only go as far as the training data can drift from the policy that generated it.

Findings. Across text-only and multimodal agent workloads, TideRL improves training goodput by as much as 5.6× over synchronous baselines and by more than 33% over asynchronous baselines at similar task performance. It also cuts per-step training time by as much as 44.3% and waiting time by as much as 77.6%.

Why it matters. This is the stall-hiding logic of agent serving applied to training, and the tradeoff is explicit. The more reordering you allow, the busier the trainer stays and the staler the data it trains on.

Background. Agentic RL post-training alternates two unlike jobs on one cluster: generation, which is inference over many concurrent trajectories, and training, which is a synchronous gradient step. Frameworks divide the GPUs between them.

Problem. That division is fixed when the job launches, and the demand behind it is not. Agent trajectories are long tailed, so how long generation holds its share varies from step to step, and a static partition leaves one side of it idle.

Key idea. Reallocate an elastic GPU pool between rollout and training while a causality-driven scheduler assigns trajectories to heterogeneous rollout buckets from tool-return signals rather than predicted sequence lengths.

Findings. On 48 A800 GPUs, Libra delivers as much as 3.0× higher throughput than the evaluated resource-management baselines and reaches a given reward as much as 2.5× sooner.

Why it matters. It names the resource-management question hiding inside every RL post-training stack, and it locates the cost of the static answer in the shape of the workload rather than in model size.

Background. RL trainers generate a prescribed mixture of rollout workloads whose sessions differ in model behavior, context growth, and tool gaps. Serving them efficiently requires reuse of KV state without changing the data mixture the trainer expects.

Problem. Prefix-local scheduling alone groups similar prompts but can admit too many long-lived sessions, exhausting KV capacity and delaying other rollout classes. Reordering for efficiency can also distort the trainer's requested mixture.

Key idea. MISA-T adapts session admission to current pressure and assigns KV capacity by workload, accounting for residency across each session while preserving the trainer's rollout mixture.

Findings. In a matched 50-iteration Step3.7 run, MISA-T reports 35.6% higher rollout throughput and 22.8% lower mean iteration time.

Why it matters. It shows why rollout scheduling cannot stop at prefix reuse. The scheduler must control how many stateful sessions enter the system and how their KV footprints compete, while respecting a statistical contract with training rather than optimizing an unconstrained request stream.

Background. Real LLM use is rarely one prompt and one completion. Few-shot prompts, chat turns, agent loops, and tree search all reissue overlapping token sequences, and serving systems of the time freed a request's KV cache the moment it finished.

Problem. Every request that shares a prefix recomputes it, and nothing in a plain completion API tells the runtime which spans will come back. The runtime has to discover the sharing itself, then decide what to keep under a fixed memory budget.

Key idea. Index the KV cache with a radix tree over token sequences, match each arriving request against its longest cached prefix, and evict with LRU. The frontend language makes a program's structure explicit, so the scheduler can also order requests to land on the cache instead of missing it.

Findings. Across agent, reasoning, few-shot, structured-output, retrieval, and chat workloads, SGLang delivers up to 6.4× more throughput than the evaluated state-of-the-art inference systems.

Why it matters. It turns prefix reuse into ordinary cache management, with a hit ratio, an eviction policy, and a scheduling decision attached, which is the framing the rest of this meeting works in.

Adoption. xAI and LinkedIn use SGLang, providing external production uptake of RadixAttention's automatic prefix reuse.

Background. The radix tree above indexes prefixes inside one engine's GPU memory. A chat deployment is where those prefixes come from, because every turn of a conversation resends the whole history as the prefix of the next request.

Problem. GPU memory cannot hold every open conversation, so evicted state is recomputed on the next turn. A lower storage tier helps only if fetching beats prefill, and context truncation can invalidate position-dependent cached state.

Key idea. Keep KV state in a storage hierarchy. Preload by layer and save asynchronously to overlap transfers with computation, use scheduler hints for placement, and separate positional encoding from stored state so truncation does not invalidate it.

Findings. Against recomputing the history KV on every turn, time to first token falls by up to 87%, prompt prefilling throughput rises by up to 7.8×, and end-to-end inference cost falls by up to 70% on multi-turn conversations.

Why it matters. Multi-turn state turns the prefix cache into a storage hierarchy rather than a GPU table. Reuse also requires cached bytes to remain valid at their new positions, a constraint that later systems must address.

Background. Prefix caching assumes cached state is indexed by token: keys and values for position i sit at slot i, so a shared prefix is a shared range of blocks. Hybrid models interleave attention layers with state-space layers.

Problem. A state-space layer carries one recurrent state that has already absorbed every token it saw, with no per-token entry to address or split. There is no prefix of it to look up, so block-level reuse and eviction have nothing to operate on.

Key idea. Decide where in a sequence recurrent state is worth checkpointing, then admit and evict knowing that hybrid entries differ both in size and in how much recomputation they save. Prefix caching becomes a policy question about state rather than a lookup over token blocks.

Findings. Across the evaluated hybrid models and workloads, Marconi raises token hit rate by up to 34.4× over prior prefix caches and reduces time to first token by up to 71.1%, or 617 ms.

Why it matters. It shows how much of prefix caching was an artifact of attention's data layout. Once cached state is not addressable by token, you reason about recompute saved per byte held instead of counting block hits, which is the framing any non-attention architecture needs.

Adoption. vLLM directly identifies its selective retention policy for hybrid caches as Marconi-style caching and uses the policy in its Kimi K3 serving path.

Background. A chat deployment's prefix cache holds one entry per conversation, and the entry is worth keeping only until the user stops replying. Every engine decides that with LRU, the policy SGLang's radix tree above uses.

Problem. LRU ranks entries by how recently the last turn arrived, which is a weak proxy for whether another turn is coming, and the gap to the offline optimum is large. The signal that would close it, whether this particular conversation continues, is not present in access timestamps at all.

Key idea. Predict continuation from the conversation itself. Train a predictor over conversational content to estimate which conversations will receive another turn, then combine that estimate with the last access time in the eviction decision. Eviction becomes a prediction problem whose features are the cached text rather than only its access history.

Findings. Across three real-world datasets, LPC reaches the hit ratios LRU reaches with caches 18-47% smaller, and raises LLM prefilling throughput by 11% in an emulated environment.

Why it matters. It is prefix caching's version of learned eviction, and the feature it uses is what makes it worth reading. A CDN policy can learn only from the request stream, whereas an LLM cache can read the object it is holding, so content becomes a predictor of future access. Read it against LRB in the eviction policies group below, which frames eviction the same way and pays for it with feature collection and inference on the critical path.

Background. Prefill and decode have different bottlenecks, which is why serving stacks began splitting them onto separate instances, and the KV cache prefill produces is worth keeping for later requests. What a deployment that commits to both looks like under real traffic is a separate question.

Problem. At production scale the scarce resource is KV cache capacity and the bandwidth to move it, not weights or FLOPs. Overload is routine rather than exceptional, so a design that assumes every arriving request is admitted plans for the wrong workload.

Key idea. Put the cache at the center. Disaggregate prefill from decode, pool KV across the cluster's underused CPU memory and storage so reuse is cluster-wide instead of per-node, and reject requests early under overload instead of admitting everything and missing latency targets.

Findings. Mooncake raises SLO-compliant throughput by up to 525% over the simulated baseline. Under Moonshot's real traffic, the deployed architecture lets Kimi handle 75% more requests.

Why it matters. It reports what a KV-centric design costs at scale, including the two parts a prototype leaves out: transfer bandwidth and admission control. Read it as the deployment counterpart to the indexing and scheduling mechanisms in this meeting.

Adoption. Mooncake's transfer engine ships as a connector in vLLM's KV-transfer interface.

Background. Two things move KV around a deployment. Disaggregated serving splits prefill and decode onto separate instances, so KV must cross the network, and context caching keeps KV across requests, so it must outlive the request that produced it.

Problem. When each instance owns its own KV memory, capacity is stranded wherever it was allocated and a cached context can be neither shared nor migrated. A request that could have reused a warm cache on a neighboring instance prefills from scratch.

Key idea. Expose one placement and lookup interface over an elastic memory pool spanning instances. Cached context becomes a shareable resource instead of private state inside the instance that computed it.

Why it matters. It separates two things usually conflated: where computation happens and where its cached state lives. Once the cache is a pool, disaggregation and cross-request reuse stop being two features and become one mechanism.

Background. A serving engine keeps its KV cache in its own GPU memory. That cache dies with the engine process, and nothing is shared between vLLM and SGLang, between replicas of one model, or across queries that arrive minutes apart.

Problem. Enterprise deployments hold far more reusable state than one GPU fits and run more engines than one engine-local cache can serve. The work then lands in data movement across GPU, CPU, storage, and network, and prefill-decode disaggregation needs that transfer anyway.

Key idea. Make the KV cache a layer under the engine. LMCache plugs beneath both vLLM and SGLang, spans GPU, CPU, storage, and network, pipelines data movement so transfers overlap compute, and carries KV across a prefill-decode split.

Why it matters. Worth reading for what enterprise deployment breaks, notably that context truncation halves the prefix cache hit rate. It treats cache capacity as a memory-hierarchy problem rather than an engine feature, which puts the design work in the interface between the two.

Adoption. LMCache ships as a KV connector in vLLM's production stack, where it backs the CPU, disk, Redis, and Mooncake offload tiers of a Kubernetes deployment.

Background. Each serving replica holds its own prefix cache, so the router sends a request to whichever instance already has its prefix. Cache-aware routing of that kind is what production routers do today.

Problem. Instance-granularity caching ties placement to the cache. Hot prefixes pin requests to one replica, so load follows the cache instead of the compute, and a long context cached on a saturated instance is unusable to an idle one.

Key idea. Pool the prefix cache at segment rather than instance granularity, behind a declarative cache interface, and balance load with knowledge of which segments are the heavy hitters. Any instance can reach any segment, so hot prefixes stop pinning requests.

Findings. On real workloads, TokenLake raises throughput by up to 2.6× over cache-aware routing and 2.0× over cache-centric prefill-decode disaggregation. Cache hit rate improves by up to 2.0× and 2.1×, respectively.

Why it matters. Read against cache-aware routing and cache-centric prefill-decode disaggregation, which are its two baselines. It separates where cached state lives from where compute happens, and that separation is what makes capacity elastic rather than per-instance.

Background. Prefix caching reuses a span only when it begins at token zero and matches exactly, which covers a system prompt at the front of every request and little else.

Problem. The text that actually recurs is often not a prefix: a document in the middle of a prompt, one tool description among several, a snippet whose position shifts between calls. Keys and values depend on position, so state cached at one offset is wrong at another.

Key idea. Declare the reusable spans as prompt modules in a schema, with their position ids fixed in advance. Each module's attention state is computed once and stays valid wherever the module lands in the assembled prompt, so reuse no longer has to run contiguously from the start.

Findings. The prototype reduces time to first token by up to 8× on GPU and 60× on CPU for long-prompt workloads, while preserving the reported output accuracy.

Why it matters. It moves reuse from something the runtime infers to something the application states, and it identifies position dependence as the obstacle to reusing anything other than a prefix.

Background. Prefix caching lets a server skip prefill for context it has already processed, but the saving only lands if the cache is reachable from the GPU that needs it. In a multi-node deployment the KV for a long context sits on another machine or on disk.

Problem. That KV is large. Fetching it across a link can take longer than recomputing prefill from scratch, so reuse becomes a bandwidth problem rather than a memory problem, and the number that settles it is time to first token rather than bytes stored. Available bandwidth also moves under load.

Key idea. Treat the cached tensors as something to encode rather than merely to store. Compress them into compact bitstreams that exploit the structure they already have, spend fewer bits where the model tolerates more error, and stream them at a compression level chosen for the bandwidth on hand, recomputing instead when even that would arrive too late.

Findings. Across four datasets and three models, CacheGen reduces time to first token by 3.1-4.7× against recomputing from text and by 3.2-3.7× against uniform KV quantization.

Why it matters. It puts a price on a cache hit. Across nodes and disks, a cache hit that arrives after recomputation would have finished is a miss.

Adoption. LMCache ships CacheGen-style KV encoding and streaming for its disk and remote cache tiers, which is how the vLLM production stack moves reused KV between nodes.

Background. Prefix caching hits only when a prompt starts with a cached prefix. A RAG prompt concatenates several retrieved chunks, and the same chunk appears at a different position, after different neighbors, in every query that retrieves it.

Problem. RAG chunks share no common prefix, so only the first chunk is ever a hit. Precomputed per-chunk KV is not directly reusable either, because it was computed without attention to the chunks placed before it, and splicing the pieces together degrades output quality.

Key idea. Reuse the precomputed KV for every chunk, then recompute selectively, only the small set of tokens whose KV deviates most from what a full prefill would have produced, so the fused cache stays close enough to be correct.

Findings. Across three models and four datasets, CacheBlend reduces time to first token by 2.2-3.3× and raises throughput by 2.8-5× over full recomputation without measured quality loss or additional storage.

Why it matters. It turns reuse from a yes-or-no correctness question into a knob you can turn, trading recompute against fidelity. That knob is what the rest of this group varies, and it is the reason non-prefix reuse is usable at all.

Adoption. LMCache ships CacheBlend as its non-prefix reuse path, which is how vLLM users get blended RAG chunk caches today.

Background. Everything above reuses KV that one model computed for that same model. A compound or agentic deployment runs several models specialized for different users, tasks, or roles, and they pass the same context between them, so the same prefix is prefilled once per model.

Problem. KV computed by one model is not KV for another. Two models sharing an architecture still hold different weights in every layer, so one model's cache is wrong in the other, and whether that error costs output quality had not been measured.

Key idea. Reuse most of the layers and recompute the rest. Study which layers' KV actually diverges between the two models, recompute only those, reuse the remainder as it stands, and pipeline the layer-wise recomputation against the loading of the reused layers so the two overlap. Reuse holds so long as the models share an architecture.

Findings. Up to 4× higher throughput and about 3.1× faster time to first token, against a baseline that shares nothing across models, with negligible loss in F1, Rouge-L, or code similarity across several datasets and model pairs.

Why it matters. It moves the selective-recompute knob to a second axis. CacheBlend recomputes the tokens a changed position invalidated, and DroidSpeak recomputes the layers a changed model invalidated, which is the axis a multi-model agent deployment sits on.

Background. Prefix caching hits only on an exact shared prefix. Position-independent caching relaxes that, reusing KV for segments that are not a shared prefix, which is what retrieval and agent prompts actually look like. Hybrid-attention models interleave full-attention layers with linear-attention layers that carry a per-request recurrent state.

Problem. Position-independent reuse breaks on hybrid-attention models because a per-token KV primitive has no counterpart in a per-request recurrent state. You can concatenate the cached KV of two segments; you cannot concatenate two recurrent states and get the state the full prompt would have produced.

Key idea. Cache each segment's cumulative transition operator alongside its zero-start end state, which composes independently cached segments in constant time, then repair cross-segment attention in the remaining full-attention layers with a small seam window.

Findings. Across four hybrid-attention models and five workloads, time to first token falls 3.25× on average and QPS rises 1.66× against prefix caching, and task quality stays within 1.71 points of full recomputation.

Why it matters. It says what a cache entry has to be for reuse to compose: an operator you can apply, not only bytes you can concatenate. That is the same checkpointing question Marconi raises above, asked where segments have to combine rather than merely be looked up.

Background. LRU has been the default eviction policy for decades, and the policies that beat it on hit ratio usually do so by tracking more metadata per object and by reordering the data structure on every access.

Problem. That bookkeeping costs throughput and lock contention, and most of it is spent on objects that never come back. Real cache workloads are dominated by one-hit wonders, objects requested exactly once, which LRU still admits to the full cache and promotes to the most-recently-used position.

Key idea. Use three FIFO queues. A small probationary queue quickly removes one-hit objects, a main queue holds objects accessed twice, and a ghost queue remembers evictions. Hits never reorder a queue.

Why it matters. It beats LRU on hit ratio and on throughput at the same time, where those two are normally traded against each other, and it is the clearest demonstration that one-hit wonders dominate real workloads. Read it before designing any prefix-cache eviction policy.

Adoption. S3-FIFO runs in production at Google, VMware, and Redpanda, and it is an eviction policy in open-source caches including Rust's foyer and Go's otter.

Background. Policies that improve on LRU usually arrive as new data structures: extra queues, frequency sketches, learned predictors. Adopting one means rewriting the cache, which is part of why so many production caches still run LRU.

Problem. Promotion on every hit is what makes LRU expensive, because it mutates shared list state on the read path. Dropping promotion normally costs the hit ratio that promotion buys, so simplicity and competitiveness look like opposite choices.

Key idea. Keep the FIFO queue and promote lazily. A hand walks the queue, evicting objects that have not been accessed since it last passed and clearing the bit on those that have, so retained objects never move. That is about ten lines of diff over FIFO.

Why it matters. A good model for how simple a competitive policy can be, and for the difference between eager and lazy promotion. What a policy costs is not only its miss ratio; it is also what adopting it does to the read path and to the code.

Adoption. SIEVE was merged into five production cache libraries with fewer than twenty lines of change each, and it now appears as a policy option in Go, Rust, and JavaScript caches.

Background. Belady's algorithm evicts the object whose next request is furthest in the future. It is optimal and it is offline, because it needs the future request sequence, so every deployed policy is a heuristic stand-in for it.

Problem. A heuristic encodes one fixed assumption about reuse, and a workload that violates the assumption loses hit ratio. Worse, the policy itself tells you nothing about how much of the offline optimum you are leaving on the table.

Key idea. Relax Belady. Rather than predict an exact next-access time, train a model on features collected online to predict which sampled candidates have their next request beyond a boundary, and evict from those. Eviction becomes a supervised prediction problem.

Why it matters. Read it for the framing of eviction as prediction, and for how much machinery that framing costs. Feature collection, training, and inference all land on the cache's critical path, which is the price of chasing the optimum.

Background. Cache policy discussions usually start and end with eviction: given a full cache, which object leaves. Admission, the decision of whether a newly requested object belongs in the cache at all, is the other half and is usually left implicit.

Problem. Frequency is the right signal for admission, and exact frequency counts over a long history cost more memory than the objects they protect. A cache that spends its space on counters has less space left for data.

Key idea. Approximate the history with a sketch. A few bits per entry hold an aging estimate of each key's recent frequency, and a candidate is admitted only if its estimated frequency beats the eviction victim's. Windowed TinyLFU puts a small LRU window in front so bursty new keys still get a chance.

Why it matters. Admission, not eviction, is often the lever, and a sketch is enough to make the decision. It is also the clean case of trading bounded accuracy for a large cut in metadata, which is the same trade a per-block prefix cache has to make.

Adoption. W-TinyLFU is the admission filter in Caffeine, which backs the caches in Cassandra, Solr, and Druid, and the design was carried into Go's Ristretto and Rust's moka.

Background. Cache research is comparative. A new policy is only interesting relative to LRU, FIFO, and the offline optimum on the same traces, and running that comparison from scratch means reimplementing every baseline and every trace format.

Problem. Reimplemented baselines are where hit-ratio claims go wrong. Two papers reporting LRU on the same trace can disagree because of object-size handling, admission, warmup, or trace parsing, and none of that disagreement is visible in the numbers they publish.

Key idea. Package standard baselines in one simulator with a library interface, so a new policy is a small addition to a shared harness rather than a standalone program. Every policy runs on the same trace reader.

Why it matters. It is the standard cache simulator for this kind of work, so read the docs before you write a policy. It produced the S3-FIFO and SIEVE evaluations above, and it replays the open Twitter, Meta, and Wikimedia traces that most published miss-ratio comparisons now use.

Background. A large web service runs many separate caches, in front of its CDN, its social graph, its key-value storage, and more. Each has its own workload and object type, and historically each team built its own cache.

Problem. Most of what a production cache does has nothing to do with the eviction policy. It has to size memory across pools, admit selectively to flash so the device is not written to death, and come back warm after a deploy. A team re-solving all of that solves some of it badly.

Key idea. Provide one embeddable library behind a common API for DRAM and flash tiers, flash admission, per-pool sizing, structured item types, and cache contents that survive process restarts.

Findings. The production study reports that one CacheLib server can replace tens of backend database servers, with 20× higher throughput and hit ratios above 80% for the cited workload.

Why it matters. It is the clearest account of what a production cache has to handle beyond the eviction policy: sizing, admission, flash, and warm restarts. Read it as the list of conditions a research policy has to survive contact with.