Background. A serving paper reports a speedup against a baseline its authors chose, on a workload they selected, in a metric they defined. A reader rarely has the configuration, arrival process, or latency distributions needed to check the claim.

Problem. Three properties of this workload make an evaluation easy to get wrong. First, prefill and decode are separate operations with separate costs, so a single latency figure averages two different things. Second, production request lengths are heterogeneous, so a fixed prompt length samples nothing. Third, the output arrives over time, so a statistic summarizing all tokens can hide a stall the user would notice. An unfair baseline compounds all three, because engineering effort in one implementation then reads as algorithmic novelty in the other.

Key idea. Audit the literature instead of proposing another system. The paper reads 12 recent serving systems for recurring anti-patterns along three dimensions, which are baseline fairness, evaluation setup, and metric design, then states the result as a checklist. A case study on speculative decoding makes the failure concrete, since bursty and non-uniform token generation is exactly what an average over tokens misreports.

Findings. Of the 12 systems surveyed, 2 satisfy the workload-diversity criterion and 1 avoids obscuring normalization. Speculative decoding emits roughly 75% of its tokens at near-zero time between tokens, and against an autoregressive baseline it lowers median time per output token by 1.3× while raising the p99 figure by 1.16×, so the median and the tail move in opposite directions. In one reported measurement 60% of requests wait more than 25 s in scheduling delay, and the normalized latency plot shows almost none of it.

Why it matters. The evaluation pitfalls covered in this meeting come from this paper, and each one is a question to ask of the speedups reported in the entries below: whether the baseline was tuned, which workload was replayed, and whether the reported metric would show a stall. The checklist applies to the numbers you report for an assignment benchmark exactly as it applies to a paper, and its speculative-decoding case study prepares the November 11 meeting.

Background. Serving systems before Orca batched at request granularity. A batch of requests entered the model together and left together, the way an image-classification server batches independent inputs.

Problem. Generative requests do not finish together, because their output lengths differ. A request-level batch runs until its longest member is done, so finished sequences keep occupying slots and newly arrived requests wait for the whole batch to drain.

Key idea. Schedule at the granularity of one iteration rather than one request. After every token the scheduler re-forms the batch, admitting arrivals and retiring completed sequences, which requires running attention per sequence while the linear layers stay batched.

Findings. On GPT-3 175B, Orca delivers 36.9× the throughput of NVIDIA FasterTransformer at the same latency under the paper's request traces.

Why it matters. Once the batch can change every token, throughput becomes a scheduling question rather than a kernel question. This is the baseline every later serving paper is measured against, and it is also a model of a well-argued systems evaluation.

Adoption. Continuous batching is the scheduler in vLLM, SGLang, TensorRT-LLM, and Hugging Face TGI.

Background. Iteration-level batching admits a new request's prefill into the running batch between decode steps. Prefill is one compute-bound pass over every prompt token; decode is one token per sequence and bound by memory bandwidth.

Problem. A prefill admitted alongside decodes stalls all of them for the length of its forward pass, so inter-token latency tracks the longest prompt in the queue. Scheduling prefill first protects time to first token at the cost of those stalls, and scheduling decode first protects time between tokens at the cost of queueing new requests, so batching for throughput and batching for latency conflict.

Key idea. Chunked prefill cuts a long prompt into fixed-size token chunks, and stall-free batching admits a chunk only when it fits the token budget left over after the running decodes. A decode is then never blocked behind a full prefill.

Findings. Against vLLM under tail-latency constraints, Sarathi-Serve reports 2.6× higher serving capacity for Mistral-7B on one A100 and up to 3.7× for Yi-34B on two A100s. With pipeline parallelism, it reports up to 5.6× higher end-to-end serving capacity for Falcon-180B.

Why it matters. It turns the throughput-latency tradeoff into a number you set, the per-iteration token budget, rather than a property of whichever prompts happen to arrive together. The cost is memory traffic, because each chunk's attention re-reads the KV cache of every earlier chunk of the same prompt, so the chunk size trades decode latency against prefill efficiency.

Adoption. Chunked prefill is on by default in vLLM's V1 scheduler and also ships in SGLang and TensorRT-LLM, so this is how production engines put prefill and decode in one batch.

Background. One engine usually serves both phases on the same GPUs, so a batch mixes a compute-bound prefill with memory-bound decodes. Prefill sets the first-token latency, decode sets the per-token latency, and both inherit one batch size and one parallelism plan.

Problem. Colocation couples the two latencies. A prefill in the batch delays every decode behind it, and the parallelism that suits prefill's arithmetic is not the one that suits decode's demand for bandwidth. Adding GPUs then raises throughput without meeting either latency target.

Key idea. Give each phase its own instances, with its own batching and parallelism, and transfer the KV cache from the prefill instance to the decode instance once the first token is produced. Optimize goodput, the request rate served inside both the TTFT and TPOT constraints, by searching over the prefill-to-decode instance ratio and each pool's parallelism.

Findings. Across several models, applications, and latency objectives, DistServe reports up to 7.4× more requests served or 12.6× tighter objectives than prior systems while keeping more than 90% of requests within both limits.

Why it matters. This is the clean statement of the split, and it defines goodput as the quantity to optimize. It also creates a new cost: a large, latency-sensitive KV transfer now sits on the critical path between the two pools, and the prefill-to-decode ratio has to be retuned when the workload changes.

Background. The two phases want different hardware. Prefill saturates tensor cores, and decode saturates memory bandwidth while leaving most of that compute idle. A homogeneous fleet buys the same accelerator, at the same power draw, for both.

Problem. When one machine type serves both phases, you pay for the newest GPU's compute during decode, which cannot use it, and for the power that compute draws. Fleet cost and power end up set by the phase that does not need the hardware.

Key idea. Split the phases across separate pools and let the pools differ. Newer accelerators take prefill, cheaper ones take decode, and the KV cache moves over the interconnect between them. The cluster designs are evaluated against a production inference trace.

Findings. Splitwise reports 1.4× higher throughput at 20% lower cost than a homogeneous baseline, or 2.35× the throughput under the same cost and power budgets.

Why it matters. It reframes disaggregation as a cost and power argument rather than only a latency one. Once the phases are separate they need not share an accelerator generation, which is what makes a heterogeneous fleet worth reasoning about at all.

Background. vLLM and SGLang admit waiting requests first-come, first-served. Shortest-job-first minimizes mean waiting time, but it needs each job's size before the job runs, and an LLM request's size depends on an output length that is unknown until decoding ends.

Problem. Under first-come, first-served, one request with a long output delays every request queued behind it (head-of-line blocking). Earlier approximations of shortest-job-first predict each request's length directly, either by prompting the serving model to state it or by classifying it into length buckets, and an exact length is hard to predict.

Key idea. The scheduler needs only the order of output lengths within the waiting batch, not the lengths themselves. A small OPT model (125M parameters when serving Llama-3-8B) is trained with a listwise ranking loss to score prompts, the queue is sorted by score, and a request that has waited past a threshold is promoted for a fixed quantum so that it cannot starve.

Findings. With 2,000 requests arriving in a burst on Llama-3-8B and LMSYS-Chat-1M, the ranking scheduler lowers P90 latency per output token by 2.8× relative to a scheduler that prompts the serving model for its own output length. In offline synthetic-data generation, it completes 1,000 requests 6.5× faster than first-come, first-served.

Why it matters. It removes the usual reason for first-come, first-served, which is that output lengths cannot be predicted: a relative order is easier to predict than a length and is all shortest-job-first needs. The starvation guard is the price, because a strict shortest-job-first queue never serves a long request while short ones keep arriving.

Background. LLM serving engines run each admitted request to completion and admit requests first-come, first-served, because how many tokens a request will generate is unknown when it arrives. The linked arXiv preprint dates from 2023; the paper later appeared at NSDI 2026.

Problem. First-come, first-served means one long generation holds its slot while short requests queue behind it, so head-of-line blocking sets the tail latency. You cannot sort by job size either, because the size is the output length and nobody knows it yet.

Key idea. Preempt at iteration granularity and manage the queue the way an operating system does. A multi-level feedback queue demotes a request each time it uses up its queue's time quantum, and skip-join places an arrival in the highest-priority queue whose quantum covers its prefill time, so a long prompt skips the queues it would only be demoted from. Paused requests' KV state is moved proactively between GPU and host memory.

Findings. Against vLLM, FastServe reports up to 31.4× higher throughput under the same average-latency requirement and 17.9× under the same tail-latency requirement.

Why it matters. It applies a standard operating-system scheduler to serving without predicting output length: long jobs sink through the queues, so short jobs finish first. It also exposes what preemption costs here, because a paused request's KV cache must stay in GPU memory or move to host memory, so every priority change is also a memory decision.

Background. Before this paper, a serving engine gave each sequence one contiguous KV cache buffer, sized for the longest output the request might produce. The reservation was made on arrival and held until the sequence finished.

Problem. Fragmentation, not compute, was the binding constraint. A contiguous reservation for an unknown output length wastes most of its own buffer, and the holes left between buffers block new sequences even when the total free memory would hold them.

Key idea. Apply virtual memory to the KV cache. Store it in fixed-size blocks, keep a per-sequence page table from logical positions to physical blocks, and let the attention kernel read blocks that are not contiguous, so sharing and copy-on-write between sequences become allocator operations.

Findings. Across the paper's workloads, vLLM provides 2-4× the throughput of FasterTransformer and Orca at the same latency. The advantage grows with longer sequences, larger models, and more complex decoding.

Why it matters. It is the single most important systems paper on the list. Once memory is paged, the batch a scheduler can form is set by an allocator you control rather than by the worst case you were forced to reserve for.

Adoption. vLLM originated the design; SGLang and TensorRT-LLM likewise manage the KV cache in fixed-size blocks behind a page table.

Background. PagedAttention assumes every layer of every model stores the same number of bytes per token, so a single uniform page size fits the whole cache. That held for the dense attention stacks it was designed against.

Problem. The assumption breaks on layers whose per-token KV footprints differ, such as sliding-window attention in Gemma-2 and hybrid Mamba and attention stacks in Jamba. One uniform page must then be sized for the largest layer, and every smaller layer wastes the difference.

Key idea. Use a two-level allocator that generalizes paging to heterogeneous embedding sizes, so each layer's blocks hold only what that layer stores. The tradeoff is allocator complexity for less waste.

Findings. Across diverse models, workloads, and GPU configurations, Jenga improves GPU memory utilization by up to 79.6% and serving throughput by 1.80× on average and up to 4.92×.

Why it matters. This SOSP 2025 paper is the direct sequel to PagedAttention. Architectures changed, and the allocator assumption that made vLLM fast became the part that had to change, showing how a systems abstraction responds to a workload it was not designed for.

Background. Chat turns, few-shot prompts, agent loops, and shared system prompts resend token sequences the server has already processed. Serving systems of the time freed a request's KV cache the moment the request finished.

Problem. Every later request that shares a prefix recomputes it, and nothing in a plain completion API tells the runtime which spans will come back. The runtime has to discover the sharing itself, then decide what to keep under a fixed memory budget. Reuse must also be exact, because a token's KV depends on every token before it.

Key idea. Keep finished requests' KV on the GPU and index it with a radix tree over token sequences. An arriving request walks the tree to its longest cached prefix and prefills only the rest, and eviction removes least-recently-used leaves that no running request references. Because the tree matches at token granularity, a cached prefix that ends partway through a block can still be reused, whereas vLLM's block-hash index reuses only full blocks.

Findings. Across agent, reasoning, few-shot, structured-output, retrieval, and chat workloads, SGLang delivers up to 6.4× more throughput than the evaluated state-of-the-art inference systems.

Why it matters. It turns prefix reuse into ordinary cache management, with a hit ratio, an eviction policy, and a scheduling decision attached. The block-hash and radix-tree indexes sit over the same paged KV storage and differ only in how a new prompt finds its match, which is why the comparison belongs next to PagedAttention.

Adoption. xAI and LinkedIn use SGLang, providing external production uptake of RadixAttention's automatic prefix reuse.

Background. Fair queueing in networks and operating systems shares a resource by charging each client for what it consumes. A definition of fairness therefore starts by fixing the unit of service that gets counted.

Problem. The cost of one LLM request varies by orders of magnitude with its input and output lengths. A ten-token prompt and a ten-thousand-token prompt each count as one request, so serving clients in round-robin request order lets a heavy client starve a light one while the scheduler appears fair. Per-client rate limits, the usual alternative, cap each client even when the server has spare capacity.

Key idea. Count service in tokens rather than requests, weighting input and output tokens by what they cost, and keep a virtual token counter per client. Inside continuous batching, the scheduler admits the next request from whichever backlogged client has received the least weighted service, and the paper proves a bound on the service difference between any two backlogged clients.

Why it matters. It separates two questions serving systems usually conflate: what fairness means, and which scheduler achieves it. Once the unit of service is a token, fairness and throughput can be traded against each other deliberately rather than by accident.

Background. A multi-tenant serving scheduler pursues two goals with separate mechanisms. VTC gives every client a fair share of token service, and a prefix cache rewards running requests whose prefix the cache already holds.

Problem. The two goals conflict. Serving the client owed the most service can evict a prefix that another client is about to reuse, so a fair scheduler loses locality, while a scheduler that always picks the longest cached prefix can starve clients whose prompts share nothing with the cache.

Key idea. Deficit Longest Prefix Match (DLPM) gives each client a per-round service quota, and among clients with quota remaining it runs the request with the longest cached prefix. The quota bounds how much service locality can take from fairness. A distributed variant, D²LPM, also balances load across workers.

Findings. The proposed schedulers preserve fairness while reporting up to 2.87× higher throughput than VTC and up to 7.18× lower per-client latency than a state-of-the-art distributed serving system.

Why it matters. It makes the interaction between fairness and prefix caching explicit instead of letting each policy assume the other is free. Two mechanisms that are each correct in isolation can cost each other throughput or fairness once deployed together.

Background. A collective is a communication operation in which every process in a group participates, such as broadcast, all-gather, reduce-scatter, all-reduce, or all-to-all. MPI specifies what each collective computes and leaves the algorithm to the library.

Problem. No single algorithm is fastest at every message size. Algorithms such as binomial trees and recursive doubling need few message rounds, which dominates the cost of short messages, but they send more bytes per process than necessary, which dominates the cost of long ones. A library that uses one algorithm per collective is therefore slow at one end of the size range.

Key idea. Model each algorithm's cost with a per-message latency α, a per-byte transfer time β, and a per-byte reduction time γ, then select among several algorithms by message size and process count. For a long all-reduce over p processes, the paper uses Rabenseifner's algorithm, a reduce-scatter followed by an all-gather, so each process sends about 2(p-1)/p of the data instead of the lg p full copies that recursive doubling sends.

Findings. On a Myrinet-connected Linux cluster the new algorithms outperform MPICH's previous ones for every collective the paper describes, and on an IBM SP they often outperform IBM's own MPI. The optimized all-reduce and reduce run several times faster than the native implementations on the Myrinet cluster, an IBM SP, and a Cray T3E.

Why it matters. The per-device costs used to compare parallelism layouts in this meeting come from this analysis. An all-reduce of M bytes over G devices is a reduce-scatter plus an all-gather, each moving (G-1)/G × M per device, so a tensor-parallel all-reduce at G = 4 sends 1.5 M per device. NCCL, which runs these collectives in GPU serving engines, likewise selects between ring and tree algorithms by message size.

Background. A model can be split within layers, across layers, or across sequence positions, and each layout creates different communication costs.

Problem. A layout that maximizes throughput may miss its latency target, and FLOP balance may hide memory or collective bottlenecks. Prefill and decode can favor different layouts.

Key idea. Model the arithmetic, memory traffic, and collective communication of each layout. Choose per phase and batch size against a latency-throughput frontier rather than one peak number.

Findings. On PaLM 540B with a 2,048-token context, the optimized layouts reach 29 ms per generated token at low batch size with int8 weights and 76% model-FLOPs utilization during large-batch prefill.

Why it matters. The model predicts which resource binds for a model shape, device count, batch size, and latency target before an expensive deployment sweep.

Background. A transformer layer contains two feed-forward matrix multiplications and attention over independent heads. Splitting a layer across devices requires choosing which matrix dimensions and heads each device holds.

Problem. A naive split synchronizes whenever devices need one another's partial results. Those collectives sit on every forward pass, so communication can erase the latency reduction from adding devices.

Key idea. Partition the first feed-forward matrix by columns and the second by rows, so the intermediate activation stays local and only the block output is reduced. Give each device whole attention heads. The resulting layer needs two all-reduces per forward pass.

Findings. Training an 8.3B-parameter model on 512 GPUs sustains 15.1 PFLOP/s and 76% scaling efficiency relative to a strong single-GPU baseline that sustains 39 TFLOP/s.

Why it matters. This layout defines tensor parallelism in current serving engines. Because its collectives are on every token's critical path, deployments usually keep the tensor-parallel group within one node.

Adoption. vLLM, SGLang, TensorRT-LLM, and DeepSpeed expose this tensor-parallel partitioning for multi-GPU inference.

Background. Pipeline parallelism gives each device a contiguous run of layers. Each device holds fewer weights, and devices communicate activations only at stage boundaries.

Problem. Layers remain sequential. Sending one input through K stages leaves K-1 stages idle at each instant, spending the memory benefit on pipeline bubbles.

Key idea. Split a batch into M micro-batches and send them through the stages back to back. The idle fraction falls from (K-1)/K to (K-1)/(M+K-1), while one synchronous update preserves training semantics.

Findings. For a Transformer with 32 micro-batches, moving from two to eight TPU partitions raises normalized throughput from 1.8 to 6.3. Bubble overhead becomes nearly negligible once the micro-batch count is at least four times the partition count.

Why it matters. In serving, the number of in-flight requests determines the available micro-batches. Pipeline parallelism therefore trades less communication per token for an idle fraction that worsens when the queue is short.

Adoption. vLLM, DeepSpeed, and Megatron-LM expose pipeline parallelism and use micro-batching to fill stages.

Background. Sequence parallelism splits one sequence's tokens across devices, so a context too long for one device's memory can still be processed. The degree is normally fixed when the deployment is configured, before any request has arrived.

Problem. One fixed degree has to serve every request. Size it for the longest context and short requests pay communication they never needed; size it for the common case and long requests do not fit. A single request wants different degrees in its two phases, because prefill is compute-heavy over the whole sequence while decode extends it one token at a time.

Key idea. Make the degree elastic. Vary it per request, and within a request across phases, migrating KV state between devices as the degree changes rather than fixing the device mapping when the cluster is configured.

Findings. Across real workload traces, LoongServe raises maximum throughput by up to 3.85× over chunked-prefill systems and 5.81× over prefill-decode disaggregation. On 16 GPUs, it improves total throughput by up to 1.86× over vLLM.

Why it matters. It moves a deployment-time constant into the scheduler's decision space, which is what makes it belong on this page rather than in a parallelism survey. Once the degree is elastic the scheduler trades communication cost against memory headroom on every admission, so parallelism and admission control stop being separable decisions.