Background. Multi-head attention gives every head its own key and value projections, so incremental decoding has to store, and then re-read, one key vector and one value vector per head per layer for every token in the sequence.

Problem. A decode step does very little arithmetic and reads the whole cache to do it, so generation speed is set by how many bytes of keys and values move rather than by FLOPs. The cache scales with heads, layers, batch size, and length at once.

Key idea. Keep the query heads and collapse the rest, so one shared key head and one shared value head serve all of them. The cache shrinks by the number of heads, and every decode step reads that much less.

Findings. On WMT14 at batch size 1,024, multi-query attention cuts incremental decoder cost from 46 to 3.8 TPUv2 microseconds per output token, while beam-search BLEU rises from 28.4 to 28.5.

Why it matters. It changes the size of the problem before any serving system touches it, and it pays for the reduction in model quality rather than in engineering. Every optimization later in this meeting operates on a cache whose baseline size this one decision already set.

Adoption. PaLM projects keys and values to one shared head while retaining every query head. Gemini 1.0 names multi-query attention as the efficient attention behind its 32K context, and StarCoder, Falcon-7B, and Gemma 2B each ship one key-value head.

Background. Multi-query attention minimizes the KV cache, while multi-head attention preserves quality. Their different parameter shapes made the layout appear to be a choice fixed before pretraining.

Problem. Multi-query attention loses quality on some tasks, but converting an existing multi-head checkpoint appeared to require pretraining again from scratch.

Key idea. Use an intermediate number of key and value heads, each shared by a group of query heads. Convert a multi-head checkpoint by mean-pooling projections within each group, then repair the lossy conversion with continued training.

Findings. The repair uses 5% of the original pretraining steps. The converted eight-group model scores 47.1 on the paper's seven-task average against 47.2 for multi-head attention, while reducing time per sample from 1.51 seconds to 0.28.

Why it matters. Cache size becomes a dial set by the number of groups rather than a binary choice, and an existing model can move along that tradeoff. The approximation is paid during conversion; serving remains exact for the converted model.

Adoption. Grouped-query attention appears in Llama 2 70B, every Llama 3 and Llama 4 release, Qwen 2.5, and Mistral models. Hugging Face Transformers, vLLM, SGLang, llama.cpp, and TensorRT-LLM support the layout.

Background. Multi-query and grouped-query attention reduce the number of key and value heads but preserve the shape of each cached entry.

Problem. Reducing the group count eventually costs quality, so larger savings require changing what an entry stores.

Key idea. Project the hidden state into a low-dimensional latent and cache it with one small shared key carrying rotary position information. Fold the up-projections into the query and output matrices so decoding never materializes full keys and values in memory.

Findings. DeepSeek-V2 caches 576 elements per token per layer, 3.56× fewer than grouped-query attention with eight groups. Its controlled comparison reports 4% of multi-head attention's cache at the larger evaluated scale; the often-quoted 93.3% reduction also includes six-bit cache quantization.

Why it matters. Latent attention compresses the representation rather than the head count. Storage and read traffic fall together, but attention arithmetic rises because the folded form scores a wider latent vector. Dedicated decode kernels are therefore part of the design.

Adoption. DeepSeek V2 through V3.2, Kimi K2, Kimi Linear, and LongCat-Flash use latent attention. vLLM, SGLang, TensorRT-LLM, FlashInfer, and llama.cpp provide dedicated kernels.

Background. Earlier methods reduce heads or entry dimensions while preserving a distinct cache for every layer.

Problem. Model depth multiplies the cache size, and that multiplier grows as model families scale.

Key idea. Give key and value projections to only some layers and let adjacent layers reuse their output. Every layer retains its own queries, attention, and output projection, so the model shares stored state rather than computation.

Findings. Pairwise sharing halves the cache on top of multi-query attention. Validation perplexity worsens by 0.04 points at 1B parameters and 0.05 at 3B, although both models were trained at only 2,048-token context.

Why it matters. Capacity and bandwidth are different budgets. Sharing doubles how many sequences fit, but each layer still reads the shared entry during decode, so per-step traffic does not fall.

Adoption. Character.AI, Apple's on-device foundation model, Gemma 3n and Gemma 4, and Hunyuan-Large share key-value state across layers.

Background. Cross-layer attention divides the depth multiplier by sharing state among a few neighboring layers.

Problem. Sharing across the entire model is harder because late layers would attend to state computed near the input.

Key idea. Split the model into a self-decoder and a cross-decoder. The lower half produces one global set of keys and values; every upper layer attends to that set. Cache size no longer scales with upper-layer depth, and prefill can stop producing cache entries halfway through the model.

Findings. At one million tokens, the trained 3B model uses 12.4 GB of inference memory, 9.4× less than the grouped-query baseline with the same kernel optimizations. Results quoted for 30B and 65B are projections rather than measurements.

Why it matters. YOCO removes the depth multiplier from storage, but not the sequence-length term from decode. Every cross-decoder layer still reads the global cache, so bandwidth and arithmetic continue to grow with context.

Adoption. Apple's on-device foundation model uses cache-free upper layers, and Microsoft's Phi-4-mini-flash-reasoning cross-attends from later layers to one global attention layer.

Background. Earlier methods change what a model stores per token while still letting every query read every retained position.

Problem. A fixed pattern is cheap to execute, but it may hide distant information that the current query needs.

Key idea. Let each query attend only to the latest W positions and store the cache in a fixed circular buffer. Per-token work stops growing after W, while stacking layers expands the theoretical receptive field beyond one window.

Findings. Mistral 7B uses a 4,096-token window in all 32 layers and reports 8× less cache at a 32k sequence. That factor is sequence length divided by window size; the paper does not evaluate quality beyond the window.

Why it matters. A sliding window reduces storage, read traffic, and arithmetic together while remaining exact for a model trained with it. However, recency rather than relevance decides what survives.

Adoption. Gemma 2, Gemma 3, GPT-OSS, Cohere Command A, and Ministral 8B interleave windowed and full-attention layers, preserving some global paths while bounding most layers' caches.

Background. Every entry above keeps a per-token cache and makes it smaller, by sharing heads, compressing an entry, sharing layers, or reading fewer positions. A state-space layer carries a recurrent state whose size is fixed by the state and channel dimensions, so length drops out of the cache term instead of being scaled down. Earlier structured state-space models were fast but their transitions did not depend on the input, which left them unable to do content-based recall.

Problem. A recurrence that ignores its input cannot decide what to keep, and that is why state-space models lost to attention on language. Making the transitions functions of the input repairs the recall, and it also removes the convolutional form the fast state-space algorithms relied on, so the modeling fix destroys the speed that motivated the family.

Key idea. Let the state-space parameters be functions of the input, which the paper calls selection, then recover throughput with a hardware-aware parallel scan that keeps the expanded state in SRAM rather than materializing it in HBM. A decode step then carries one state per sequence per layer, and that state is the same size after a million tokens as after one.

Findings. The paper reports 5× the generation throughput of a Transformer of comparable size, and Mamba-3B matching Transformers of twice its parameter count both in pretraining perplexity and on downstream evaluations.

Why it matters. It is the one entry here that deletes the KV-cache term rather than reducing it, so batch capacity stops depending on context length and the bytes read per generated token stop growing with it. Two costs come with that. Fixed state recalls a specific earlier token less reliably than attention, and it exposes no per-token entries to address, so the serving machinery the rest of this course builds on a token-indexed cache has nothing to act on: paged allocation, prefix reuse, and eviction all name entries the layer does not have.

Adoption. Mamba blocks ship in Hugging Face Transformers and vLLM. Jamba, Codestral Mamba, Falcon-Mamba, and NVIDIA's Nemotron-H all interleave state-space layers with attention layers rather than dropping attention entirely, which keeps a bounded number of layers that can still be addressed per token.

Background. Every full-attention layer retains per-token state even after heads, dimensions, layers, or windows are reduced.

Problem. Compression cannot eliminate growth with context; pure linear attention can, but it recalls specific earlier tokens less reliably.

Key idea. Combine mostly linear-attention layers, whose recurrent state has fixed size, with periodic latent-attention layers for exact lookup. Cache size follows the number of full-attention layers rather than total depth.

Findings. The 48B model uses seven latent-attention layers among 27 and reports up to 75% less cache than its full-attention counterpart. At batch size one and one million tokens, decode is 2.2× faster; the often-quoted 6× assumes a larger batch enabled by the freed memory.

Why it matters. Most layers replace sequence-growing cache with fixed recurrent state. The tradeoff moves into pretraining, where the ratio of linear to full layers is fixed rather than tunable per deployment.

Background. Many sparse-attention methods drop positions at inference from a model trained with full attention.

Problem. The model then sees an unfamiliar attention pattern, and scattered token-level sparsity may save FLOPs without reducing memory transactions.

Key idea. Pretrain three parallel attention branches: compressed block summaries, selected raw blocks, and a sliding window. Independent sigmoid gates combine their outputs, and blockwise selection preserves contiguous memory access.

Findings. At 64k tokens, the kernel is 9.0× faster forward and 6.0× faster backward than a Triton FlashAttention-2 implementation. The separate 11.6× figure is an expected ratio from memory-access volume, not a measured speedup.

Why it matters. Sparse arithmetic and reduced memory traffic are separate achievements. Because the selected blocks change by query, the full cache remains stored; the design reduces bytes read and attention arithmetic, not capacity.

Background. Mixture-of-experts layers route each token selectively, while attention usually reads every key or follows a fixed sparse pattern.

Problem. A fixed pattern may discard useful long-range dependencies, while full attention grows quadratically with sequence length.

Key idea. Treat context blocks as experts. A parameter-free gate scores each block from the query and its mean-pooled keys, then routes attention to the top blocks while always retaining the current block for causality.

Findings. At 95.3% sparsity, the attention layer of an 8B model is up to 6.5× faster than FlashAttention at one million tokens and 16× faster at ten million, with validation loss within 0.001 of full attention.

Why it matters. Conditional computation applies to context rather than parameters. Because routing adds no parameters, a layer can alternate between sparse and full attention during training or across serving phases.

Adoption. Independent implementations include a CUDA kernel from MIT and NVIDIA and support in flash-linear-attention.

Background. Native sparse-attention methods require pretraining, which is impractical for an existing frontier checkpoint.

Problem. Reading fewer past tokens can reduce long-context cost, but it may also damage the reasoning and tool use that need the context.

Key idea. Add a lightweight indexer to the dense checkpoint. It scores every past position from one shared key per token, and attention reads only the top 2,048 latent entries. First train the indexer to imitate dense attention with the model frozen, then continue training the full model.

Findings. At 128k tokens, reported serving cost falls about 3.5× for prefill and 9× for decode while MMLU-Pro remains 85.0. However, the indexer still scans the full context on every step.

Why it matters. Reading less is not storing less. The full latent cache remains and the indexer adds state, so stored bytes rise about one fifth even as read traffic at 128k falls 4.6×. The indexer's full scan also caps the asymptotic saving.

Background. Token selection reduces what attention reads but still stores one cache entry per token.

Problem. At one million tokens, both cache capacity and the selector's full-context scan remain expensive.

Key idea. Compress the token axis before selection. One layer type forms learned summaries at one quarter of the original length and applies top-k selection; another pools every 128 positions and attends densely. Both retain an uncompressed 128-token local window.

Findings. At one million tokens, the larger model uses 27% of DeepSeek-V3.2's single-token inference FLOPs and 10% of its cache, despite activating more parameters. It scores 83.5 on million-token MRCR.

Why it matters. This method shrinks the number of entries rather than their width or number of copies. Compression reduces storage, read traffic, and arithmetic together; selection then reduces only the last two.

Adoption. vLLM, SGLang, and the Ascend and Cambricon ports of vLLM implement the mechanism.

Background. Training-free sparse-attention methods report results under different sequence lengths, model sizes, tasks, and sparsity levels.

Problem. Those results do not answer which method and sparsity level an operator should use for a particular workload.

Key idea. Evaluate six training-free methods under one protocol while varying sequence length, model size, and sparsity together. The set spans both prefill and decode.

Findings. Tolerance depends on the task, phase, model size, and length. A 7B model's usable decode compression falls from 12× at 16k tokens to 5× at 128k, while 32B and 72B models remain near 17×.

Why it matters. One fixed sparsity setting cannot serve every workload. A sparse-attention speedup is not portable unless it names the task, sequence length, model size, and phase. Aggregate averages hide this operating regime.