Background. Neural translation systems worked over a fixed word vocabulary of tens of thousands of entries, with every word outside that list collapsed into a single unknown token.

Problem. Rare words, names, and inflected forms fall outside any fixed word list, and a model cannot produce a word it has only ever seen as unknown. Enlarging the vocabulary moves the boundary without removing it.

Key idea. Build the vocabulary from characters upward. Repeatedly merge the most frequent adjacent pair until the vocabulary is full, so frequent words end up whole and rare ones decompose into pieces the model has already seen.

Why it matters. Tokenization is where a user's text becomes a token count, and that count is what the serving system prefills, caches, and bills for. Every sequence length in this course is denominated in units this algorithm chose.

Adoption. Byte-level BPE is the tokenizer for GPT-2 through GPT-4 and for Llama 3, and OpenAI's tiktoken and Hugging Face's tokenizers library both implement the merge algorithm directly.

Background. Sequence models before this paper were recurrent or convolutional, with attention bolted on top as a way for a decoder to look back at the encoder's states.

Problem. Recurrence forces computation to proceed one position at a time, so training cannot fill a parallel machine, and a dependency between distant tokens has to survive every intermediate step to reach the other end.

Key idea. Drop recurrence and convolution and build the model from self-attention plus position-wise feed-forward layers, with position supplied explicitly. Every position attends to every other inside one matmul-heavy block, and identical blocks stack.

Findings. On WMT 2014 English-to-German translation, the Transformer reaches 28.4 BLEU, more than 2 BLEU above the previous best result, including ensembles. On English-to-French, one model reaches 41.8 BLEU after 3.5 days of training on eight GPUs.

Why it matters. This is the architecture the whole course optimizes. Read it for shapes, not for BLEU scores: which operations are matmuls, which are attention, and what has to be kept around from one token to the next. Attention's quadratic cost and that per-token state are the two facts everything downstream reacts to.

Adoption. PyTorch's nn.TransformerEncoderLayer, Hugging Face Transformers, and vLLM implement the block layout directly.

Background. The Transformer paper states its architecture as equations over tensors, assuming you can already hold shapes, batching, and multi-head projections in your head while you read.

Problem. Read that way, the hard part is not the mathematics but the bookkeeping: which axis carries the head, where the sequence dimension goes, and how the query, key, and value projections line up. Equations hide exactly the part a systems reader needs.

Key idea. Walk the same computation as pictures, one tensor at a time, from tokens and embeddings through self-attention and multiple heads to the decoder stack. Each step shows the array being reshaped rather than the formula that produces it.

Why it matters. It is the fastest visual route to the shapes, and shapes are what this meeting reasons about. Read it before the papers if tensor shapes are not yet automatic.

Background. Harvard NLP's annotated implementation sets the text of Attention Is All You Need beside the code that implements each part, in the order the paper presents it.

Problem. An architecture description leaves the mechanical details unstated: masking, initialization, the label-smoothed loss, and the learning-rate warmup. Those details decide whether an implementation trains at all, and none of them are visible in the equations.

Key idea. Implement the original architecture line by line and annotate each choice in place. The code connects the equations in Attention Is All You Need to executable modules and tensor operations while making implicit details explicit.

Why it matters. It is the shortest path from the paper to something you can run and instrument, which is where every measurement in this course begins.

Background. Every Transformer block is wrapped in LayerNorm, which subtracts the mean of an activation, divides by its standard deviation, then applies a learned gain and bias.

Problem. Normalization costs almost no arithmetic and a full read and write of the activation tensor, so it is bound by memory traffic. LayerNorm also computes two statistics, and the paper argues that the re-centering one earns little: the invariance that matters is to re-scaling.

Key idea. Drop the mean subtraction and the bias. Divide the activation by its root mean square and apply a learned gain, which keeps the re-scaling invariance while removing one reduction and several elementwise operations.

Why it matters. It is the standard example of a memory-bound operation worth fusing, since nearly all of its cost is the trip to memory rather than the maths; a small paper that shows up in every fusion discussion. It also sets the pattern for the rest of the block: cheap ops are judged by bytes moved, not by FLOPs.

Adoption. RMSNorm replaced LayerNorm in Llama, Qwen, Mistral, Gemma, and DeepSeek, and fused RMSNorm kernels ship in vLLM, FlashInfer, and NVIDIA's Transformer Engine.

Background. The feed-forward block in the original Transformer is two weight matrices with a ReLU between them: project up to a wider hidden dimension, apply the nonlinearity, project back down.

Problem. That nonlinearity was inherited rather than argued for, and the block holds most of a model's parameters and most of its prefill FLOPs. Anything that buys quality per parameter there is worth having, and anything that changes its shape changes the whole cost model.

Key idea. Replace the single nonlinearity with a gated linear unit. Two separate projections of the input are multiplied elementwise, one of them passed through Swish first, and the product is projected back down. That is why the MLP block has three weight matrices instead of two.

Why it matters. It changes the FLOP and parameter accounting of the block you spend the most on, so any roofline estimate or memory budget for a current model has to count three projections rather than two. The paper offers no mechanism for why gating helps, and says so; the field adopted it on the measurement alone.

Adoption. SwiGLU is the MLP in PaLM, Llama, Qwen, Mistral, and Gemma, and fused gate-up-and-multiply kernels for it ship in vLLM and Unsloth.

Background. A current dense decoder assembles the pieces above: pre-norm blocks with RMSNorm, a SwiGLU feed-forward layer, rotary position embeddings, and attention in which several query heads share one key-value head. A model report states the result as a table of shapes, and those shapes are the inputs to every serving estimate.

Problem. At 405B parameters on a 16K-GPU run, the architecture is a stability decision as much as a quality one. A mixture of experts would cut the FLOPs per token, but it adds routing and load-balancing failure modes to a run that is already hard to keep alive.

Key idea. Keep the dense Transformer of Llama 2 with minor adaptations, chosen, in the report's words, to maximize training stability, and spend the effort on data quality and scale. The adaptations that matter for serving are grouped-query attention with 8 key-value heads, adopted to improve inference speed and to shrink the KV cache during decoding, a 128K-token vocabulary, and a RoPE base frequency of 500,000 for contexts up to 128K tokens.

Why it matters. These are the shapes the slides size end to end. From the report's Table 3 the 70B model has 80 layers, a model dimension of 8,192, an FFN dimension of 28,672, and 64 query heads sharing 8 key-value heads, so its KV cache costs 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes (320 KB) per token in BF16, one eighth of what 64 key-value heads would need. Derive the weight bytes and FLOPs per token the same way and check each slide figure yourself. Section 3 also reports the failure rates of the training run, which is the infrastructure side of the same report.

Background. A serving estimate starts from three numbers per model: the bytes of its weights, the FLOPs a token costs, and the bytes of KV cache a token leaves behind. Each follows from the model's shape, and none of them needs a profiler.

Problem. A benchmark reports latency for one model on one machine, and the number does not transfer. Without the derivation behind it you cannot tell whether a decode step waits on memory or on arithmetic, so you cannot predict what a larger batch, another GPU, or another model will do.

Key idea. Derive each quantity from first principles and compare it with the hardware's ratio of FLOP/s to memory bandwidth. A forward pass costs about 2 FLOPs per parameter per token, the KV cache costs 2 × 2 × layers × heads × head dimension bytes per token in 16-bit formats, and a step stays bound by memory bandwidth until the batch holds enough tokens to use each weight byte as many times as that ratio requires.

Findings. For an A100 40GB at 312 TFLOP/s and 1.5 TB/s, the post puts the ratio at 208, so decode with fewer than about 208 tokens in the batch is bound by memory bandwidth. Checked against FasterTransformer on a 13B model at batch size 1, its predicted 16.8 ms per token compares with a measured 22.0 ms, and the post closes most of the gap by assuming 90% of peak bandwidth and adding the activation traffic.

Why it matters. It is the derivation behind the slides' table of weights, compute, and KV cache, written out so you can redo it for any model in this course. The threshold depends on the GPU: on the A100 80GB SXM (312 TFLOP/s, 2,039 GB/s) it is 312 ÷ 2.039 ≈ 153, and on the H100 SXM (about 989 TFLOP/s, 3.35 TB/s) it is about 295. The batching, quantization, and kernel results later in Part II are claims about which side of this line a workload sits on.

Background. GShard established the mixture-of-experts layer with each token routed to two experts. The reasoning was that a router only learns useful gradients when it has at least two experts to compare, so top-2 was treated as the floor.

Problem. Sending every token to two experts doubles both the expert compute and the all-to-all traffic that carries tokens to the devices holding those experts. Sparse models of the period were also reported as unstable to train, so cutting to one expert risked losing the model outright.

Key idea. Route each token to exactly one expert, and show the simplification holds. A load-balancing auxiliary loss keeps the router from concentrating on a few experts, and a capacity factor caps how many tokens any expert accepts, dropping the overflow.

Why it matters. The routing decision is data-dependent, which is what turns expert placement and load balancing into a serving problem. Top-1 is also the cheapest point on that curve, so it sets the baseline every later routing scheme has to beat on quality per unit of communication.

Adoption. The load-balancing auxiliary loss and capacity-factor token dropping are standard in the MoE layers of Megatron-LM and DeepSpeed-MoE, both of which also expose top-1 routing.

Background. The MoE results through 2021 came from models whose weights stayed inside the labs that trained them, so how a real sparse model behaves under serving load could only be studied by its owner.

Problem. You cannot measure expert load skew, all-to-all cost, or resident memory footprint for parameters you do not have. Sparse models hold far more parameters than any single token uses, and that gap is exactly what makes their serving cost hard to predict from a paper.

Key idea. Release an open-weight sparse mixture of experts in which each token reaches only a subset of experts per layer. The server keeps far more weights resident than a token touches, and the license lets anyone measure that gap.

Findings. Each token uses 13 billion active parameters while accessing a 47-billion-parameter pool, and Mixtral matches or exceeds Llama 2 70B across the reported benchmarks.

Why it matters. It is what makes MoE serving experiments feasible for a course project. With the weights in hand you can see the load skew your own traffic produces, measure what expert parallelism costs on your own interconnect, and decide where the memory goes.

Adoption. vLLM, SGLang, llama.cpp, and Hugging Face Transformers support the Mixtral 8x7B and 8x22B architectures.

Background. A conventional mixture-of-experts layer holds a modest number of large experts and routes each token to one or two of them, on the assumption that an expert will specialize if the router is trained to send it consistent work.

Problem. Coarse experts limit specialization from two directions. Each expert is large enough that it must cover several unrelated things at once, and the knowledge every token needs gets learned separately inside all of them, so capacity is spent on duplicates.

Key idea. Split each expert into finer-grained pieces and route to more of them, which gives the router many more combinations to choose from. Then set aside a few shared experts that every token passes through, so the common knowledge lives in one place and the routed experts are free to specialize.

Why it matters. This is the design the DeepSeek model line is built on, so it fixes the expert shape a serving stack has to place, balance, and keep resident. Many small experts per token means more all-to-all messages, each one smaller, which is a different network problem from top-1 routing.

Background. An open-weight mixture-of-experts model at frontier scale, 671B total parameters with 37B activated per token, pretrained on 14.8T tokens for 2.788M H800 GPU hours. By 2024 the frontier labs had stopped publishing numbers of this kind.

Problem. Training a sparse model this large is where the failures live. Routing collapses onto a few experts, the auxiliary loss that prevents the collapse pulls against the language-modeling objective, and expert-parallel all-to-all traffic contends with compute for the same devices.

Key idea. Read the systems half. Auxiliary-loss-free load balancing steers the router with a per-expert bias instead of a gradient penalty, a multi-token prediction objective trains heads that predict further ahead than the next token, and FP8 carries most of the arithmetic. The report states what the run cost and how stable it was.

Why it matters. It is the clearest case of the split between memory and compute. A server must hold all 671B parameters, 1,342 GB in BF16 at 2 bytes each, while a token costs about 2 × 37B = 74 GFLOPs, roughly one eleventh of the 810 GFLOPs a dense 405B model spends. Its attention layers use MLA, which the KV-cache meeting on Nov 16 covers. Its mechanisms also left the paper: the multi-token prediction heads became a speculative decoding path in vLLM and SGLang, and the kernels behind the run were released as DeepGEMM and DeepEP.

Background. A standard Transformer fixes its depth at training time, so every token costs the same number of layers. A reasoning model that wants more computation per answer gets it by generating more tokens, which grows the context and the KV cache with every step.

Problem. Spending test-time compute through tokens ties the amount of computation to sequence length. More reasoning means more decode steps and more cache, and training it requires data in which the reasoning is written out.

Key idea. Loop a block instead of stacking more of them. A 2-layer prelude embeds the input, a 4-layer core block is applied r times, and a 2-layer coda produces the output, so 32 iterations give an effective depth of 132 layers from the weights of 8. Training samples r from a heavy-tailed distribution with a mean of 32, so the model learns to use whatever depth it is given at test time.

Findings. The 3.5B-parameter model, trained on about 800B tokens, improves on reasoning benchmarks as iterations increase: ARC-Easy rises from 49.07 at 4 iterations to 69.91 at 32. The authors report gains up to a compute load equivalent to a 50B-parameter model.

Why it matters. Depth becomes a runtime parameter, which moves cost from weight memory to compute and latency. The loops are sequential, so a token that iterates 32 times pays for 32 passes through the core block while the weights stay at 3.5B parameters. The paper also tests three serving consequences without retraining: a token can exit early when successive iterations agree, iterations can share a fixed KV-cache budget, and fewer iterations can draft tokens that more iterations then verify.

Background. Every model above generates left to right, one token per forward pass, which is why decode is a sequence of steps bound by memory bandwidth. Image diffusion models generate differently, by refining a whole sample over a number of denoising steps.

Problem. The common assumption is that in-context learning and instruction following come from the autoregressive objective. A model that generates in another order has to reach comparable capability at comparable scale before its decoding properties are worth a serving system's attention.

Key idea. Train a Transformer without a causal mask to predict masked tokens. The forward process masks a random fraction of a sequence, and generation runs the reverse: start from a fully masked response and unmask it over a chosen number of steps, each step a forward pass over the whole sequence.

Findings. LLaDA 8B, pretrained from scratch on 2.3T tokens in 0.13 million H800 GPU hours, scores 65.9 on 5-shot MMLU against 65.4 for Llama 3 8B, and 70.3 on 4-shot GSM8K against 48.7, in the paper's own evaluation of both base models. The number of sampling steps trades efficiency against sample quality.

Why it matters. It changes what one decode step is. A forward pass can fill several tokens, but it covers the whole sequence with bidirectional attention, and the paper states that LLaDA is incompatible with KV caching. The per-step cost and the cache that the rest of Part II optimizes therefore do not carry over unchanged, and that accounting is the reason to read it.

Background. LLaDA trained a diffusion language model from scratch on 2.3T tokens. Autoregressive models of similar size carry far more training, such as Qwen2.5 7B at 18T tokens.

Problem. Training each diffusion model from scratch repeats the most expensive part of building an LLM. The question is whether an autoregressive model's weights can serve as the starting point instead.

Key idea. Initialize the diffusion model from Qwen2.5 7B with the same configuration, then continue training on 580B tokens with a masked-diffusion objective and a noise schedule that adapts to each token's context. Generation keeps the diffusion properties: arbitrary order, infilling, and a step count that trades speed for quality.

Findings. On 8-shot GSM8K in the paper's comparison, Dream 7B scores 77.2, against 70.9 for LLaDA 8B and 78.9 for Qwen2.5 7B. Fewer diffusion steps speed up inference at some cost in quality, and on the Countdown planning task the paper reports that 5-20 steps beat Qwen2.5 7B on both speed and quality.

Why it matters. The step count is a per-request parameter that sets latency and quality together, which makes it a scheduling input in the way output length is for an autoregressive model. Reusing autoregressive weights also lowers the cost of the approach: the diffusion variant needed 580B tokens of training against the 18T its starting model was pretrained on.