Understand the systems that run models.

The Structure of an LLM: From the Embedding Layer to the LM Head

Follow text through token IDs, vectors, and vocabulary scores, and explore the roles of the embedding layer, decoder blocks, and LM Head.

The Flow Through a Decoder Block: Residual Connections and RMSNorm

Explore a decoder block around its residual stream, then follow vector addition and RMSNorm across each token’s d components.

Attention and MLP: Information Across Tokens and Transformations Within a Token

Compare Attention and MLP, then explore matrix multiplication and the computations inside basic and gated MLPs.

Attention Projections: Q, K, V and Multiple Heads

Explore Q, K and V projections, the organization of multiple heads, and how the output projection combines head results for the same token.

Core Attention: Combining Information Across Tokens

Follow Core Attention within each head through QKᵀ, scaling, a causal mask, Softmax, and a weighted sum of Value vectors.

RoPE: Incorporating Token Positions into Attention

Explore how RoPE rotates the components of Q and K by token position and how relative positions affect Attention scores.

MoE: Choosing Which MLPs to Use for Each Token

Compare dense MLPs and MoE, then follow routing, dispatch, expert computation, combination, and the shared expert path.

Revisiting the Flow Through the Model

Connect the components from Embedding to the LM Head, review Attention and MLP or MoE, and identify where information is combined across tokens.

MQA and GQA: Sharing KV Across Queries

Use MHA as a baseline to compare KV sharing in MQA and GQA, and examine how they reduce storage while preserving per-query outputs and what trade-offs follow.

MLA storage: Representing KV with a small latent vector

Explore how a joint latent vector represents head-specific keys and values, why a positional key is stored separately, what is shared across heads, and how KV cache storage grows.

MLA Computation: Attention Without Expanding KV

Follow how the current query reads a latent KV cache, and see why moving the key projection to the query and the value projection after the weighted sum preserves the result.

Local and Sparse Attention: Reading Fewer Token Positions

Start with sliding window attention as a form of sparse attention, trace fixed connection patterns and information flow through layers, and distinguish the read range from KV cache storage.

Sparse Attention Indexers: Choosing Positions by Content

Trace how an index query and keys select KV positions, how selection changes during generation, and how a sliding window combines with selection, while distinguishing reading cost from cache storage.

Token-Axis Compression: Reading Summaries of Multiple KV Positions

Trace how content and score projections build summary vectors, why completed summaries are read alongside recent per-token KV, and how compression combines with an indexer.

Linear Attention: Accumulating KV in a Fixed-Size State

Understand why softmax is replaced with independent feature maps, then follow outer-product writes, Query reads, and normalization by the sum of scores.

The Delta Rule: Revising State Associations Toward a New Value

Start from the purpose of writing KV associations, compare additive accumulation with the delta rule, and distinguish checking the old value with a Key from reading an output with a Query.

GDN and KDA: Retaining State and Applying Delta Corrections

Distinguish retention of the old state from correction toward a new Value, then follow GDN updates through KDA’s component-wise retention and input-dependent control values.

SSMs: Carrying Context Through State and New Inputs

Compare state updates in Linear Attention and SSMs, then use small numerical examples to trace retention, writing, reading, and the lingering influence of past inputs.

Mamba: Choosing What to Remember Based on the Input

Trace selective SSM coefficients for retention, writing, and reading, locate that computation within a Mamba block, and compare it with delta corrections in GDN and KDA.

Hybrid Models: Using State and Attention Together

Explore why models combine fixed-size state with position-indexed KV, the quality and storage trade-offs, and how one token passes through both types of layers.

HC and mHC: Widening and Connecting Residual Streams

Trace reading, bypass mixing, and writing from the same input, then examine why mHC constrains repeated mixing and what the wider residual path costs.

Gated Residual: Reading Components and Writing to Originals

Follow one numerical example through branch normalization, componentwise read gates, and branchwise scalar writes in Gated Residual.

Attention Residuals: Selecting Outputs from Earlier Layers

Calculate a weighted read over earlier layer outputs, derive its weights with a learned query, and examine the storage and selection trade-off of Block Attention Residuals.

2026-09-29

Cross-Layer KV Sharing: Reusing Keys and Values from Earlier Layers

Compare KV storage across six layers, follow a new token as its Keys and Values are reused by the next layer, and examine the trade-off between storage and computation.

2026-09-29

YOCO: Separating Layers That Produce and Read Shared KV

Follow shared KV from early to late layers and trace new-token processing to understand why prefill can skip past computations unnecessary for the first output.

2026-09-29

CED: Continuing Context with a Causal Encoder and Decoder

Compare generation paths and the sources of global and local KV, then connect sparse selection, three reuse modes, sequence compression, and prefill replay.

Hardware

CPU and GPU: Two Devices That Execute Model Computation

Explore why GPUs are used, how CPU and GPU designs differ, how host and device work together, and why compute units need a supply of data.

GPU Architecture: Compute Units and Memory

Explore SMs, CUDA Cores, Tensor Cores, and GPU memory, then connect capacity, bandwidth, and access latency to running a model.

2026-09-11

Parallelism in Model Operations: Element-wise Operations, Reductions, and Matrix Multiplication

Compare element-wise operations, reductions, and matrix multiplication through independent output elements and the inputs and computation needed for each result.

2026-09-11

Parallel Execution on GPUs: From Threads to Warp Scheduling

Start with a common computation procedure and indexed executions, then explore threads, blocks, grids, SM assignment, warp scheduling, and latency hiding.

How the CPU and GPU Execute Work Together

Distinguish CPU requests from GPU execution, then use buffers, streams, and events to explain data preparation, completion waits, and execution order.

2026-09-12

Starting GPU Optimization: Arithmetic Intensity and Data Movement

Connect memory and compute bottlenecks through arithmetic intensity and Roofline, then explore reduced data movement, concurrent progress, and element-wise kernel fusion.

2026-09-12

Optimizing Matrix Multiplication: Input Reuse and Tiling

Explore input reuse in shared memory and registers, the relationship between tile size and resource usage, and how data movement can overlap with computation.

2026-09-12

Why Attention Is Difficult to Optimize

Examine the cost of storing and rereading large score and probability matrices, and why a subset of scores cannot determine final Softmax probabilities.

2026-09-13

Processing Scores in Chunks with Online Softmax

Explore why stable softmax subtracts the maximum, how the maximum and exponential sum work together, and how online softmax updates them with each new chunk of scores.

2026-09-14

Reducing Memory Traffic with Output Accumulation in FlashAttention

Explore joint accumulation of the Value weighted sum and exponential sum, then follow FlashAttention-2 tiles to complete the output and reduce memory traffic from large intermediate matrices.

2026-09-14

From Operation Optimization to Whole-Model Performance

Explore how improving one operation affects whole-model execution time, why optimization requires repeated bottleneck measurements, and when performance goals or memory capacity call for more resources.

2026-09-14

Scaling Across Multiple GPUs

Explore compute and memory across GPUs, distinguish capacity, latency, and throughput goals, and examine how communication and synchronization-related waiting affect total execution time.

Transferring Data Between GPUs

Prepare communication participants and buffers, connect CPU send/receive submission to GPU execution, and explain overlapping independent computation and safe buffer reuse.

SMs and Copy Engines in GPU Communication

Distinguish the devices and connections used for GPU communication, and examine how resource contention with computation affects overall completion time.

GPU Communication Across Servers and the CPU's Role

Explore transfer paths that reduce intermediate copies through CPU memory and the CPU role in helping network communication progress.

The Basic Operations of Collective Communication

Compare Send/Recv with collectives, then follow the inputs and outputs of Broadcast, Scatter, Gather, and Reduce.

Collective Communication: Combinations and Extensions

Understand All-Gather, All-Reduce, Reduce-Scatter, and All-to-All through the data and output placement needed by the next computation.

How Collectives Move Data: Ring and Tree

Follow Ring and Tree implementations of the same All-Reduce result, and relate communication time to steps, data volume, and physical links.

DP: Replicate the Model, Partition the Inputs

Replicate a model across GPUs, distribute inputs, and distinguish inference request distribution from training gradient synchronization.

TP: Split One Operation Across GPUs

Partition weight matrices by columns and rows, then connect these partitions in FFN and attention to avoid intermediate communication.

SP: Partition Activations Alongside TP

Reduce activation duplication between TP regions, and alternate between token and feature partitioning to compute a Transformer layer.

CP: Split Long Contexts Across GPUs

Compare CP with SP, then follow how Ring and Ulysses connect information across GPUs to complete attention.

PP: Partition and Execute Model Layers

Place model layers across GPUs, then split the same batch into microbatches to overlap computation for different inputs.

EP: Partition Experts and Route Tokens

Use DP for attention and EP for experts on the same GPUs, then follow tokens as they fan out to experts and return to their original positions.

How Should We Arrange Multiple GPUs?

First partition the model so the workload can run, then replicate that configuration and adjust GPU placement for latency and throughput.

Workloads

Same Model, Different Workloads: Inference and Training

Compare training on prepared data with inference responding to newly arriving requests, then follow the execution flow from token generation and weight updates to request management.

Inference

Workloads

Inference and the KV Cache

Follow repeated token generation to see why computation repeats, how the KV cache reuses it, and where output-token selection differs from KV computation.

Prefill and Decode

Compare Prefill, which processes input tokens together, with Decode, which processes one new token at a time, and examine how token count and weight reuse change GPU bottlenecks.

Batching and Scheduling

Explore the limits of static batching, how batch membership changes between executions, how to check computation and KV space requirements, and why unused reservations make requests wait.

KV Cache Management and PagedAttention

Allocate KV blocks as needed and compute attention over scattered KV. Learn how block sharing and copy-on-write reduce storage when generating multiple responses from the same input.

When KV Capacity Runs Out: Pausing and Resuming Requests

When KV space runs out during generation, pause some requests to reclaim blocks, then resume them from retained token history or KV. Explore how this process delays output.