How DeepSeek Rewrote the Transformer (MLA)
DeepSeek shrank the transformer's biggest memory bottleneck by a factor of 57 — and made the model better at the same time. Here is the linear algebra that did it.
Watch on YouTubeThe short version
- DeepSeek's multi-head latent attention (MLA) strikes at the core of the transformer, not its margins — it changes how attention itself stores keys and values.
- The KV cache is the bottleneck: caching keys and values turns attention's cost from quadratic to linear in token count, but for DeepSeek R1 that means 4 MB read per token and ~400 GB of memory reads to generate one token at a 100k context.
- Earlier fixes — multi-query and grouped-query attention — shrink the cache by sharing keys and values across heads, but pay for it with worse model performance.
- MLA instead projects the input into a compressed latent space shared across heads, then projects back up to per-head keys and values, cutting the cache 57x while improving performance.
- The trick that makes it free at inference: the up-projection weights get algebraically absorbed into the query and output computations, so the extra matrix multiply costs nothing once training is done.
01 · The release
DeepSeek hit the core of the transformer
In January 2025, DeepSeek's R1 matched leading language models on a fraction of the compute — and, unlike most American labs, shipped the weights, the inference code, and a steady stream of technical reports, roughly one a month through 2024. Most of DeepSeek's tricks live at the margins of the stack. Multi-head latent attention, introduced in June 2024, does not: it changes the transformer itself, the architecture nearly every large language model shares.
02 · Attention patterns
What attention actually computes
A language model generates one token at a time, and each new token is a function of every token before it. Attention is the mechanism that moves information between those token positions. It works by computing attention patterns — square matrices, one per head, sized to the number of input tokens. For the nine-token input "the American flag is red, white, and," every pattern is a 9x9 matrix.
A single head can learn a specific job. One pattern in GPT-2's third layer maps "American" onto "flag," fusing them into the concept American flag; a later head pulls "flag," "red," and "white" toward the final "and," setting up the correct next token: blue.
03 · Queries and keys
Building attention from matrices
Take the input matrix X — one row per token, one column per embedding dimension (768 for GPT-2 small, 7,168 for R1). Multiply X by two sets of learned weights, W_Q and W_K, to get the query matrix Q and key matrix K. Attention then searches for token pairs whose queries and keys point the same way.
- Score every pairTranspose K and multiply by Q, producing the 9x9 matrix of all key-query dot products at once.
- MaskZero out the upper-right corner so a token can't peek at tokens that come after it.
- NormalizeDivide by the square root of the dimension and apply softmax, so each row sums to one — that's the attention pattern.
- Apply itCompute a value matrix V = XW_V, then multiply the pattern by V to take a weighted sum of values. Stack the heads and multiply by W_O for the block's output.
04 · The KV cache
From quadratic to linear
An attention pattern's size is the token count squared — a real problem when context windows run past 100,000 tokens, about the length of the first Harry Potter book. But there's a shortcut. When the model appends "blue" and re-runs on ten tokens, the first nine rows of Q, K, and V are unchanged. The masking means only the new bottom row of the pattern needs computing, and that needs all the keys but only the newest query row.
05 · The cost of caching
Four megabytes per token
The shortcut trades compute for memory. The system has to hold keys and values for the entire session, across every head and every layer. For a model with L layers, NH heads, head dimension DH, and N tokens, the cache holds 2 x N x DH x NH x L numbers.
06 · Sharing keys and values
MQA and GQA pay in performance
Oversized caches predate DeepSeek, and the standard fixes share keys and values across heads. Multi-query attention uses a single shared K and V for all heads, shrinking the cache by the head count — 128 for R1 — but forcing every head onto the same keys and values kills specialization. Grouped-query attention softens that: heads share K and V within small groups. Meta's Llama 3 uses groups of eight, cutting the cache eightfold.
Grouped query attention reduces KV cache size, but still takes a performance hit relative to full multi-head attention.— Welch Labs
07 · Multi-head latent attention
Let the model compress its own cache
DeepSeek's move borrows a familiar idea — a latent space — and applies it in a new place. MLA adds a step between each head's input and its keys and values: project the input down into a compressed latent space shared across all heads (L_KV = XW_DKV), then project that latent back up into per-head keys and values with weights W_UK and W_UV unique to each head. That up-projection is what MQA and GQA lack, and it's where the flexibility — and the performance — comes back.
What if the model could learn to efficiently compress its own keys and values?— Welch Labs
08 · The result
A real improvement to the transformer
With MLA, the cache no longer depends on the number of heads — only on the size of the shared latent, 576 per token for R1. Standard attention would need 4 MB per token; grouped-query attention with groups of eight, 500 KB; MLA, just 70 KB. That's the 57x.
