bycloud · AI research deep-dive

Why Let LLMs Predict 4 Tokens at Once? Multi-Token Prediction Explained

A non-reasoning model can't tell you how many words its next sentence will have — because predicting one token at a time, left to right, makes it bad at seeing its own future. Multi-token prediction asks the model to call the next few tokens at once. It sounds like it should make things worse; it mostly makes them better, and it's most of why DeepSeek V3 trains the way it does.

Creator: bycloudRuntime: 11 minVideo published: May 27, 202557.2K views when summarized

Watch on YouTube

TL;DR

MTP in one pass

Next-token prediction's core flaw: one token at a time, left to right, so the model can't foresee its own output — it can't even tell you how many words its next sentence will have.
Multi-token prediction (MTP) predicts t+1 through t+4 at once via a shared transformer trunk plus several output heads — up to 4x faster on paper.
Naively it hurts quality, but trained jointly it boosts structure and syntax: coding +1.7–4.5% on the largest model, the ARC benchmark improves the more tokens you predict, and you keep a 3–4x speedup. It works because models already build implicit foresight (one hidden state predicts t+2 at ~48% accuracy), and bigger models have more room for it.
DeepSeek V3's twist: use MTP as a training objective and discard it at inference, with sequential modules instead of parallel heads so the causal chain holds and the gradients enrich the shared trunk.
Payoff: across 15.7B and 228.7B-parameter models, MTP-trained beats baseline almost across the board with little overhead — and keeping one MTP module for speculative decoding gives a 1.8x speedup at an 85–90% acceptance rate.

01 · The flaw in next-token prediction

It can't see its own future

Ask a non-reasoning model how many words its next sentence will have and it'll usually miss — not because it can't count, but because predicting one token at a time, left to right, makes it bad at anticipating its own output. That's the core weakness of the next-token paradigm. There are alternatives — diffusion language models fill a whole window at once, so every token gets a say, not just the ones already on the left; BERT attends in both directions — but both are big architectural leaps, and next-token prediction is already so well engineered they struggle to catch up. So here's the cheaper question: what if you just predict the next two, three, or four tokens at once?

The setup: a model generating left to right can't tell you how many words the sentence it's about to write will have.00:01:00

02 · Multi-token prediction

Shotgun the next few tokens

MTP is what it sounds like: instead of only predicting token t+1 from the context up to t, also predict t+2, t+3, and t+4 simultaneously. The first version (proposed about a year ago) is architecturally boring in a good way — a standard transformer "trunk" processes the input into one hidden state, and instead of a single output head you add several independent heads, each predicting a future position in parallel off that same shared state.

A shared trunk feeds multiple output heads, each predicting a different future token. One pass shotguns four tokens — up to 4x faster on paper.00:03:48

03 · Doesn't that wreck precision?

The weatherman fix

Predicting four tokens blind sounds like it should make everything sloppier, and naively it does — the research says quality degrades badly. bycloud's framing: it's four weather forecasters each predicting a different day with no communication. But train them together and it's more like they all went through the same school, so their predictions don't drift far apart.

The weatherman analogy: independent forecasters drift; jointly-trained ones stay coherent.00:04:24

What joint MTP training buys youStructural and syntactic ability goes up. Coding improves 1.7–4.5% on the largest model across two benchmarks; the ARC abstract-reasoning benchmark gets better the more tokens you predict at once; and you still get a 3–4x generation speedup. As bycloud puts it, "it's actually so free."

04 · Why it works

Models already have foresight

It's not a fluke. Researchers found that a single hidden state at position T already carries enough to approximate the t+2 token at about 48% accuracy. So even a model trained only on next-token prediction is implicitly building some foresight — MTP just makes it explicit and rewards it. That also explains why bigger models gain more from MTP: they have the internal room to build the foresight it promotes.

More internal space, more foresight — the bigger the model, the more MTP has to work with.00:05:40

Even if a model is only trained to predict the next token, their internal hidden states would still implicitly force the model to develop some level of foresight.— bycloud

05 · DeepSeek V3's twist

MTP as a training objective

So why haven't the big labs shipped it? They have — the current best open-source model, DeepSeek V3, just buried it under everything else it does. Its insight is to flip MTP from an inference-time prediction trick into a training objective, then drop it at inference (the deployed model does plain next-token prediction). And instead of the inconsistent parallel heads, it uses sequential MTP modules so information flows from the t+1 prediction into the t+2 one — one extra transformer block per extra token, reusing the main model's output head.

Sequential modules: one token plus one extra equals two, with the causal chain preserved. Back to the weathermen — now the day-two forecaster can see what day one predicted.00:07:43

The key is that the gradients from the MTP modules don't just train the extra blocks; they flow back through the shared trunk and output head, forcing the core model to learn richer, longer-range representations.

06 · Does it pay off?

Ablations and a free speedup

DeepSeek validated it carefully: 15.7B and 228.7B-parameter models, baseline next-token training versus the same models trained with one MTP objective (then discarded). The MTP-trained versions won almost across the board, with barely any added compute.

The ablation: MTP-trained beats the next-token baseline across both model sizes, nearly everywhere.00:09:20

The cherry on top: keep that one MTP module around at inference and use it for speculative decoding — predicting two tokens at once — and you get a 1.8x tokens-per-second speedup, with the second token accepted 85–90% of the time. It takes the best of multi-token prediction while sidestepping the parallel-head pitfalls.