Beyond Next-Token: How Diffusion LLMs Like Mercury 2 and LLaDA Hit 1,000+ Tokens Per Second in 2026
For five years, every flagship LLM has generated text the same way: one token at a time, left to right. In 2026, a new generation of diffusion language models is breaking that contract, and pushing inference past 1,000 tokens per second on commodity hardware.
1. Introduction: The 1,000 Token-Per-Second Wall
If you’ve used GPT-5, Claude 4.6, or Gemini 3 in the last year, you’ve watched tokens stream onto your screen one at a time. That isn’t a UI choice, it’s a hard architectural constraint. Autoregressive (AR) transformers, the family that powers virtually every chatbot you’ve used, are mathematically incapable of generating token n+1 until they’ve finished generating token n. No matter how big your GPU is, decoding is sequential.
That’s why a stat from February 2026 raised eyebrows: Inception Labs’ Mercury 2 reached 1,009 tokens per second on a single NVIDIA Blackwell GPU, over 5× faster than Claude 4.5 Haiku Reasoning (~89 tok/s) and GPT-5 Mini (~71 tok/s). It also wasn’t a tiny model: Mercury 2 is reasoning-capable, scoring 91.1 on AIME 2025 and 67.3 on LiveCodeBench.
The trick isn’t a faster GPU. It’s a fundamentally different decoding paradigm called diffusion language modeling (dLLM). In this post, we’ll unpack:
- How masked diffusion replaces next-token prediction with iterative denoising
- The math behind LLaDA’s forward and reverse processes
- Why bidirectional attention changes the inference economics
- How block diffusion and approximate KV caching made dLLMs practical
- What LLaDA 2.x, Mercury 2, Fast-dLLM v2, and SDAR look like under the hood
- The benchmarks, the trade-offs, and where this is going next
2. Background: The Autoregressive Bottleneck
An autoregressive language model factorizes the probability of a sequence as:
p(x₁, x₂, ..., xₙ) = ∏ p(xₜ | x<ₜ)Every token is conditioned on every prior token. That gives AR models their sharpness, but it also means generation is fundamentally serial. You can batch many requests across users, but a single response can’t be parallelized at the token level. Speculative decoding helps somewhat, but it’s a patch on a serial pipeline.
Two problems compound this:
- Causal attention is one-directional. Token 50 cannot revisit its decision after seeing tokens 51–100. Errors propagate.
- Compute is baked in. An AR model spends roughly the same FLOPs on the word “the” as on the answer to a math problem. There’s no easy knob to spend more compute on harder tokens.
Diffusion LLMs were designed to break both constraints at once.
3. The Core Idea: Generation as Iterative Denoising
Image diffusion models like Stable Diffusion start from pure noise and gradually denoise it into a picture. Diffusion LLMs apply the same playbook to text, except the “noise” is a sequence of [MASK] tokens, and "denoising" means filling them in.
3.1 The Forward (Masking) Process
During training, LLaDA, the foundational open-weight diffusion LLM from Renmin University released in February 2025, defines a forward process that gradually corrupts a clean sequence x₀ by independently masking each token with probability t, where t ∼ U[0, 1]:
xₜ = mask each token of x₀ independently with prob tAt t = 0 the sequence is clean. At t = 1 it's fully masked. Crucially, masking is independent per token, there's no causal direction.
3.2 The Reverse (Denoising) Process
The model learns a mask predictor p_θ(·|xₜ), a transformer that, given a partially masked sequence, predicts every masked token simultaneously. Training uses cross-entropy loss applied only to masked positions:
L(θ) = -E_{t, x₀, xₜ} [ (1/t) · Σᵢ 1[xₜⁱ = MASK] · log p_θ(x₀ⁱ | xₜ) ]This loss is a principled upper bound on negative log-likelihood, which means LLaDA is a true generative model, not a heuristic. At inference, generation walks backward from t = 1 to t = 0: start with all [MASK], predict all tokens, keep the most confident ones, re-mask the rest, and repeat.
LLaDA 8B was pretrained on 2.3 trillion tokens, surpassing LLaMA 2 7B on nearly all 15 standard zero/few-shot tasks and matching LLaMA 3 8B, the first hard evidence that diffusion can scale to frontier-quality language modeling.
4. Bidirectional Attention: Why Diffusion Sees More
The single biggest architectural difference between AR and diffusion LLMs is the attention mask.
How attention masks differ across model families:
- Autoregressive (GPT, Llama). Mask: causal (lower triangular). Each token sees only past tokens.
- First-generation Diffusion (LLaDA 1.0). Mask: full bidirectional. Each token sees every token in the sequence.
- Block Diffusion (Fast-dLLM v2, SDAR). Mask: block-causal with intra-block bidirectional. Each token sees all previous blocks plus every token inside the current block.
Bidirectional attention is the secret sauce. When the model fills in token 50, it isn’t just conditioning on tokens 1–49, it’s also looking at tokens 51–100 in the current draft. This is closer to how a human writer revises a paragraph: with full visibility of context on both sides.
The cost is that you can’t reuse a standard KV cache. In an AR model, the keys and values for token 49 never change once computed. In a diffusion model, every iteration may update every token, so naively, you have to recompute the full cache every step. That was the wall first-gen dLLMs hit.
5. The Two Generations of Diffusion LLMs
5.1 First Generation: Full-Sequence Refinement
LLaDA 1.0, Dream, and the original masked-diffusion lineage refined the entire sequence in lockstep. Quality was competitive, but inference was painfully slow, sometimes slower than equivalently sized AR models, for two reasons:
- No KV cache. Bidirectional attention recomputes everything every step.
- Token dependency disruption. When you decode many tokens in parallel under a conditional-independence assumption, you get drift and incoherence.
5.2 Second Generation: Block Diffusion
The breakthrough came in 2025 with Fast-dLLM (NVIDIA, May 2025) and its successor Fast-dLLM v2 (October 2025), alongside SDAR, WeDLM, and LLaDA 2.x. The trick: don’t refine the whole sequence at once. Refine it in blocks of 8 to 64 tokens, with a hybrid attention mask:
- Tokens within a block see each other bidirectionally.
- Tokens across blocks see only earlier blocks (causal).
This is sometimes called a blockwise causal mask. It preserves enough bidirectionality to keep the diffusion benefits, refinement, parallel decoding, while restoring enough causality to enable caching.
6. Block-Wise KV Caching, Explained
Fast-dLLM introduced a block-wise approximate KV cache that works as follows:
- Generate block k by running multiple denoising iterations on it.
- Within those iterations, treat the keys and values of all previous blocks (1 through k-1) as frozen and pull them from cache.
- After block k stabilizes, recompute and store its KV.
- Move to block k+1.
It’s an “approximate” cache because earlier blocks could in principle be revised after seeing later ones, but empirically, the approximation costs almost nothing. Fast-dLLM v2 layers a hierarchical cache on top: a coarse block-level cache plus a fine sub-block cache, so even within a 64-token block, partially-decoded prefixes get reused.
Combined with a confidence-aware parallel decoder, which only commits tokens whose softmax confidence exceeds a threshold and re-masks the rest, block diffusion finally clears the inference-speed bar AR models had held for years.
7. The Models Leading the Pack in 2026
7.1 LLaDA 2.x and LLaDA-MoE
The LLaDA family from Renmin University and Ant Group has progressed quickly:
- LLaDA 8B (Feb 2025): proved diffusion can match LLaMA 3 at scale.
- LLaDA 2.0 / 2.1: added block diffusion and token editing. Block-style sampling alone bumps GSM8K by +10.1 points and Math by +13.1 points on LLaDA-8B-Instruct.
- LLaDA-MoE (Sept 2025): a sparse mixture-of-experts diffusion model, combining the parallelism of diffusion with the parameter efficiency of MoE.
- LLaDA 2.x in production: clocks above 800 tok/s and is approaching 150,000 monthly downloads on Hugging Face.
7.2 Mercury 2 (Inception Labs)
Inception Labs, the team behind the first commercial-scale dLLM, launched Mercury 2 on February 24, 2026. It’s the first reasoning-capable diffusion LLM and the headline numbers are striking:
- 1,009 tokens/second on NVIDIA Blackwell
- 1.7-second end-to-end latency
- 5× faster than the leading speed-optimized reasoning LLMs
- AIME 2025: 91.1 · GPQA: 73.6 · IFBench: 71.3 · LiveCodeBench: 67.3 · SciCode: 38.4 · Tau2: 52.9
Mercury demonstrates the most underrated property of dLLMs: reasoning gets cheaper. Long chain-of-thought traces, which slaughter AR latency, become almost free when you can fill them in parallel.
7.3 Fast-dLLM v2, SDAR, and WeDLM
Fast-dLLM v2 took a different angle: instead of training a dLLM from scratch, it adapts a pretrained AR model into a block diffusion LLM with only ~1B tokens of fine-tuning. That’s a game-changer for cost, every existing Llama-class checkpoint becomes a candidate dLLM.
SDAR-1.7B and WeDLM-8B push the Pareto frontier further: WeDLM-8B reportedly beats both AR baselines and earlier dLLMs on quality and speed across multiple benchmarks.
8. The Dynamic Compute Trade-off
Here’s a property AR models structurally cannot offer: quality-vs-latency at inference time, with no model swap.
A diffusion LLM exposes the number of denoising steps as a runtime knob. Want a draft? Run 4 steps per block. Want a polished answer? Run 32. Same weights, same deployment, same code path. AR models can’t do this, your only knobs are model size (requires a redeploy), temperature (changes randomness, not compute), and best-of-n (multiplies cost linearly).
9. Comparing Diffusion vs Autoregressive LLMs
Autoregressive (GPT-5, Claude 4.6) vs Diffusion (Mercury 2, LLaDA 2.x), property by property:
- Decoding direction. AR: strictly left-to-right. Diffusion: parallel, iterative refinement.
- Attention. AR: causal. Diffusion: block-bidirectional.
- Throughput (single request). AR: 70 to 200 tok/s. Diffusion: 800 to 1,000+ tok/s.
- KV cache. AR: exact, native. Diffusion: approximate, block-level.
- Runtime compute knob. AR: none. Diffusion: number of denoising steps.
- Long-context coherence. AR: strong. Diffusion: improving rapidly.
- Reasoning latency. AR: high (long chain-of-thought is serial). Diffusion: low (chain-of-thought fills in parallel).
- Ecosystem maturity. AR: massive. Diffusion: early but accelerating.
10. Limitations and Open Challenges
Diffusion LLMs aren’t a free lunch. Several real issues remain:
- Approximate KV caching is approximate. On hard, long-range dependencies, block-level caching can introduce subtle inconsistencies.
- Variable-length output is awkward. dLLMs typically generate inside a fixed-length canvas; emitting clean stop tokens needs careful design.
- Tooling lag. vLLM, SGLang, TensorRT-LLM, and most serving stacks are deeply optimized for AR. dLLM kernels are catching up but aren’t yet at parity.
- Streaming UX. Block diffusion can stream block-by-block, but per-token streaming UX needs rework.
- Instruction-tuning regressions. Early LLaDA SFT runs showed MMLU dips traceable to SFT data quality, fine-tuning recipes are still maturing.
11. Where Diffusion LLMs Make the Biggest Difference
Three deployment scenarios where dLLMs are already the right answer:
- Latency-critical agents. Voice agents, real-time copilots, and tool-using agents that need sub-second turn latency benefit disproportionately from 5–10× throughput.
- High-volume reasoning. Test-time compute is the dominant cost driver in 2026. Mercury 2 shows that long reasoning traces don’t have to mean long wall-clock latency.
- Edge and on-device inference. A 1.7B diffusion model that fills 64 tokens in parallel on an NPU is far closer to feeling “instant” than a 7B AR model on the same chip.
12. The Road Ahead: Hybrid Architectures
The most likely future isn’t pure diffusion, it’s hybrid. Several trends to watch through the rest of 2026:
- Diffusion + MoE (LLaDA-MoE-style) for parameter-efficient scaling.
- Diffusion + Mamba hybrids for ultra-long-context generation where attention cost dominates.
- Speculative diffusion decoding: small dLLM drafts, big AR verifies, or vice versa.
- Multimodal diffusion LLMs like LLaDA-V, where the iterative-refinement paradigm naturally extends to images and audio.
- Cheaper conversion paths: Fast-dLLM v2 showed 1B tokens can convert an AR model into a dLLM. Expect this number to drop further.
13. Conclusion
For most of the deep-learning era, “language model” has meant “next-token predictor.” That assumption is finally being tested. Diffusion LLMs deliver something AR models structurally cannot: parallel generation, runtime quality knobs, and bidirectional refinement, at speeds that make 1,000 tokens per second a real production number, not a demo.
None of this means GPT-style AR models are going away. They have a decade of tooling, intuition, and ecosystem behind them. But the field has clearly broken out of the autoregressive monoculture. Mercury 2’s reasoning scores at 5× the speed, LLaDA 2.x matching Llama 3 at 8B, and Fast-dLLM v2 turning any AR checkpoint into a dLLM with a billion tokens of fine-tuning, these aren’t research curiosities. They’re early production signals.
The companies that figure out where dLLMs slot into their stack first, agents, voice, reasoning, edge, will have a structural latency advantage that’s very hard to copy with bigger GPUs alone.
References & Further Reading
- Nie et al., Large Language Diffusion Models, arXiv:2502.09992 (LLaDA paper)
- Wu et al., Fast-dLLM: Training-free Acceleration of Diffusion LLM, arXiv:2505.22618
- Wu et al., Fast-dLLM v2: Efficient Block-Diffusion LLM, arXiv:2509.26328
- LLaDA-MoE: A Sparse MoE Diffusion Language Model, arXiv:2509.24389
- Inception Labs, Introducing Mercury 2 (Feb 2026)
- Red Hat Developer, Beyond the Next Token: Why Diffusion LLMs Are Changing the Game(April 2026)
- LLaDA project page: ml-gsai.github.io/LLaDA-demo
- Fast-dLLM project page: nvlabs.github.io/Fast-dLLM
Connect
If you found this useful, follow me here on Medium for more deep dives on LLM architecture, quantization, and inference. Got questions or want to discuss diffusion LLMs? Reach out, I’d love to hear what you’re building.
- Email: abdullahramzan120@gmail.com
- GitHub: github.com/buzzgrewal
