Artificial Intelligence

Machine Learning

Large Language Models

Deep Learning

Deepseek

How AI Finally Killed Quadratic Attention: NSA, Mamba-3, and the Architectures Making Million-Token Context 11× Cheaper in 2026

Self-attention has carried the transformer for nearly a decade, and quietly overcharged it the whole time. Because every token attends to every other token, the very mechanism that makes these models reason also makes cost balloon: double the context and you roughly quadruple the compute. That O(L²) penalty quietly capped how long a prompt could get and how cheap a token could be, year after year. In 2026, the cap broke. A new sparse-attention design took home the ACL 2025 Best Paper award, then landed inside a frontier model that promptly slashed its own API prices by more than half[1][3]. A linear-attention model stretched to a 4-million-token context at 456B parameters[4]. State space models hit transformer-grade quality on half the memory[7]. Here’s how the field finally stopped paying the quadratic tax, with the receipts, and one viral rumor put to rest.

The 2026 thesis in one figure. Left: despite being sparse, DeepSeek’s NSA beats full attention on average across general, long-context, and reasoning benchmarks. Right: at 64k tokens it is several times faster in every phase, decode, forward, and backward. Source: Yuan et al., NSA (ACL 2025 Best Paper).

1. Introduction: The Wall Everyone Built On

“Attention Is All You Need” gave us a model where, to compute the representation of token n, you score it against every other token in the sequence. For a sequence of length L, that's an L×L matrix of scores, so attention is O(L²) in compute and O(L²) in the attention matrix, while the key-value (KV) cache you must keep for generation grows O(L) with every token[15]. Double the context, and the matmul roughly quadruples.

For years that was fine, because context windows were short. But 2025 and 2026 turned long context into the main event: reasoning models that emit tens of thousands of tokens of chain-of-thought, agents that hold hundreds of tool calls in working memory, codebases dumped wholesale into the prompt. Suddenly the quadratic term wasn’t an academic footnote, it was the dominant line on the GPU bill.

The escape didn’t come from one idea. It came from four, attacking the problem from different angles:

• Sparse attention, learn which tokens actually matter and skip the rest (NSA, MoBA, DSA).
• Linear attention, drop the softmax so attention collapses into a fixed-size recurrent state (MiniMax-01’s Lightning Attention, Gated DeltaNet, RWKV-7).
• State space models, replace attention entirely with a selective linear recurrence (the Mamba family).
• Hybrids, keep a few exact-attention layers for precise recall and make everything else cheap (Jamba, Nemotron-H, and, as we’ll see, DeepSeek-V4 itself).

This post walks through all four, the mechanics, the verified benchmark numbers, and where each one is already in production. Along the way we’ll do the math on why the KV cache, not the matmul, is the real villain, and we’ll settle the most over-hyped rumor of the cycle: DeepSeek-V4’s mythical “Engram” memory.

2. Background: It’s the KV Cache That Kills You

The first thing to internalize is that long-context inference has two very different phases, and they bottleneck on different hardware.

Prefill (reading the prompt) is compute-bound. The whole prompt is processed in parallel as big matmuls; on an H100 this runs at 90–95% compute utilization with arithmetic intensity around 200–400 ops/byte[16]. This is the O(L²) phase.

Decode (generating tokens one at a time) is memory-bandwidth-bound. To produce each new token, the GPU must re-read the entire KV cache from high-bandwidth memory (HBM). The tensor cores finish in microseconds and then sit idle, waiting on memory; utilization collapses to 20–40% while the memory bus saturates at 85–95%[16]. This is the phase that dominates the cost of long generations.

And the KV cache is brutal. For a 70B model with grouped-query attention, each token costs about 0.32 MB of KV state. A modest 4,096-token prompt is already 1.34 GB[16]; extrapolate that 0.32 MB/token to a 128K context and the math lands at roughly 40–42 GB of KV cache, which on an 80 GB card leaves almost nothing for the weights. Take it to the extreme and the picture gets absurd: Magic.dev calculated that holding a 100-million-token KV cache for Llama 3.1 405B would require 638 H100s per user, just for the cache[18].

Why is HBM the wall? Because the fast memory is tiny. An A100 has 40–80 GB of HBM at 1.5–2.0 TB/s, but only 192 KB of on-chip SRAM per streaming multiprocessor running at ~19 TB/s, a roughly 10× bandwidth gap[15]. FlashAttention won its fame by tiling attention to minimize HBM round-trips; it makes exact attention faster, but it doesn’t change the O(L²) compute or the O(L) cache. To break those, you have to change the architecture.

3. The Map: Four Families and the “Efficient-Exact” Contrast

A useful way to organize the zoo, drawn from the 2025 survey on efficient attention[14], splits the post-quadratic world into four families, with a fifth group of “efficient but still exact” tricks as the baseline to beat.

The trick that underpins the whole family: a linear-attention/SSM layer can be computed two equivalent ways, a parallel “chunkwise” form for fast training (intra-chunk, Qintra) that passes a single fixed-size state S between chunks (inter-chunk, Qinter) for constant-memory recurrent inference. That duality is what makes these models both trainable at scale and cheap to decode. Source: Efficient Attention Mechanisms for LLMs, survey (2025).

• Efficient-exact (the baseline): FlashAttention, grouped-query attention (GQA), and DeepSeek’s Multi-head Latent Attention (MLA). These keep attention mathematically exact but shrink the cache or the memory traffic.
• Sparse attention: NSA, MoBA, DSA, compute attention over a learned subset of tokens.
• Linear attention: MiniMax Lightning, Gated DeltaNet, RWKV-7, replace softmax with a kernel so attention becomes a recurrence.
• State space models: Mamba, Mamba-2, Mamba-3, a selective linear recurrence that drops attention entirely.
• Hybrids: Jamba, Nemotron-H, DeepSeek-V4, mix a few attention layers into a cheap backbone.

Let’s start with the family that quietly underpins everything DeepSeek does.

4. Efficient-Exact: MLA and the KV-Cache Diet

Before sparsity or linearity, DeepSeek attacked the cache directly. Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 and carried into V3, keeps attention exact but stores far less[10][11].

MLA (left) compresses each token’s keys and values into one small latent vector, the only thing cached, then reconstructs per-head K/V on the fly. A small “decoupled RoPE” component carries position separately. Source: DeepSeek-V3 technical report.

Standard multi-head attention caches a full key and value for every head at every layer. MLA instead down-projects each token into a single shared latent vector (a low-rank joint compression of keys and values) and caches only that. At attention time, up-projection matrices reconstruct the per-head keys and values. Because rotary position embeddings can’t survive that compression cleanly, MLA adds a small “decoupled RoPE” component that carries position separately.

The payoff is large and exact: against DeepSeek’s previous 67B dense model, MLA cut the KV cache by 93.3% and lifted maximum generation throughput 5.76×, while the paper reports accuracy stronger than full multi-head attention[10]. MLA is the foundation the rest of the DeepSeek lineage, including the sparse attention below, is built on top of.

5. Trainable Sparse Attention: NSA and the End of Post-Hoc Hacks

People have tried to sparsify attention for years, mostly after training: evict low-scoring tokens (H2O), keep only recent ones (StreamingLLM), or estimate which pages matter (Quest). The trouble is that the model was optimized for dense attention and never learned to live with a sparse pattern, so quality degrades, and these tricks usually only help decode, not training or prefill[1].

2025’s breakthrough was making sparsity native, part of the architecture, learned end-to-end during pretraining. DeepSeek’s Native Sparse Attention (NSA) won the ACL 2025 Best Paper award for exactly this[1][28].

NSA replaces one dense attention with three parallel branches: compression (a coarse, summarized view of the whole context), selection (full-resolution attention on the few blocks that matter most), and a sliding window (recent local tokens). A learned gate blends them per token. Source: Yuan et al., 2025.

5.1 How NSA Works

For every query token, NSA runs three attention branches over the same history at different resolutions:

• Token compression (coarse). Consecutive blocks of keys/values are squashed into single block-level vectors by a learnable MLP, giving a cheap, blurry view of the entire context.
• Token selection (fine). NSA reuses the compression-branch attention scores to rank blocks by importance, then runs exact, full-resolution attention only on the top-n blocks. Crucially, the importance scores come from a softmax (a differentiable quantity), not a hard argmax, so gradients flow and the selection is trainable.
• Sliding window (local). A standard window over the most recent tokens handles local syntax. Isolating it into its own branch stops the model from “shortcutting” through easy local patterns and starving the other two branches of gradient.

A learned gate decides, per token, how much to trust each branch. And because the whole thing is blockwise, it’s co-designed with a hardware-aligned Triton kernel that loads all the query heads in a GQA group together and amortizes the expensive KV fetch across them, fixing the arithmetic-intensity imbalance that sinks naive sparse attention.

NSA’s kernel is group-centric: it loads all query heads sharing a KV group together (outer loop), fetches only the selected sparse KV blocks (inner loop), and computes on fast SRAM (green) instead of HBM (blue). Hardware-alignment is what turns “fewer tokens” into actual wall-clock speedup. Source: Yuan et al., 2025.

5.2 The Numbers

NSA was pretrained as a 27B-parameter MoE (3B active) on ~270B tokens. At a 64k context, versus full attention, it delivered[1]:

• 9.0× faster forward, 6.0× faster backward, and 11.6× faster decoding. The decode number is grounded in memory math: at 64k, NSA loads ~5,632 tokens per attention op versus 65,536 for full attention.
• Better average quality, not just comparable. NSA scored 0.456 average across nine general benchmarks vs 0.443 for full attention (MMLU 0.565 vs 0.567, GSM8K 0.520 vs 0.486), and 0.469 vs 0.437 on LongBench.
• Perfect needle-in-a-haystack retrieval at 64k. The sparsity costs nothing on exact recall.

NSA vs a FlashAttention-2 kernel across context lengths: the speedup widens as sequences grow, because the selected token budget stays roughly fixed while full attention keeps paying O(L²). Source: Yuan et al., 2025.

5.3 MoBA: Mixture-of-Experts, but for Attention Blocks

Moonshot AI’s MoBA arrived the same month with a lighter-touch take[2]. It applies the MoE routing idea to attention: split the context into blocks, score each block by the dot product between the query and the block’s mean-pooled keys, and attend only to the top-k blocks (plus the current one, always). The elegant part is that MoBA shares identical parameters with full attention, so you can flip any layer between sparse and dense with no surgery, and train mostly-sparse while keeping a few full-attention layers for safety.

MoBA routes each query to its top-k most relevant blocks of the KV cache, like MoE routing to expert FFNs, while causally masking the future. It’s the long-context attention backend behind Moonshot’s Kimi models. Source: Lu et al., MoBA, 2025.

Applied to Llama 3.1 8B extended to a 1M-token context, MoBA matched full attention within a point or two (RULER@128K of 0.7818 vs 0.7849) while delivering roughly a 6.5× attention speedup at 1M tokens and up to 16× at 10M[2]. It’s deployed in production behind Kimi’s long-context requests.

5.4 DSA: Sparse Attention Goes Frontier

The real validation came in September 2025, when DeepSeek shipped DeepSeek Sparse Attention (DSA) in the production model DeepSeek-V3.2-Exp[3]. DSA refines NSA’s idea: a lightweight “lightning indexer” (a few-head, ReLU-gated, FP8 scoring module) ranks all preceding tokens against the current query, then a fine-grained selector keeps the top-2,048 tokens, and core attention runs only over those. That drops the core attention cost from O(L²) to O(Lk) with k « L. The key evolution over NSA is that selection is now token-level rather than block-level.

DeepSeek-V3.2-Exp (DSA) vs V3.1-Terminus (full attention): benchmark parity. MMLU-Pro held at 85.0; AIME 2025 actually rose 88.4 → 89.3. The efficiency was effectively free. Source: DeepSeek official announcement.

The headline wasn’t a benchmark, it was a price tag. On the same day, DeepSeek cut its API prices by more than 50%, with input (cache-miss) dropping from $0.56 to $0.28 and output from $1.68 to $0.42 per million tokens[3]. Sparse attention had gone from a research curiosity to a line-item discount passed straight to developers.

The DSA dividend: DeepSeek’s own price comparison, more than 50% off across input and output tokens, effective the day V3.2-Exp shipped. Source: DeepSeek API docs.

6. Linear Attention: Throw Away the Softmax

Sparse attention still computes softmax attention, just over fewer tokens. Linear attention takes a more radical swing: get rid of softmax entirely.

6.1 The Associativity Trick

Standard attention computes softmax(QKᵀ)V. The QKᵀ term is the L×L matrix that costs you O(L²). Replace the softmax with a kernel feature map φ, and the score becomes φ(q)·φ(k), which is just a dot product, no normalization that couples all tokens together. Once softmax is gone, matrix-multiplication associativity lets you re-bracket the computation:

Quadratic:  (φ(Q) φ(K)ᵀ) V    → build the L×L matrix first  → O(L²d)
Linear:      φ(Q) (φ(K)ᵀ V)    → build the small d×d matrix first → O(Ld²)

Since the head dimension d is far smaller than the sequence length L, the linear form is, well, linear in sequence length. Better still, the model only ever needs the running sum S = Σ φ(kᵢ)vᵢᵀ, a fixed-size d×d state matrix. Generation becomes an RNN: update the state with each new token, read out the output, never store a growing KV cache. Decode is O(1) per token in memory.

The catch is fundamental. A fixed-size state is a lossy, compressed memory, every token is summed into the same matrix, so old information can be overwritten (“memory collision”). Softmax attention keeps every key and value exactly, which is why it has perfect recall and why it costs O(L²). Linear attention = bounded memory, lossy recall; softmax = exact recall, unbounded cost. The entire modern family is a series of increasingly clever answers to one question: how do you make that fixed state forget and update intelligently?

6.2 MiniMax-01: Linear Attention at 456B Parameters

The proof that this scales to a real frontier model is MiniMax-01 (456B total parameters, 45.9B active)[4]. Its Lightning Attention is an I/O-aware, chunked linear attention: within a chunk it uses the cheap quadratic form, and across chunks it carries state with the linear recurrence, getting linear scaling and tensor-core-friendly matmuls at once.

But MiniMax made a telling design choice: pure linear attention struggles with precise retrieval, so they interleave a softmax layer after every 7 lightning layers, a 7:1 ratio across 80 layers. That hybrid instinct, cheap layers for the bulk, exact layers for recall, recurs everywhere in 2026.

MiniMax-01’s needle-in-a-haystack retrieval stays near-perfect out to a 4-million-token context, trained at 1M and extrapolated to 4M, roughly 20–32× the window of GPT-4o or Claude 3.5 Sonnet at the time. Source: MiniMax-Text-01 model card.

The results held up on real benchmarks, MMLU 88.5, and a RULER long-context score around 0.910 at 1M tokens where softmax baselines fall apart, all while keeping over 75% model FLOPs utilization on the hardware[4]. The follow-up, MiniMax-M1, turned this into a reasoning model that reportedly consumes only ~25% of DeepSeek-R1’s FLOPs at a 100K-token generation length, exactly the regime where cheap long context matters most[27].

6.3 The Family: Gated DeltaNet and RWKV-7

The research frontier of linear attention is all about smarter state updates. NVIDIA’s Gated DeltaNet combines two complementary operations: a gate (a data-dependent decay that can quickly erase the whole state, Mamba-2’s strength) and the delta rule (a targeted read-modify-write of just the slot tied to the current key, DeltaNet’s strength)[8]. The combination dominates: at 1.3B parameters it beat Mamba-2 on language-modeling perplexity (16.42 vs 16.56 Wikitext) and crushed it on retrieval (91.8% vs 30.4% on single-needle passkey). Its design has since been adopted into production hybrids, Qwen3-Next and the Qwen3.5 line use a 3:1 gated-delta-net-to-attention layout.

Gated DeltaNet fuses Mamba-2-style gating (fast forgetting) with the delta rule (precise, targeted updates), then offers hybrid variants (H1, H2) that add a sprinkle of sliding-window attention for recall. Source: NVIDIA Gated DeltaNet.

RWKV-7 “Goose” pushes further with per-channel gating and in-context learning rates, and proves a theoretical point: it can track state and recognize all regular languages while staying parallelizable to train, something plain transformers provably cannot do. Its 2.9B model, trained fully open, set a new 3B-class multilingual state of the art[9].

7. State Space Models: The Mamba Lineage

Coming from a different tradition, signal processing rather than attention, state space models (SSMs) arrived at the same place: a linear recurrence with a fixed-size state and no KV cache.

7.1 Mamba

An SSM maintains a hidden state hᵗ = Ā·hᵗ₋₁ + B̄·xᵗ and reads out yᵗ = C·hᵗ. Classic SSMs were time-invariant (A, B, C fixed), which made them fast but content-blind. Mamba's insight was to make B, C, and the timestep Δ input-dependent, so the model can selectively remember or forget based on what it's reading, with a hardware-aware parallel scan to keep training fast[5].

Mamba’s selective SSM: input-dependent parameters let the model decide what to keep, and a hardware-aware scan keeps the state in fast SRAM. Linear-time training, constant-memory decode, no KV cache. Source: Gu & Dao, Mamba.

Mamba-2.8B matched transformers twice its size (63.3% average vs Pythia-2.8B’s 59.1%), ran 5× faster at inference with no KV cache, and on a synthetic induction-heads task, trained at length 256, it generalized to 1 million tokens, 4000× longer than training[5].

7.2 Mamba-2 and the Duality That United Everything

Mamba-2 delivered a conceptual bombshell: State Space Duality (SSD), a proof that SSMs and attention are two computational forms of the same object[6]. A selective SSM can be written as a structured matrix transform that is mathematically equivalent to a masked linear attention. This is why “SSM” and “linear attention” have effectively merged into one family in 2026, they differ mainly in how the state decays and updates. Practically, SSD let Mamba-2 train 2–8× faster than Mamba-1 and carry a much larger state (16 → 64–256), improving associative recall.

7.3 Mamba-3

The newest entry, Mamba-3 (ICLR 2026), brings three upgrades[7]: a second-order trapezoidal discretization (more accurate state updates, letting it drop the short-convolution layer everyone thought was essential), complex-valued state updates that implement data-dependent rotations (giving it true state-tracking, parity, modular arithmetic), and a MIMO formulation that turns the state update into dense matmuls that saturate tensor cores.

The headline result: Mamba-3 matches Mamba-2’s perplexity at half the state size, so the same quality at half the decode latency. The MIMO variant pushes the Pareto frontier further still. Source: Mamba-3 (ICLR 2026).

Concretely, at 1.5B scale Mamba-3 MIMO hit 10.24 perplexity vs Mamba-2’s 10.47, added +1.8 points of downstream accuracy, and decoded faster (0.156ms vs 0.203ms per token). On state-tracking it’s a different universe: 100% on parity where Mamba-2 scores 0.9%[7].

8. Why Pure Models Lose, and Hybrids Win

Here’s the inconvenient truth that shapes every production deployment: a fixed-size state is a hard information bottleneck. Attention keeps every past token addressable, so it’s unbeatable at exact recall, verbatim copying, and retrieval among many distractors. A formal result, “Repeat After Me,” proved that a 2-layer transformer can copy strings of exponential length while any fixed-state model is fundamentally capped[17]. Mamba-3’s own numbers confirm it: pure Mamba-3 manages only ~34% on a 4k needle test, but a 5:1 hybrid jumps to 100%[7].

So the production answer is almost never “pure.” It’s a hybrid: keep a small number of attention layers for precise recall and make everything else a cheap linear/SSM layer. The typical recipe is roughly one attention layer for every 6–12 cheap layers.

NVIDIA’s Nemotron-H hybrids (mostly Mamba-2, ~1 attention layer per 11–12) sit on a better accuracy-vs-throughput frontier than pure transformers: the 47B is 2.9× faster than Qwen-2.5–72B at long context. Source: NVIDIA Nemotron-H.

The hybrid roster is now deep:

• Jamba / Jamba-1.5 (AI21): 1 attention layer per 7 Mamba layers, plus MoE. Jamba-1.5-Large holds a 256K context with a 9 GB KV cache where Llama-3.1–70B needs ~80 GB, and was the first open model with a genuine 256K effective context (RULER 95.7)[12].
• NVIDIA Nemotron-H: the majority of attention layers replaced by Mamba-2 (only ~8% of layers stay attention, roughly 1 in 12); the 56B beats Qwen-2.5–72B on MMLU-Pro while running far faster[13].
• NVIDIA Nemotron-3 (June 2026): the newest hybrid Mamba-Transformer MoE line; the 550B-total / 55B-active “Ultra” carries a 1M-token context at up to ~5.9× the inference throughput of comparable open models[30].
• Falcon-H1, IBM Bamba, Zamba2, Hymba: a wave of parallel- and sequential-hybrid designs all chasing the same 2–3× throughput win from shrinking the KV cache.

9. DeepSeek-V4: Rumor vs. Reality

No 2026 long-context story is complete without addressing the rumor that ate the internet. For months, secondary blogs breathlessly described DeepSeek-V4’s “Engram” architecture, an “O(1) memory” using “deterministic hash-based lookups” that supposedly made a 1M-token context “cost roughly the same as 128K” and hit “97% needle-in-a-haystack at 1M”[24].

Then DeepSeek-V4 actually shipped (preview, April 24, 2026), and the real technical report tells a different story[23]. Searching the official 58-page report, the words “Engram” and “O(1)” appear zero times, and there is no constant-time hash-based memory mechanism anywhere in it. (The report does use a “hash routing” trick, but that’s for assigning tokens to MoE experts, nothing to do with attention or memory.)

DeepSeek-V4-Pro’s official performance chart. The real story isn’t a magic memory, it’s a disciplined hybrid attention design delivering frontier scores at a fraction of V3.2’s inference cost. Source: DeepSeek-V4-Pro model card.

What V4 actually uses is a hybrid attention architecture, alternating two mechanisms that build directly on everything above:

• CSA (Compressed Sparse Attention): compresses the KV cache ~4× and applies lightning-indexer-style top-k selection over compressed blocks, NSA and DSA’s lineage, made denser.
• HCA (Heavily Compressed Attention): compresses the KV ~128× and runs dense attention over the compressed blocks.
• Plus manifold-constrained hyper-connections replacing residuals, DeepSeekMoE, and the Muon optimizer, trained on 32T+ tokens.

The verified efficiency is genuinely impressive, just earthly: at 1M tokens, V4-Pro needs only 27% of the per-token inference FLOPs and 10% of the KV cache of V3.2[23]. And the retrieval is reported honestly: on the OpenAI MRCR 8-needle test, accuracy is strong through 128K but degrades to about 0.59 at 1M, the opposite of the “97% at 1M” myth. The lesson for 2026: the real architectures are remarkable enough that you don’t need to invent fairy tales. Always check the primary source.

10. The Benchmark Reality Check: “1M Context” Is Marketing

Every other model now claims a million-token (or ten-million-token) window. Almost none can actually use it. The NVIDIA RULER benchmark exposed this by extending needle-in-a-haystack with multi-hop tracing and aggregation tasks, then defining an “effective” length as where a model still clears a reasonable accuracy bar[19].

Even models built explicitly for long context fade as the window grows: RULER accuracy for the LargeWorldModel family (trained from 128K up to 1M) slides downward with sequence length, and the shorter-trained variants drop off first, the visible gap between a model’s advertised window and its usable one. Source: Hsieh et al., RULER.

The effective-vs-claimed gap is stark:

• GPT-4: 64K effective vs 128K claimed.
• Llama-3.1–70B: 64K vs 128K. Llama-3.1–8B, Qwen2–72B, Command-R+: 32K vs 128K.
• Yi-34B: 32K vs 200K claimed.
• Gemini-1.5-Pro and Jamba-1.5-Large: among the very few that clear 128K effective at all[19].

The canonical needle-in-a-haystack heatmap (Gemini 1.5), here across text, audio, and video out to ~10M tokens. Green is a hit, and it looks like a solved problem, which is exactly why benchmarks like RULER, which test multi-hop and aggregation, matter: real comprehension lags simple retrieval badly. Source: Gemini 1.5 technical report.

The implication for this whole post: making long context cheap is only half the battle. Making it usable, high recall and real reasoning across the full window, is the harder, still-unsolved half. The architectures above lower the cost; they don’t automatically raise the quality ceiling.

11. The 2026 Production Landscape

Who’s actually shipping what:

• Google Gemini (up to 2M context): dense long-context MoE; the recall leader, the only model comfortably above 128K effective on RULER.
• Meta Llama 4 Scout (10M claimed): “iRoPE”, interleaved no-position global layers with chunked-RoPE local layers and inference-time attention scaling[20].
• Anthropic Claude (1M): launched at a long-context premium in August 2025, then dropped the premium entirely at general availability in March 2026, efficiency gains passed to price[21].
• DeepSeek (V3.2-Exp → V4): DSA then CSA/HCA hybrid; the most aggressive on cost.
• MiniMax (1M train / 4M infer): Lightning Attention, the linear-attention flag-bearer, now joined by MiniMax-M3 (June 2026) and its new MSA (MiniMax Sparse Attention), which selects blocks over uncompressed KV (the inverse of DeepSeek’s compress-then-select) at a reported >9× prefill and >15× decode speedup at 1M tokens[29].
• Qwen2.5–1M: length extension plus sparse-attention inference for 3–7× faster 1M prefill, RULER 93.1[22].
• Kimi / Moonshot: MoBA in production. Magic.dev: a proprietary non-attention “LTM”; its LTM-2-mini claims a 100M-token context at ~1000× lower per-token cost than 405B attention[18].

12. Comparing the Approaches

Full attention vs sparse vs linear vs SSM vs hybrid, property by property:

• Decode complexity. Full: O(L) per token (grows). Sparse: O(k), fixed budget. Linear/SSM: O(1), fixed state. Hybrid: O(1) plus a few O(L) layers.
• KV cache. Full: grows linearly, the cost driver. Sparse: full cache, fewer reads. Linear/SSM: none (fixed state). Hybrid: small (only the attention layers).
• Exact recall. Full: perfect. Sparse: near-perfect. Linear/SSM: lossy (the weak spot). Hybrid: strong, that’s the point.
• Trains from scratch? Sparse (NSA/MoBA/DSA): yes, natively. Linear/SSM/Hybrid: yes. Post-hoc methods (H2O, Quest): no, inference-only.
• Best at. Sparse: drop-in long-context for transformer-shaped models. Linear/SSM: maximum throughput, edge/streaming. Hybrid: the practical frontier default.
• In production. Sparse: DeepSeek, Kimi. Linear: MiniMax. SSM/Hybrid: Jamba, Nemotron-H, Falcon-H1, DeepSeek-V4.

13. Limitations and Open Challenges

• Cheap ≠ usable. RULER shows effective context lagging claimed context badly. Lowering cost doesn’t fix recall; that’s a separate, open problem.
• Lossy memory is real. Pure linear/SSM models still trail attention on copying and multi-distractor retrieval, hence the universal retreat to hybrids.
• Approximate selection can miss. Sparse attention’s top-k can drop a token that mattered; block-level selection papers over fine-grained dependencies.
• Tooling lag. The serving stack (vLLM, SGLang, TensorRT-LLM) is deeply optimized for standard attention; custom kernels for NSA-style selection or Mamba scans are catching up but aren’t at parity.
• Hype outruns evidence. The Engram saga is a reminder that secondary blogs invent specs. Trust primary reports.

14. The Road Ahead

• Hybrid is the new default. Expect the next round of frontier models to be mostly-linear/SSM with a sparse-attention or full-attention minority, DeepSeek-V4’s CSA/HCA blend is a template.
• Sparse + compressed KV stack. MLA-style compression underneath DSA-style selection underneath a hybrid layout, each technique composes with the others.
• Native sparse training everywhere. NSA proved you can pretrain with sparsity and gain quality; that lesson will generalize.
• Reasoning is the demand driver. Test-time compute, long chains of thought, hundreds of tool calls, makes cheap long context an economic necessity, not a nicety. The labs that make decode cheap win the agent era.
• Better long-context evals. RULER, LongBench v2, MRCR, and HELMET are converging into real scoreboards; “1M tokens!” as a marketing line is on borrowed time.

What’s New Since (mid-2026 and counting)

The clearest sign of how fast this is moving: two of the systems in this post shipped while it was being written. On June 1, 2026, MiniMax released MiniMax-M3 with a new MiniMax Sparse Attention (MSA), which flips DeepSeek’s recipe by selecting blocks over uncompressed KV, and claims roughly >9× faster prefill and >15× faster decode at a million tokens versus its predecessor[29]. Three days later, on June 4, NVIDIA dropped Nemotron-3 Ultra, a 550B-parameter open hybrid Mamba-Transformer MoE aimed squarely at long-running agents, with a 1M-token context and up to ~5.9× the throughput of comparable open models[30]. Meanwhile Qwen’s 3.5 line leans on Gated DeltaNet, Kimi K2.6 rides MLA, and DeepSeek-V4 is still only a preview, with a stable release expected later in the year.

In other words: by the time you read this, the specific leaderboard will have shifted again. The direction won’t. Sparse, linear, SSM, and hybrid attention are no longer research bets, they’re the default toolkit for anyone shipping long context, and the cadence of releases is now measured in days, not quarters.

15. Conclusion

For most of the deep-learning era, “transformer” meant “pay O(L² ) and like it.” That contract is broken. In 2026, long context is a portfolio: trainable sparse attention (NSA, MoBA, DSA) for transformer-shaped models, linear attention (MiniMax Lightning, Gated DeltaNet) for maximum throughput, state space models (Mamba-3) for constant-memory decode, and hybrids (Jamba, Nemotron-H, DeepSeek-V4) as the practical frontier default. The KV cache, not the matmul, was always the real enemy, and every one of these techniques is, at heart, a way to stop re-reading gigabytes of memory for every token.

The numbers tell the story: 11.6× faster decode from NSA, 50%+ off DeepSeek’s API the day DSA shipped, a 4-million-token window from a linear-attention model, Mamba-3 matching its predecessor at half the memory. None of this means softmax attention is dead, it still has the best recall and a decade of tooling. But the autoregressive transformer’s quadratic monopoly is over. The teams that ship the next generation of agents and reasoning systems won’t be the ones with the biggest attention matrices. They’ll be the ones who figured out how to stop computing most of it.

References & Further Reading

1. Yuan et al., Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, arXiv:2502.11089 (ACL 2025 Best Paper).
2. Lu et al., MoBA: Mixture of Block Attention for Long-Context LLMs, arXiv:2502.13189 (Moonshot AI / Kimi).
3. DeepSeek, DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention (DSA), 29 Sept 2025.
4. MiniMax, MiniMax-01: Scaling Foundation Models with Lightning Attention, arXiv:2501.08313.
5. Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752.
6. Dao & Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2), arXiv:2405.21060.
7. Lahoti et al., Mamba-3: Improved Sequence Modeling using State Space Principles, arXiv:2603.15569 (ICLR 2026).
8. Yang, Kautz & Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, arXiv:2412.06464 (NVIDIA).
9. Peng et al., RWKV-7 “Goose” with Expressive Dynamic State Evolution, arXiv:2503.14456.
10. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient MoE Language Model, arXiv:2405.04434 (Multi-head Latent Attention).
11. DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437.
12. AI21 Labs, Jamba-1.5: Hybrid Transformer-Mamba Models at Scale; arXiv:2408.12570.
13. NVIDIA, Nemotron-H: A Family of Hybrid Mamba-Transformer Models; arXiv:2504.03624.
14. Efficient Attention Mechanisms for Large Language Models: A Survey, arXiv:2507.19595.
15. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, arXiv:2205.14135.
16. Towards Data Science, Prefill is Compute-Bound, Decode is Memory-Bound (KV-cache and arithmetic-intensity figures).
17. Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, arXiv:2402.01032 (ICML 2024).
18. Magic.dev, 100M Token Context Windows (LTM-2; the 638-H100/user calculation).
19. Hsieh et al., RULER: What’s the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654 (NVIDIA).
20. Meta AI, The Llama 4 Herd (iRoPE, 10M context).
21. Anthropic, Claude Sonnet now supports a 1M token context window.
22. Qwen Team, Qwen2.5–1M: Deploy Your Own Qwen with 1M-Token Context; arXiv:2501.15383.
23. DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (official report & model card; CSA/HCA hybrid attention), preview 24 Apr 2026.
24. Example of the unverified “Engram / O(1) memory” claims (secondary, not in any DeepSeek primary source): “DeepSeek V4 Engram Architecture: The O(1) Memory Revolution” (vertu.com).
25. MiniMax, MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention, arXiv:2506.13585.
26. ACL 2025, Best Paper Awards (NSA).
27. MiniMax, MiniMax-M3 with MiniMax Sparse Attention (MSA), 1 June 2026 (1M context, block selection over uncompressed KV; >9× prefill and >15× decode speedup at 1M vs M2).
28. NVIDIA, Nemotron-3 Ultra: an open 550B hybrid Mamba-Transformer MoE for long-running agents, 4 June 2026.

Connect

If you found this useful, follow me here on Medium for more deep dives on LLM architecture, quantization, retrieval, and inference. Building something with long context or curious which of these to bet on? I’d love to hear about it.

• Email: abdullahramzan120@gmail.com
• GitHub: github.com/buzzgrewal

Tags

#QuadraticAttention #SparseAttention #NSA #NativeSparseAttention #DeepSeek #MoBA #DSA #DeepSeekSparseAttention #DeepSeekV4 #Engram #MLA #MultiheadLatentAttention #LinearAttention #MiniMax #LightningAttention #Mamba #Mamba2 #Mamba3 #StateSpaceModels #SSM #GatedDeltaNet #RWKV #Jamba #NemotronH #HybridLLM #LongContext #MillionTokenContext #KVCache #FlashAttention #RULER #Transformers #AttentionMechanism #LLM #LLMs #LargeLanguageModels #GenerativeAI #GenAI #AI #ArtificialIntelligence #MachineLearning #ML #DeepLearning #NLP #AIInfrastructure #MLOps #AIEngineering #ModelArchitecture #Inference #LLMInference #AI2026 #FutureOfAI #TechBlog #DeepDive #AIExplained #Qwen #Llama4 #Gemini #Claude #Anthropic #Moonshot #Kimi #NVIDIA #AI21 #arxiv #AIResearch #OpenSource