The KV Cache Wars: How MLA, PagedAttention & Sparse Attention Let LLMs Serve Millions for Pennies in 2026

16 min readJun 22, 2026

Everyone obsesses over parameter counts and benchmark scores. But the quiet war that actually decides whether an LLM costs cents or dollars to run is being fought somewhere else entirely: over a block of GPU memory called the KV cache. This is the story of that war, and the architectures, PagedAttention, GQA, Multi-head Latent Attention, KV quantization, and 2026’s sparse-attention frontier, that are collapsing the cost of inference.

The arc of the KV cache wars: from a full multi-head cache, to MLA’s compressed latent, to sparse attention that only attends to what matters.

1. Introduction: The Bottleneck Nobody Tweets About

When a new frontier model drops, the conversation is always about intelligence: SWE-bench, AIME, GPQA, context length. Almost nobody talks about the single component that decides whether that model can be served profitably at scale. That component is the KV cache, the growing pile of intermediate attention state that every autoregressive transformer must keep in GPU memory while it generates.

Here’s the uncomfortable truth of 2026 LLM serving: during generation, your expensive GPU is mostly waiting on memory, not computing. The KV cache is why. It’s why your context window has a price, why long conversations get throttled, and why two models with identical benchmark scores can differ 10x in serving cost.

Over the last three years, a remarkable wave of engineering has attacked this bottleneck from every angle, OS-style memory paging, attention-head sharing, low-rank compression, quantization, cross-layer reuse, and now native sparse attention. I’ll call it what it is: the KV cache wars. This post walks through each front, with the real numbers from the primary papers.

2. What the KV Cache Actually Is

A transformer generates text one token at a time. To produce token n+1, the attention mechanism needs the keys and values of every previous token. The naive approach would recompute them from scratch every step, which is quadratically wasteful. So instead, every model caches them: compute each token’s K and V once, store them, and reuse them for every future step. That store is the KV cache.

*Prefill computes the keys and values for all input tokens in parallel; decode appends one new K/V per step and reuses the whole cache. Source:* *NVIDIA Developer Blog*.

The cache is fantastic for compute, it turns generation from quadratic to linear, but it creates a memory monster. For standard multi-head attention (MHA), the cache size is:

kv_bytes = 2 × bytes_per_element × n_layers × d_model × seq_len × batch_size

The factor of 2 is for keys and values. Read that formula carefully: the cache grows linearly with sequence length and linearly with batch size[1]. Want to serve more users at once (bigger batch)? The cache grows. Want longer context? It grows. At long contexts and high concurrency, the KV cache, not the model weights, becomes the dominant consumer of precious HBM. That single structural fact is the reason every technique in this article exists.

3. Why Decode Is Memory-Bound: The Memory Wall

LLM inference has two phases. Prefill processes your prompt in one big parallel pass, this is compute-heavy and runs once. Decode then generates output tokens one at a time, and it repeats for every single token[2].

Decode is where the pain is. At batch size 1, generating a token is a sequence of matrix-vector products (GEMV) rather than matrix-matrix products (GEMM). As DeepSeek’s own hardware paper puts it, the KV cache “introduces a memory-bound bottleneck because the computation shifts from GEMM to GEMV, which has a much lower compute-to-memory ratio. With modern hardware offering hundreds of TFLOPS, GEMV quickly becomes limited by memory bandwidth, making memory access the primary bottleneck.”[3]

The arithmetic intensity of decode is roughly 1 FLOP per byte moved. A modern GPU can do hundreds of FLOPs per byte of bandwidth, so during decode the compute units sit idle while the hardware drags the entire KV cache out of HBM, every token, for every layer. This is the famous memory wall. Batching helps (it amortizes weight loads across requests), but a bigger batch means a bigger KV cache, which means you hit the memory ceiling sooner. Every optimization below is, at heart, a way to move fewer KV bytes.

4. Front One, Memory Management: PagedAttention & vLLM

The first breakthrough wasn’t about shrinking the cache, it was about stop wasting it. Before 2023, serving systems allocated KV cache in big contiguous chunks sized for the worst-case sequence length. The result was brutal internal and external fragmentation: independent analyses found earlier systems wasted 60–80% of their KV memory.

PagedAttention (Kwon et al., SOSP 2023), the algorithm behind vLLM, borrowed the oldest trick in operating systems: virtual memory and paging[4]. Instead of one contiguous block per request, the KV cache is split into fixed-size blocks (pages) that can live anywhere in memory, tracked by a block table. Tokens map to blocks the way bytes map to pages, and requests behave like processes.

*PagedAttention stores the KV cache in non-contiguous blocks, eliminating fragmentation and enabling KV sharing across requests. Source:* *vLLM blog*.

The payoff: near-zero memory waste (down to under 4%), and flexible KV sharing within and across requests, the foundation for prefix caching and parallel sampling. vLLM reported 2–4x higher throughput than the 2023 state of the art (FasterTransformer, Orca) at the same latency, “more pronounced with longer sequences, larger models, and more complex decoding algorithms.”[4] PagedAttention didn’t make the cache smaller; it made every byte count, and it’s now table stakes in essentially every serving stack.

5. Front Two, Sharing Heads: MQA and GQA

The next idea attacks the cache directly. Standard MHA gives every attention head its own keys and values. But do you really need n independent KV heads? Multi-Query Attention (MQA, Shazeer 2019) said no, share a single KV head across all query heads[5]. That shrinks the K/V tensors dramatically and slashes the memory bandwidth of decoding. The catch: MQA “can lead to quality degradation” and training instability.

Grouped-Query Attention (GQA, Ainslie et al., EMNLP 2023) found the sweet spot[6]. Instead of 1 KV head (MQA) or n KV heads (MHA), use an intermediate number, say 8 KV heads shared among 64 query heads in groups. The paper showed you can even “uptrain” an existing MHA checkpoint into GQA using only ~5% of the original pretraining compute, and get “quality close to multi-head attention with comparable speed to MQA.”

*MHA (left) gives every head its own K/V; MQA (right) shares one K/V head across all; GQA (middle) interpolates with grouped K/V heads. Source:* *Ainslie et al., 2023, Figure 2*.

GQA was the quiet workhorse of the Llama 2/3 and Mistral era, and it’s still the default in most production models. But it’s a blunt instrument: you’re throwing away KV heads. The question DeepSeek asked was whether you could shrink the cache without discarding representational capacity.

6. Front Three, The Compression Breakthrough: Multi-head Latent Attention (MLA)

This is the move that reset the board. Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 (May 2024), doesn’t share or drop KV heads, it compresses them[7].

The core idea is a low-rank joint compression of keys and values. Instead of caching full-dimensional K and V for every head, MLA projects them down into a single small latent vector c_t^KV of dimension d_c, and caches only that. At attention time, the keys and values are reconstructed on the fly via up-projection matrices (which can even be absorbed into neighboring weights, so reconstruction is nearly free). A small decoupled-RoPE key is cached alongside to carry positional information, since rotary embeddings don't play nicely with the compression.

*MLA (right) jointly compresses keys and values into a low-rank latent that is the only thing cached, plus a small decoupled-RoPE key. Source:* *DeepSeek-V2, Figure 3*.

So instead of caching 2 × n_h × d_h elements per token (MHA), MLA caches just (d_c + d_h^R). The numbers are striking:

DeepSeek-V2: set d_c = 4d_h and the decoupled key d_h^R = d_h/2, so the cache equals GQA with only 2.25 groups, but with performance stronger than full MHA. Versus the dense DeepSeek 67B, MLA cut the KV cache by 93.3% and (together with the MoE backbone) boosted maximum generation throughput 5.76x while saving 42.5% of training cost[7].
DeepSeek-V3: with 128 heads of dimension 128, MLA sets d_c = 512 and d_h^R = 64, so it caches just 576 elements per token, versus roughly 40,960 for the equivalent full MHA, about a 98.6% reduction, while “maintaining performance comparable to standard MHA.”[8]

In absolute terms, DeepSeek’s hardware paper puts the per-token cost in BF16 at 70.3 KB for DeepSeek-V3 (MLA) versus 327.7 KB for Qwen-2.5 72B (GQA, 4.66x more) and 516.1 KB for Llama-3.1 405B (GQA, 7.28x more)[3]. That is the single clearest illustration of why MLA is the most important KV innovation of the era: it gives you near-MHA quality at a small fraction of GQA’s memory. It’s no accident that MLA has since been adopted by other frontier labs, Moonshot’s trillion-parameter Kimi K2 uses MLA too[13].

7. Front Four, Quantizing the Cache: KIVI, KVQuant & FP8

Architecture shrinks the shape of the cache. Quantization shrinks the bytes per element. If MLA is structural compression, this front is numerical compression, and the two stack.

The non-obvious finding here is that keys and values want different quantization. KIVI (ICML 2024), a tuning-free 2-bit method, showed that the key cache should be quantized per-channel (because keys have a few large-magnitude outlier channels) while the value cache should be quantized per-token[9]. Get that asymmetry right and you can drop to 2 bits “while maintaining almost the same quality”, for 2.6x less peak memory, up to 4x larger batch size, and 2.35–3.47x throughput.

*KIVI’s key insight: quantize keys per-channel and values per-token. The asymmetry matches where the outliers live. Source:* *KIVI, Figure 1*.

KVQuant (NeurIPS 2024) pushed the same per-channel-key/per-token-value idea further, adding pre-RoPE key quantization and non-uniform datatypes, and reported under 0.1 perplexity degradation at 3-bit, enabling LLaMA-7B inference at up to 1M context on a single A100–80GB and 10M context on an 8-GPU system[10].

In production, the pragmatic default is FP8. vLLM’s 2026 analysis showed FP8 (E4M3) roughly halves KV-cache storage, cuts per-token decode cost to as little as 54% of BF16, and lifts output throughput ~15% under load, all with “at most 1–2 points” of accuracy degradation and 97–98% long-context recall retention[11]. Independent benchmarking found 8-bit KV quantization essentially lossless on MMLU, with TensorRT-LLM seeing up to 1.45x throughput in decode-heavy workloads[12]. FP8 KV cache is now a one-flag option in every major serving engine on Hopper and Blackwell hardware. The 2026 frontier here is NVFP4, NVIDIA’s 4-bit floating-point format with hardware-accelerated micro-block scaling on Blackwell, which halves KV memory again versus FP8, roughly 4x versus BF16, to unlock longer contexts and larger batches with minimal accuracy loss[23].

8. Front Five, Spending Less Per Layer and Per Token

MLA shrinks the cache within a layer. The next set of ideas attack the other two dimensions of that memory formula: layers and sequence length.

Cross-layer sharing: CLA and YOCO

Cross-Layer Attention (CLA, MIT-IBM, May 2024) shares KV across adjacent layers instead of across heads, an orthogonal axis to MQA/GQA, so it stacks on top of them. CLA delivered another 2x KV reduction on top of MQA for just +0.04–0.05 perplexity[14].

YOCO (“You Only Cache Once”, Microsoft, May 2024) is more radical: a decoder-decoder design where a self-decoder builds a single global KV cache that every upper layer reuses via cross-attention[15]. KV memory becomes roughly independent of layer count, about 80x smaller for a 65B model, enough to serve 128K tokens in 1GB of memory where a GQA transformer fits only ~1.6K. Because prefill can early-exit, 512K-token prefill latency dropped from 180 seconds to under 6 seconds.

*YOCO’s decoder-decoder design: the self-decoder produces one global KV cache that all cross-decoder layers reuse, so KV memory stops scaling with depth. Source:* *Sun et al., 2024*.

Bounding the sequence: sliding windows and attention sinks

Mistral 7B popularized sliding-window attention: each token attends only to the previous 4,096, so the KV cache is capped by a fixed-size rolling buffer instead of growing forever. Stacked across 32 layers, information still propagates over a theoretical span of ~131K tokens, while halving cache memory at 8K sequences[16].

StreamingLLM (ICLR 2024) found something delightfully weird: models dump a huge fraction of attention onto the first few tokens regardless of meaning, “attention sinks.” Naive window attention collapses when those tokens are evicted. Keep just the first 4 tokens plus a rolling window, and a model trained on finite context generalizes to 4M+ tokens at constant memory, with up to 22.2x speedup[17].

*StreamingLLM keeps a handful of “attention sink” tokens alongside a rolling window, enabling effectively infinite streaming at constant KV memory. Source:* *Xiao et al., 2023*.

Reusing the cache: prefix caching and chunked prefill

Two serving-level tricks round out this front. Prefix caching stores the KV of shared prompt prefixes, system prompts, RAG context, multi-turn history, so repeated requests skip recomputation entirely (this is the same KV-sharing that PagedAttention made possible). Chunked prefill splits long prompts into chunks co-batched with decode tokens, smoothing latency and reportedly cutting time-to-first-token by up to ~30%. Neither shrinks the cache, but both squeeze far more throughput out of the bytes you’ve already paid for.

9. Front Six, The 2026 Frontier: Sparse Attention

By 2026, the war moved to a new front. The previous techniques still load the whole KV cache every step; sparse attention asks a sharper question: why attend to every past token at all? For a 128K-token context, most tokens are irrelevant to the one you’re generating. If you could cheaply find the few thousand that matter, you’d turn the quadratic O(L²) attention into roughly O(L·k).

Native Sparse Attention (NSA)

NSA (DeepSeek, Feb 2025) made sparsity natively trainable rather than a post-hoc inference hack[18]. Each query runs through three parallel branches, compression (coarse block summaries for global context), selection (the most relevant top-k token blocks), and a sliding window (local context), combined by a learned gate. Crucially, it’s hardware-aligned for Tensor Cores. At 64K context, NSA delivered up to 11.6x faster decoding, 9.0x forward, and 6.0x backward, while matching or beating full attention (7 of 9 benchmarks).

*NSA’s three branches, compressed (coarse), selected (important blocks), and sliding-window (local), are combined by a learned gate, and the whole thing is trainable end to end. Source:* *Yuan et al., 2025, Figure 2*.

DeepSeek Sparse Attention (DSA) and V3.2

Then in late 2025, DeepSeek shipped sparse attention in a flagship model. DeepSeek-V3.2-Exp (released September 29, 2025) introduced DeepSeek Sparse Attention (DSA), built directly on top of MLA[19]. A lightweight “lightning indexer”, a few-head, ReLU-based, FP8 scoring module, ranks past tokens, and the main attention only computes over the top-2,048 selected entries.

*DSA instantiated on top of MLA: a cheap lightning indexer selects the top-k key-value entries (green) so the main attention only runs over what matters. Source:* *DeepSeek-V3.2 technical report*.

The headline wasn’t a benchmark, it was a price tag. With quality “on par with V3.1-Terminus” on both short and long-context tasks, DeepSeek cut its API prices by more than 50% overnight, with output tokens dropping roughly 4x. Combining the structural efficiency of MLA with the long-context efficiency of sparse selection is the clearest signal yet of where serving is heading.

And the line didn’t stop there. In April 2026 DeepSeek shipped DeepSeek-V4 (Preview), which doubles down on the sparse-attention bet rather than reverting to plain MLA[22]. V4’s “novel attention” pairs token-wise compression with DSA: a learned compressor squashes every few tokens into a single KV entry, and sparse selection runs on top, pushing 1M-token context while reportedly using roughly 10% of the KV cache and ~27% of the per-token FLOPs of V3.2. It ships in two tiers, V4-Flash (284B total / 13B active) and V4-Pro (1.6T total / 49B active), and the price war continued: V4-Flash lists at $0.14 input (cache-miss) / $0.28 output per million, with cache hits at $0.0028, roughly 10x cheaper than V3.2-Exp on the cache-hit tier. The trajectory is unmistakable: each generation makes the KV cache smaller and the tokens cheaper.

Linear attention: MiniMax and Kimi

A parallel camp bets on linear attention, which eliminates the growing KV cache entirely by keeping a fixed-size recurrent state instead of caching every token. MiniMax-01 (Jan 2025) interleaved 7 lightning-attention (linear) layers for every 1 softmax layer, scaling to a 456B-parameter MoE with a 1M-token training context (4M at inference) and a claimed 15.6x decode speedup at 1M tokens[20]. Kimi Linear (Oct 2025) introduced Kimi Delta Attention, a hybrid of linear and MLA layers that “reduces KV cache usage by up to 75% and delivers up to 6x decoding throughput at 1M context”[21].

The honest caveat: pure linear/sparse attention can hurt hard multi-hop reasoning, and at least one lab publicly walked back, then re-introduced, an aggressive sparse design after recall regressions. The 2026 consensus is hybrid: mostly-efficient attention with a few full-attention layers to preserve sharp recall.

10. The Economics: What This Does to $/Token

Stack these techniques and the cost curve bends sharply. The cleanest real-world illustration is DeepSeek’s tiered API pricing, where a cache hit (prefix already in the KV cache) costs roughly 1–2% of a cache miss. That’s KV reuse priced directly into the bill. At the V3.2-Exp launch, input cache-hit tokens fell to $0.028 per million, “less than 3 cents”, with output at $0.42 per million.

On the hardware side, MLA plus FP8 lets DeepSeek-V3 sustain roughly 2,200+ tokens/second per H200 GPU in wide expert-parallel deployments, and an 8-GPU B200 node can push tens of thousands of output tokens/second on large models. The mechanism is always the same: smaller KV cache → bigger batches fit in HBM → the memory-bound decode phase finally has enough work to keep the GPU busy → cost-per-token drops. Every front in this war ultimately cashes out as that one chain.

11. The 2026 KV Cache Cheat Sheet

Each front attacks a different term in the KV cache equation. Listed by what it targets:

PagedAttention / vLLM (2023) — memory fragmentation: cuts wasted KV memory from 60–80% to under 4%, for 2–4x throughput.
MQA (2019) — KV heads: a single shared KV head; fast, but some quality loss.
GQA (2023) — KV heads: grouped KV heads deliver near-MHA quality at near-MQA speed.
MLA, DeepSeek (2024) — KV dimension: low-rank latent compression, ~93% smaller, just 576 elements/token in V3.
KIVI / KVQuant / FP8 / NVFP4 (2024–26) — bytes per element: 2-bit to FP8/FP4 cache; 2–4x memory savings, near-lossless.
CLA / YOCO (2024) — layers: cross-layer reuse; 2x on top of MQA, up to ~80x at 65B scale.
Sliding window / StreamingLLM (2023) — sequence length: constant memory, up to 22x speedup.
NSA / DSA (2025) — tokens attended: turns O(L²) attention into O(L·k); up to 11.6x decode at 64K.
Lightning / Kimi Linear (2025) — the cache itself: fixed recurrent state; up to 75% less KV, 6x decode at 1M.
Token compression + DSA, DeepSeek-V4 (2026) — tokens + KV bytes: ~10% of V3.2’s KV cache at 1M context.

12. Limitations & Open Challenges

Sparsity vs. recall. Aggressive sparse and linear attention can quietly degrade multi-hop reasoning and exact recall. Hybrids mitigate it, but the failure mode is subtle and benchmark-dependent.
Quantization isn’t free. 2-bit KV is “almost” lossless, but on outlier-heavy or very long contexts the last point of accuracy matters. Per-channel/per-token schemes add kernel complexity.
Training-time cost. MLA, NSA, and YOCO change the architecture, so they must be baked in at pretraining (or expensively uptrained). They’re not drop-in inference flags like PagedAttention or FP8.
Composition is fiddly. Combining chunked prefill with prefix caching, or sparse attention with FP8 KV, can interact badly; the serving stack matters as much as the algorithm.
Moving target. Prices and throughput figures here are 2025–2026 snapshots on specific hardware. The ratios are durable; the absolute numbers will keep falling.

13. Conclusion

For years the LLM story was “scale the parameters.” The 2026 story is increasingly “scale the serving”, and serving is gated by the KV cache. PagedAttention stopped us wasting memory. GQA and MLA shrank the cache’s shape, with MLA the standout, delivering near-MHA quality at a fraction of GQA’s footprint. Quantization shrank the bytes. CLA, YOCO, sliding windows, and StreamingLLM attacked layers and length. And the 2026 frontier, NSA, DeepSeek Sparse Attention, and linear-attention hybrids, is now attacking the assumption that you must attend to every token at all.

The cumulative effect is a 10–100x collapse in the cost of intelligence per token, and it’s still accelerating. The labs that win the next phase won’t just have the smartest models. They’ll have the cheapest KV cache. That’s the war, and right now, compression is winning.

References & Further Reading

A Survey on Efficient Inference for Large Language Models, arXiv:2402.16363 (KV-cache memory growth; prefill vs. decode).
Same as [1]: prefill runs once, decode repeats per token and is memory-bound.
DeepSeek-AI, Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, arXiv:2505.09343 (GEMM→GEMV bottleneck; KV-per-token Table 1).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023, arXiv:2309.06180.
Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (MQA), arXiv:1911.02150.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023, arXiv:2305.13245.
DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA), arXiv:2405.04434.
DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437 (MLA hyper-parameters; 576 elems/token).
Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024, arXiv:2402.02750.
Hooper et al., KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, NeurIPS 2024, arXiv:2401.18079.
vLLM Team, FP8 KV Cache in vLLM (April 2026).
SqueezeBits, vLLM vs TensorRT-LLM: KV Cache Quantization (FP8 vs INT8).
Moonshot AI, Kimi K2: Open Agentic Intelligence, arXiv:2507.20534 (uses MLA).
Brandon et al., Reducing Transformer KV Cache Size with Cross-Layer Attention (CLA), arXiv:2405.12981.
Sun et al., You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO), arXiv:2405.05254.
Jiang et al., Mistral 7B (sliding-window attention, rolling-buffer KV cache), arXiv:2310.06825.
Xiao et al., Efficient Streaming Language Models with Attention Sinks (StreamingLLM), ICLR 2024, arXiv:2309.17453.
Yuan et al., Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, arXiv:2502.11089.
DeepSeek-AI, DeepSeek-V3.2 Technical Report (DeepSeek Sparse Attention), arXiv:2512.02556; V3.2-Exp announcement, Sept 29, 2025.
MiniMax, MiniMax-01: Scaling Foundation Models with Lightning Attention, arXiv:2501.08313; MiniMax-M1, arXiv:2506.13585.
Moonshot AI, Kimi Linear: An Expressive, Efficient Attention Architecture, arXiv:2510.26692.
DeepSeek-AI, DeepSeek-V4 Preview Release (April 24, 2026); API pricing; see also Simon Willison’s notes.
NVIDIA, Optimizing Inference for Long-Context and Large-Batch Sizes with NVFP4 KV Cache (2026).

Connect

If you found this useful, follow me here on Medium for more deep dives on LLM architecture, inference optimization, and AI infrastructure. Building something that has to serve models cheaply at scale? I’d love to hear about it.

Email: abdullahramzan120@gmail.com
GitHub: github.com/buzzgrewal