Architecture & Memory Deep Dive
What is KV Cache? Sizing, Formulas & VRAM Impact
Why your local LLM runs out of memory on long prompts, how the attention mechanism caches tokens, and how Grouped-Query Attention (GQA) and DeepSeek's MLA slashed memory requirements by up to 90%.
Interactive KV Cache Calculator
See how architecture, context tokens, and precision dictate real GPU memory consumption.
32 layers · 32 query heads · 8 KV heads (4:1 GQA group)
Multi-user servers (like vLLM) multiply the KV cache by active parallel requests.
Formula: 2 (Keys + Values) × 32 layers × 8 KV heads × 128 head dim × 32,768 tokens × 2 bytes
Because this model uses GQA (8:1 ratio), this KV cache is 4x smaller than it would be with legacy MHA (which would have required 16.0 GiB).
1. Why Does Autoregressive Generation Need a KV Cache?
When an LLM generates a response, it produces one token at a time in an autoregressive loop. To predict the very next token, the self-attention mechanism must compute how the current token relates to every preceding token in the prompt and response history.
In naive matrix multiplication, generating token #2,000 would require running the entire neural network over tokens 1 through 1,999 from scratch. This would cause generation speed to slow to a crawl, scaling quadratically as $O(N^2)$.
The result: token generation speed remains constant ($O(1)$). But the price you pay is GPU memory consumption that grows linearly with every token processed.
2. The Exact KV Cache Mathematical Formula
For standard transformer architectures (such as Llama 3, Mistral, Qwen, Gemma, and DeepSeek), the size of the KV cache in bytes is given by:
3. The Attention Evolution: MHA vs. GQA vs. MLA
The history of open LLMs over the past two years is largely a fight to prevent the KV cache from overwhelming GPU VRAM.
Ratio: 1 Query Head : 1 KV Head.
Every query head keeps its own independent Key and Value tensor. Extremely memory-intensive: at 32k tokens, Llama 1 7B consumed ~8 GiB just for KV cache.
Ratio: 4 to 8 Query Heads : 1 KV Head.
Used by Llama 3, Mistral, Qwen 2.5, and Gemma. Groups multiple query heads to share a single KV head. Reduces KV cache memory by 75% to 87.5% with zero perceptible loss in reasoning quality.
Ratio: Low-Rank Latent Compression.
Invented by DeepSeek for V2, V3, and R1. Projects Key and Value matrices into a compact 512-dim latent vector before caching. Achieves 90%+ memory reduction compared to standard MHA.
4. How to Quantize the KV Cache (Save 50% to 75% Memory)
If your model fits into VRAM at short context but you need long-context capability (for coding, PDF reading, or multi-turn agent conversations), you can quantize the cache:
Cuts cache memory in half by moving from FP16 (2 bytes) to FP8 (1 byte). Supported on modern NVIDIA Ada Lovelace (RTX 4090) and Hopper (H100) GPUs.
vllm serve model --kv-cache-dtype fp8Quantizes keys and values into 8-bit or 4-bit integer blocks with FlashAttention. Enables 64k+ context on consumer 16GB cards.
llama-cli --cache-type-k q4_0 --cache-type-v q4_0Frequently Asked Questions about KV Cache
When a model first loads into VRAM, only the model weights and runtime context buffers are resident (~5 GB for an 8B Q4 model). As your conversation grows, the KV cache must store the key and value vectors for every single past token. At 32k or 64k tokens, the KV cache can consume an extra 2 GB to 6 GB. If your GPU (such as an 8GB RTX 3060 Laptop or 4060) has no headroom left, CUDA triggers an immediate Out of Memory crash.
For standard Grouped-Query Attention (GQA) and Multi-Head Attention (MHA), the formula is: Bytes = 2 (Keys + Values) × Layers × KV Heads × Head Dimension × Context Tokens × Concurrent Sequences × Bytes Per Element. For 16-bit precision (FP16/BF16), Bytes Per Element is 2. For FP8, it is 1. Divide the total bytes by 1024³ to get GiB.
Instead of caching high-dimensional key and value vectors for dozens of attention heads, MLA compresses the key and value representations into a single low-rank latent vector (rank 512) alongside a decoupled RoPE head vector (rank 64). During generation, the engine caches only 576 dimensions per token rather than 4,096+, allowing massive 128k context windows to fit on a single GPU.
No! Weight quantization only shrinks the static weight matrices of the neural network on disk and in VRAM. By default, the KV cache remains unquantized 16-bit floating-point (FP16). To shrink the KV cache itself, you must explicitly enable KV cache quantization in your inference engine (e.g. FP8 in vLLM or Q8_0/Q4_0 in Ollama / llama.cpp).
In llama.cpp, pass the flags `--cache-type-k q8_0 --cache-type-v q8_0` (or `q4_0` for maximum memory savings). In Ollama, you can enable Flash Attention (`OLLAMA_FLASH_ATTENTION=1`) and configure your Modelfile with quantized context settings.
Ready to calculate the exact VRAM for your model?
Select any real checkpoint in our VRAM Calculator to see its weights, KV cache, and runtime reserve.