Big ideas.
Small memory footprint.LLM VRAM Calculator: What Models Can Your GPU Run?

At Q4_K_M with a moderate context (4K to 8K), most 7B and 8B models land around 5 to 8 GiB total. An 8GB card can run them if you keep context in check. FP16 still wants roughly 14 to 16 GiB of weights before cache. Pick the model and Q4 in the calculator to see the breakdown.

Q4_K_M weights for a dense 70B are already in the ~40 GiB class. With short context and overhead, plan on about 45 to 50 GiB on-GPU. FP16 is ~140 GiB and is data-center territory. A single 24GB card cannot hold a usable Q4 70B without heavy CPU offload.

Not fully on-GPU at Q4 quality. A 4090 or 3090 (24GB) can hybrid-offload 70B, but expect slow single-digit tok/s. Fit is not the same as fast. Practical paths: dual 24GB, a 48GB+ card, a high-memory Mac, a smaller MoE, or a cloud GPU.

Map by VRAM, not TFLOPS. About 8GB covers 7B/8B Q4; 12GB covers 13B/14B; 16 to 24GB covers many 20B to 32B and strong MoE options; ~48GB total is the common consumer path for dense 70B Q4. Always include KV cache in the fit check, then confirm with the calculator.

Yes. After model size, quant is the biggest lever. FP16 is ~2 bytes per parameter; Q4_K_M is roughly 0.5 to 0.6. A 7B drops from ~14 GiB of weights at FP16 to ~4 to 5 GiB at Q4. Weight quant does not shrink the KV cache unless you also quantize the cache.

Yes. The KV cache grows with tokens and concurrent sequences. At 32K or 128K it can rival or exceed the weight size. A model that fits at 4K can OOM at 32K. Set max context to what you actually use, or quantize the KV cache before buying a bigger GPU.

The KV cache stores attention keys and values so the model does not recompute the whole prompt every token. It lives in VRAM, scales with context length (and batch/chats), and is why GGUF file size is not the VRAM you need. GQA/MLA architectures shrink it versus naive multi-head attention.

VRAM is GPU memory and is what makes local inference fast. System RAM holds offloaded layers, the OS, and CPU-only paths. More RAM lets you run a model that does not fit, but PCIe/RAM bandwidth tanks tok/s. Buy VRAM for speed; use RAM as a fallback, not a substitute.

Yes, with GGUF/llama.cpp-style loaders if VRAM plus RAM covers the model. Speed often drops to a few tok/s, which is fine for batch work and poor for snappy chat. Prefer a smaller quant or a model that fits fully on GPU if you care about tokens per second.

Treat 8B Llama 3.x like other 8B dense models: Q4 with modest context usually fits in 8GB with a little care, and is comfortable on 12GB+. FP16 still wants a 16GB-class card. Select the exact Llama 3.x 8B artifact above to see weights versus KV for your context.

Plan ~40 GiB of weights plus cache and overhead, so about 45 to 50 GiB on-GPU for a usable Q4_K_M 70B. Dual 24GB (2x 3090/4090) is the usual consumer build. One 24GB card will offload and feel slow. Confirm with the 70B Q4_K_M row in the calculator.

Enough to learn and to run 7B/8B at Q4 with short context. Tight for modern coding/agent workloads and for long context. If you are buying new, 12 to 16GB+ is a safer floor. 8GB laptops can work with tiny models or hybrid offload, not dense 13B+ at speed.

Yes for 7B to 14B Q4 with room for context, and for some MoE setups with care. Dense 32B and 70B are offload territory. For local LLMs, 12GB almost always beats 8GB of faster compute (3060 12GB over 3070 8GB).

There is no single winner: 24GB (used 3090 or 4090) is the popular single-card sweet spot for ~27B to 32B Q4. Dense 70B wants ~48GB total. High-memory Macs win on capacity, NVIDIA wins on CUDA ecosystem and tok/s. Use the Buy and Rent tabs after you size the model.

Pick the highest quant that still fits with context headroom. Q4_K_M is the community sweet spot for chat and coding. Use Q5/Q6/Q8 when you have spare VRAM. Avoid Q2/Q3 unless nothing else fits. Quality loss shows first on hard reasoning, not on everyday chat.

Yes. Unified memory is one pool, so large quants can load without a discrete VRAM ceiling. Leave headroom for macOS. Rough tiers: 16GB for small models, 32GB for comfortable mid-size, 64GB+ before 70B-class is practical. Bandwidth still sets tok/s; a 64GB Mac can load what a 24GB NVIDIA cannot.

Ollama and llama.cpp still need weights plus KV plus overhead. File size is only the weights. Partial GPU offload (`n-gpu-layers`) changes how much lands in VRAM versus RAM. After you pick a model here, load it and measure with `nvidia-smi` or `ollama ps`, because context growth is real.

The GGUF is weights only. KV cache, activations, the desktop/display reservation, fragmentation, and long prompts all sit on top. Leave headroom, lower context, quantize the KV cache, or pick a smaller quant. Fitting the file size is not the same as fitting a conversation.

Buy when you run many hours a week and want privacy and always-on. Rent for bursts, 48 to 80GB experiments, or to test a 70B setup before you spend on dual-GPU hardware. A hybrid is common: a modest local card for daily 8B to 32B work, cloud when you need more VRAM.