Hardware Sizing & Buying Guide
The Ultimate GPU Guide for Local LLMs
Forget gaming benchmarks. Learn why memory bandwidth determines your generation speed, which VRAM tier you need for each model class, and how Apple Silicon compares to Nvidia.
1. The Golden Rule of AI Hardware: Bandwidth Over Compute
In autoregressive LLM inference, the computational bottleneck is rarely raw compute power (TFLOPS). Instead, it is almost always memory bandwidth: the speed at which model weights can be read from VRAM into the compute cores.
During generation, every single weight in the model must be loaded from memory into the GPU cores for every single generated token. A GPU with massive compute but narrow memory bandwidth will sit idle waiting for weights.
Example: A 70B model quantized to Q4 requires approximately 40 GB of VRAM. On an RTX 3090 with 936 GB/s bandwidth, the maximum theoretical speed is 936 / 40 = 23.4 tokens/second. On DDR5 system RAM with 60 GB/s bandwidth, that same model generates at just 60 / 40 = 1.5 tokens/second.
2. The 4 VRAM Capacity Tiers: What Can You Actually Run?
7B to 8B models at Q4_K_M (Llama 3 8B, Mistral 7B, Qwen 2.5 7B). Short context up to 8k tokens.
RTX 3060 12GB (budget champ), RTX 4060 Ti 16GB, or RTX 5060 Ti 16GB.
14B to 32B models at Q4_K_M (Qwen 2.5 32B, Command-R), or 70B models with aggressive quantization (IQ2/Q3).
RTX 3090 24GB (used, unbeatable value), RTX 4090 24GB, or RTX 5090 32GB.
Full 70B to 72B models at Q4_K_M with generous 32k context windows and FP8 KV cache.
Dual RTX 3090s (48GB total), dual RTX 4090s, or Apple Mac Studio M2/M3/M4 Max (48GB to 64GB).
Full Llama 3.3 70B at Q8_0, Qwen 2.5 72B at FP16, DeepSeek R1/V3 distilled models, and Llama 405B at Q2/Q3.
Apple Mac Studio M2/M3 Ultra (128GB to 192GB unified RAM) or workstation quad-GPU rigs.
3. Apple Silicon Mac vs. Nvidia PC: Which Should You Buy?
- Massive VRAM Capacity: Up to 128GB or 192GB unified memory on a single quiet desktop.
- Near Silent & Low Power: Draws 80W to 120W under full load compared to 850W+ for multi-GPU PCs.
- Moderate Bandwidth: 300 to 800 GB/s yields practical generation speeds of 15 to 25 tok/s on 70B.
- Software Ecosystem: Native MLX and llama.cpp Metal backends run great; CUDA/vLLM tools are unavailable.
- Maximum Generation Speed: 1,008 GB/s bandwidth on RTX 4090 produces blazing fast 30 to 60 tok/s.
- Industry Standard: 100% compatibility with CUDA, PyTorch, vLLM, TensorRT-LLM, FlashAttention-2.
- VRAM Wall: Hard 24GB ceiling per consumer card. Running 70B requires dual-GPU builds.
- Power & Heat: Requires 1,000W+ power supplies, large airflow cases, and dedicated circuit capacity.
4. Building a Dual-GPU Workstation for 70B Models
If your goal is to run dense 70B models at full speed on a PC, a dual-card configuration is the most cost-effective path:
Ensure your motherboard supports x8 / x8 bifurcation on CPU lanes, with at least 3 to 4 slot spacing between cards for cooling.
Budget at least a 1,200W or 1,500W Platinum/Titanium PSU to handle the transient power spikes of dual RTX 3090s or 4090s.
Use vLLM with Tensor Parallelism (--tensor-parallel-size 2) or llama.cpp multi-GPU to shard weights across both cards.
Frequently Asked Questions
Because in local LLMs, VRAM capacity is king. A 24GB RTX 3090 can fit 32B models at Q4 or dense 14B models at FP16 with a 32k context window completely on-device. The 16GB card cannot fit 32B models at full speed and is forced to offload layers to slow system RAM. Furthermore, the RTX 3090 features 936 GB/s memory bandwidth across a 384-bit bus, outperforming the RTX 4080's 717 GB/s.
8GB is enough to learn and run 7B/8B models at Q4_K_M with short context (up to 4k to 8k tokens). However, it is tight for modern coding agents, large document analysis, or running a desktop display simultaneously. If buying new, 12GB to 16GB is the recommended minimum floor.
Yes. Apple Silicon's unified memory architecture allows GPU cores to access the entire RAM pool without PCIe bottlenecks. A 64GB Mac Studio can run Llama 3.3 70B at Q4_K_M comfortably (~45 GB total) at 15 to 22 tokens/second. A 128GB or 192GB Mac Studio can run 70B at full FP16 or run large MoE models that would otherwise require $10,000+ in PC hardware.
When a model fits 100% in VRAM, data moves at 500 to 1,000 GB/s. When layers are offloaded to system RAM, data must travel over the PCIe slot (PCIe 4.0 x16 tops out at ~31.5 GB/s, and DDR5 RAM runs at 60 to 90 GB/s). Because generation is memory-bandwidth bound, offloading even a few layers often drops throughput from 45 tok/s to 3 to 6 tok/s.
For 70B models, yes. A single RTX 4090 has 24GB of VRAM and cannot fit a quantized 70B model without severe offload. Dual RTX 3090s give you 48GB of pooled VRAM (via Tensor Parallelism in vLLM or layer splitting in llama.cpp), allowing you to run 70B Q4_K_M completely in VRAM at 25+ tokens/second for roughly the same total price as one 4090.
Need more VRAM than your local GPU?
Rent on-demand RTX 4090 or H100 instances from RunPod and Vast.ai starting at $0.34/hr.
Knowledge & Deep Dives
Local LLM & VRAM Knowledge Hub
In-depth guides, architectural breakdowns, and hardware sizing manuals to help you build, run, and scale offline AI models with zero guesswork.