Hardware Sizing & Buying Guide
The Ultimate GPU Guide for Local LLMs
Forget gaming benchmarks. Learn why memory bandwidth determines your generation speed, which VRAM tier you need for each model class, and how Apple Silicon compares to Nvidia.
1. The Golden Rule of AI Hardware: Bandwidth Over Compute
In autoregressive LLM inference, the computational bottleneck is rarely raw compute power (TFLOPS). Instead, it is almost always memory bandwidth: the speed at which model weights can be read from VRAM into the compute cores.
During generation, every single weight in the model must be loaded from memory into the GPU cores for every single generated token. A GPU with massive compute but narrow memory bandwidth will sit idle waiting for weights.
Example: A 70B model quantized to Q4 requires approximately 40 GB of VRAM. On an RTX 3090 with 936 GB/s bandwidth, the maximum theoretical speed is 936 / 40 = 23.4 tokens/second. On DDR5 system RAM with 60 GB/s bandwidth, that same model generates at just 60 / 40 = 1.5 tokens/second.
2. The 4 VRAM Capacity Tiers: What Can You Actually Run?
7B to 8B models at Q4_K_M (Llama 3 8B, Mistral 7B, Qwen 2.5 7B). Short context up to 8k tokens.
RTX 3060 12GB (budget champ), RTX 4060 Ti 16GB, or RTX 5060 Ti 16GB.
14B to 32B models at Q4_K_M (Qwen 2.5 32B, Command-R), or 70B models with aggressive quantization (IQ2/Q3).
RTX 3090 24GB (used, unbeatable value), RTX 4090 24GB, or RTX 5090 32GB.
Full 70B to 72B models at Q4_K_M with generous 32k context windows and FP8 KV cache.
Dual RTX 3090s (48GB total), dual RTX 4090s, or Apple Mac Studio M2/M3/M4 Max (48GB to 64GB).
Full Llama 3.3 70B at Q8_0, Qwen 2.5 72B at FP16, DeepSeek R1/V3 distilled models, and Llama 405B at Q2/Q3.
Apple Mac Studio M2/M3 Ultra (128GB to 192GB unified RAM) or workstation quad-GPU rigs.
3. Apple Silicon Mac vs. Nvidia PC: Which Should You Buy?
- Massive VRAM Capacity: Up to 128GB or 192GB unified memory on a single quiet desktop.
- Near Silent & Low Power: Draws 80W to 120W under full load compared to 850W+ for multi-GPU PCs.
- Moderate Bandwidth: 300 to 800 GB/s yields practical generation speeds of 15 to 25 tok/s on 70B.
- Software Ecosystem: Native MLX and llama.cpp Metal backends run great; CUDA/vLLM tools are unavailable.
- Maximum Generation Speed: 1,008 GB/s bandwidth on RTX 4090 produces blazing fast 30 to 60 tok/s.
- Industry Standard: 100% compatibility with CUDA, PyTorch, vLLM, TensorRT-LLM, FlashAttention-2.
- VRAM Wall: Hard 24GB ceiling per consumer card. Running 70B requires dual-GPU builds.
- Power & Heat: Requires 1,000W+ power supplies, large airflow cases, and dedicated circuit capacity.
4. Building a Dual-GPU Workstation for 70B Models
If your goal is to run dense 70B models at full speed on a PC, a dual-card configuration is the most cost-effective path:
Ensure your motherboard supports x8 / x8 bifurcation on CPU lanes, with at least 3 to 4 slot spacing between cards for cooling.
Budget at least a 1,200W or 1,500W Platinum/Titanium PSU to handle the transient power spikes of dual RTX 3090s or 4090s.
Use vLLM with Tensor Parallelism (--tensor-parallel-size 2) or llama.cpp multi-GPU to shard weights across both cards.
Frequently Asked Questions
Why is an older 24GB RTX 3090 better for local LLMs than a brand new 16GB RTX 4080?
Is 8GB VRAM enough to run local LLMs in 2026?
Can I run a 70B model on an Apple Silicon Mac Studio with 64GB or 128GB Unified Memory?
Why does my generation speed collapse when I use CPU offload (n-gpu-layers)?
Is a dual RTX 3090 setup better than a single RTX 4090?
Need more VRAM than your local GPU?
Rent on-demand RTX 4090 or H100 instances from RunPod and Vast.ai starting at $0.34/hr.
Knowledge & Deep Dives
Local LLM & VRAM Knowledge Hub
In-depth guides, architectural breakdowns, and hardware sizing manuals to help you build, run, and scale offline AI models with zero guesswork.