Hardware Sizing & Buying Guide
The Ultimate GPU Guide for Local LLMs
Forget raw gaming benchmarks. Learn why memory bandwidth dictates your tokens-per-second, which VRAM tier you need for each model class, and how Apple Silicon compares to Nvidia.
1. The Golden Rule of AI Hardware: Bandwidth Over Compute
In 3D gaming and video rendering, raw compute performance (TFLOPS) and CUDA core count are the primary metrics. In large language model inference, the bottleneck is almost 100% memory bandwidth.
During token generation, the GPU cores must read every single parameter of the model from memory once for every single token generated. The mathematical speed limit is defined by a simple formula:
Consider a 40 GB model (e.g. 70B at Q4_K_M):
PCIe bus + slow RAM bandwidth makes generation virtually unusable.
Very usable for reading and coding; completely silent and power-efficient.
Ultra-fast interactive chatting, but requires dual cards to hold 40GB.
2. The 4 VRAM Capacity Tiers: What Actually Runs?
What Runs: 3B models at FP16, or 7B/8B models at Q4_K_M with short context (up to 4k–8k tokens).
What Fails: 14B models, 32B models, long 32k context, and concurrent browser tasks.
What Runs: 8B models at full unquantized FP16, 14B models at Q5_K_M with 32k context, or small MoEs.
Hardware Highlights: The RTX 3060 12GB remains the best budget AI card on the market ($280). The RTX 4060 Ti 16GB is the cheapest 16GB entry point.
What Runs: 27B to 32B models at Q4/Q5, dense 14B at FP16, and massive 64k+ context windows with zero compromise.
Pro Tip: A used RTX 3090 24GB (~$700) delivers 90% of the AI value of a $1,700 RTX 4090 because both have identical 24GB capacity.
What Runs: Full Llama 3.3 70B and Qwen 2.5 72B at Q4_K_M or Q8_0 completely in memory at snappy interactive speeds.
Build Choices: Dual RTX 3090s in a PC workstation ($1,500 total build) OR an Apple Mac Studio M2/M3 Max/Ultra (64GB to 192GB).
3. Apple Silicon Mac vs. Nvidia PC: Which Should You Buy?
- Massive VRAM Capacity: Up to 128GB or 192GB unified RAM on a single desktop.
- Near Silent & Low Power: Draws 80–120W under full load compared to 850W+ for dual-GPU PCs.
- Slower Bandwidth: 300 to 800 GB/s means lower peak tokens/second than high-end Nvidia cards.
- Ecosystem: Native MLX and llama.cpp Metal backends run great; vLLM/CUDA tools are unavailable.
- Maximum Generation Speed: 1,008 GB/s bandwidth on RTX 4090 produces blazing fast 30–60 tok/s.
- Industry Standard: 100% compatibility with CUDA, PyTorch, vLLM, TensorRT-LLM, FlashAttention-2.
- VRAM Wall: Hard 24GB ceiling per consumer card. Running 70B requires complex multi-GPU builds.
- Power & Heat: Requires 1000W+ power supplies, high-airflow cases, and dedicated cooling.
4. Building a Dual-GPU Workstation for 70B Models
If your goal is to run dense 70B models at full speed on a PC, a dual-card configuration is the most cost-effective path:
Ensure your motherboard supports x8 / x8 bifurcation on CPU lanes, with at least 3 to 4 slot spacing between cards for cooling.
Budget at least a 1,200W or 1,500W Platinum/Titanium PSU to handle the transient power spikes of dual RTX 3090s or 4090s.
Use vLLM with Tensor Parallelism (`--tensor-parallel-size 2`) or llama.cpp multi-GPU to shard weights across both cards.
Frequently Asked Questions
Because in local LLMs, VRAM capacity is king. A 24GB RTX 3090 can fit 32B models at Q4 or dense 14B models at FP16 with a 32k context window completely on-device. The 16GB card cannot fit 32B models at full speed and is forced to offload layers to slow system RAM. Furthermore, the RTX 3090 features 936 GB/s memory bandwidth across a 384-bit bus, outperforming the RTX 4080's 717 GB/s.
8GB is enough to learn and run 7B/8B models at Q4_K_M with short context (up to 4k–8k tokens). However, it is tight for modern coding agents, large document analysis, or running a desktop display simultaneously. If buying new, 12GB to 16GB is the recommended minimum floor.
Yes! Apple Silicon's unified memory architecture allows the GPU cores to access the entire RAM pool without PCIe bottlenecks. A 64GB Mac Studio can run Llama 3.3 70B at Q4_K_M comfortably (~45 GB total) at 15–22 tokens/second. A 128GB or 192GB Mac Studio can run 70B at full FP16 or run large MoE models that would otherwise require $10,000+ in PC hardware.
When a model fits 100% in VRAM, data moves at 500 to 1,000 GB/s. When layers are offloaded to system RAM, data must travel over the PCIe slot (PCIe 4.0 x16 tops out at ~31.5 GB/s, and DDR5 RAM runs at 60–90 GB/s). Because generation is memory-bandwidth bound, offloading even a few layers often drops throughput from 45 tok/s to 3–6 tok/s.
For 70B models, yes! A single RTX 4090 has 24GB of VRAM and cannot fit a quantized 70B model without severe offload. Dual RTX 3090s give you 48GB of pooled VRAM (via Tensor Parallelism in vLLM or layer splitting in llama.cpp), allowing you to run 70B Q4_K_M completely in VRAM at 25+ tokens/second for roughly the same total price as one 4090.
Test your exact GPU against the model catalog
Select your GPU in our VRAM Calculator to see which models fit, or compare cloud rental prices.
Knowledge & Deep Dives
VRAM & Local AI Knowledge Hub
Everything you need to know about sizing hardware, understanding model architecture, and running local LLMs without out-of-memory crashes.