Technical Evaluation Guide
How to Read Hugging Face Model Cards & Configs
Cut through marketing hype. Learn how to open config.json, decode attention heads, identify the MoE total-parameter trap, and decipher GGUF quantization strings.
Interactive Hugging Face config.json Inspector
Click any line in the code block to see what it means, how it impacts your GPU, and the traps to avoid.
{
"model_type": "\"llama\"",weights "max_position_embeddings": 131072,context "num_hidden_layers": 32,attention "num_attention_heads": 32,attention "num_key_value_heads": 8,attention "head_dim": 128,attention "sliding_window": 4096,context "num_local_experts": 8,moe "num_experts_per_tok": 2,moe "torch_dtype": "\"bfloat16\"",weights}"num_key_value_heads": 8
Key-Value Heads (GQA Indicator)
The number of KV heads stored in the KV cache. When this value is smaller than num_attention_heads (here: 8 vs 32), the model uses Grouped-Query Attention (GQA).
In this example (8 KV heads vs 32 query heads), KV cache VRAM is slashed by 75%.
If num_key_value_heads equals num_attention_heads, the model uses legacy MHA and will consume 4x to 8x more VRAM for context!
1. The Three Layers of a Hugging Face Model Repository
Contains human-written summaries, benchmark scores, intended use cases, and prompt formatting templates.
The machine-readable JSON specification containing layer counts, KV head ratios, RoPE scaling factors, and maximum context limits.
Where the actual tensors reside: .safetensors shards (e.g. 5GB each) or standalone .gguf files.
2. The MoE Memory Trap: Active vs. Total Parameters
Mixture of Experts (MoE) architectures (e.g. Mixtral 8x7B, DeepSeek-V3, and Qwen 2.5 57B-A14B) are frequently misunderstood by buyers and self-hosters.
“Mixtral 8x7B only uses 2 experts per token (~13B active parameters). Therefore, it should fit easily on my 16GB RTX 4060 Ti!”
The Reality: You need ~47GB of VRAM to run it at speed.While only 2 experts calculate the output for any given token, the model's router can select any of the 8 experts on the very next token in fractions of a millisecond. If all 8 experts are not resident in VRAM or RAM, the system will freeze waiting for disk transfers.
Determines inference speed (tokens/sec) and compute FLOPs. A 671B MoE with 37B active parameters generates tokens at the speed of a 37B dense model.
Determines VRAM & RAM storage requirements. You must have enough physical memory to hold all weights simultaneously.
3. Decoding GGUF Filenames: What Does Q4_K_M Actually Mean?
When browsing community quantization hubs (like TheBloke or Bartowski), you will encounter long filenames like:Llama-3.1-8B-Instruct-Q4_K_M.gguf
The primary numerical precision. Q4 is ~4 bits per weight; Q8 is 8 bits (near lossless).
‘K’ denotes super-block k-quants. ‘IQ’ denotes importance-matrix quants (optimized for sub-4-bit).
S (Small: uniform quant), M (Medium: critical attention at higher bits), L (Large: max precision mixture).
Always download Q4_K_M as your default. If you have surplus VRAM headroom, step up to Q5_K_M. Avoid Q2_K or Q3_K unless nothing else fits, as code and complex reasoning degrade rapidly below 3.5 bits.
4. Safetensors Shards: How to Calculate Base FP16 Memory
When a model is stored in unquantized Safetensors format, Hugging Face splits large models into shards:
You can also check model.safetensors.index.json, which maps every individual tensor name (e.g. model.layers.0.self_attn.q_proj.weight) to its respective shard file.
Frequently Asked Questions
Do not rely exclusively on the text in the README. Click the 'Files and versions' tab, open `config.json`, and look for `max_position_embeddings`. Also check if a `sliding_window` parameter exists; if it does, some layers may evict tokens beyond that window.
This is the #1 confusion in local AI. In a Mixture of Experts model, only 2 experts execute compute for any single token (~13B active parameters), so generation speed is fast. However, because the router can send the very next token to any of the 8 experts, ALL 8 experts (47B total parameters) must be held in VRAM at all times.
'Q4' means 4-bit baseline quantization. 'K' refers to k-quants (an advanced scheme in llama.cpp grouping weights into super-blocks). 'M' stands for Medium (critical attention and output tensors are preserved at higher 5-bit or 6-bit precision, while feed-forward weights use 4-bit). 'S' is Small, and 'L' is Large.
`.safetensors` files contain unquantized (or AWQ/GPTQ) raw tensors intended for PyTorch, Transformers, and server engines like vLLM. They require loading the entire model onto a dedicated GPU. `.gguf` files are single-file, highly quantized packages designed for llama.cpp and Ollama that support partial CPU/RAM offloading and Mac Metal acceleration.
Simply paste the Hugging Face repository URL (e.g. `meta-llama/Llama-3.1-8B-Instruct`) into our VRAM Calculator search bar. Our backend automatically parses its `config.json`, reads the layer counts, KV heads, and context boundaries, and estimates the exact memory required for weights, KV cache, and runtime overhead.
Skip manual math: Paste any Hugging Face URL
Our VRAM Calculator automatically inspects the repo's config.json and calculates exact weights and KV cache.
Knowledge & Deep Dives
VRAM & Local AI Knowledge Hub
Everything you need to know about sizing hardware, understanding model architecture, and running local LLMs without out-of-memory crashes.