The Local LLM & VRAM Glossary

Total parameters represent all weights stored on disk and loaded into memory; active parameters represent the subset actually executed during compute for a single token.

A 4-bit GPU quantization technique that preserves the top 1% of salient weights based on activation magnitudes, ensuring near-FP16 accuracy with high inference speed.

The effective average number of storage bits allocated to each parameter in a model after quantization and scaling factors are accounted for.

A custom open-weights license designed by large tech corporations that grants free access for research and small-to-medium businesses, but restricts mega-scale commercial use.

The maximum total number of tokens (prompt input plus generated output) that a model can comprehend and attend to in a single conversation session.

A fatal runtime exception thrown by NVIDIA CUDA when an allocation request for weights, KV cache, or activations exceeds available physical VRAM.

An advanced mixed-precision quantization format designed specifically for NVIDIA GPUs by turboderp, offering continuous bitrates and industry-leading generation speed.

An exact attention algorithm that reorganizes memory accesses to keep operations inside fast GPU SRAM (cache) rather than repeatedly reading from slower HBM/VRAM.

A model repository on platforms like Hugging Face that requires users to accept legal terms or request explicit author permission before downloading weights.

A standardized, extensible single-file binary format developed by Georgi Gerganov and the llama.cpp community designed for fast local CPU/GPU inference and layer offloading.

An attention architecture where multiple query heads share a single key-value head, reducing KV cache VRAM by 4x to 8x while preserving reasoning capability.

Advanced quantization schemes in llama.cpp that group weights into super-blocks and allocate higher precision to critical attention and feed-forward layers.

A dedicated block of GPU memory that stores previously computed attention key and value tensors for every token in a conversation, preventing redundant recomputations.

Splitting a model across hardware by loading as many transformer layers as fit into GPU VRAM while leaving the remaining layers in system CPU RAM.

The foundational, highly optimized C/C++ inference engine created by Georgi Gerganov that powers the vast majority of local LLM tools across CPU, CUDA, Metal, and ROCm.

The rate at which a processor can read data from and write data to its memory, measured in gigabytes per second; this directly dictates LLM generation speed.

The classic attention architecture where every single query head possesses its own dedicated key and value head, creating the largest and most VRAM-hungry KV cache.

A neural network architecture where multiple specialized sub-networks ('experts') exist, but only a small subset is activated for each individual token.

DeepSeek's pioneering attention mechanism that compresses key-value vectors into a low-rank latent vector before caching, slashing KV cache memory by over 90%.

Apple's native machine learning framework designed specifically for Apple Silicon M-series unified architecture, delivering maximum speed and efficiency on macOS.

A user-friendly, CLI-driven application and daemon that packages llama.cpp into a Docker-like model management system with a clean REST API.

An AI system meeting the Open Source Initiative (OSI) OSAID 1.0 standard, requiring open model weights, full training code, and comprehensive training data information.

A release model where a creator publishes the neural network weights for download and self-hosting, but keeps the underlying dataset, preprocessing scripts, or training pipeline private.

An attention algorithm that stores KV cache tensors in non-contiguous physical memory blocks (pages), eliminating memory fragmentation and enabling near-zero memory waste.

Software and weight licenses (like Apache 2.0 or MIT) that grant full commercial rights, modifications, and redistribution with minimal restrictions.

A mathematical metric that measures how well a language model predicts a sample test dataset; lower perplexity indicates higher accuracy and better reasoning.

The process of reducing the numerical precision of a model's weights and activations (e.g., from 16-bit float down to 8-bit, 4-bit, or 2-bit) to drastically reduce memory usage and speed up computation.

A fast, secure, memory-mappable file format created by Hugging Face for storing deep learning tensors without the arbitrary code execution risks of Python pickles.

An attention strategy where tokens only attend to a fixed window of preceding tokens (e.g., the last 4,096 tokens), placing a hard ceiling on KV cache memory growth.

Clauses in AI terms of service that prohibit using the model's outputs to train, fine-tune, or evaluate other competing artificial intelligence models.

Methods for distributing a large model across multiple GPUs; Tensor Parallelism splits individual weight matrices, while Pipeline Parallelism places different layers on different cards.

A shared memory architecture used in Apple M-series chips where CPU, GPU, and Neural Engine access the same high-bandwidth pool of physical memory without bus copies.

A high-throughput, enterprise-grade LLM serving engine developed at UC Berkeley, renowned for PagedAttention, continuous batching, and chunked prefill.

Dedicated ultra-high-bandwidth memory physically located on a discrete graphics card (GPU) used for storing model weights, KV cache, and intermediate activations.