llama.cpp

llama.cpp is a C/C++ inference engine for large language models, created by Georgi Gerganov and first released in March 2023. What started as a single-evening hack to run Meta's Llama model on a MacBook without a GPU has become the foundational infrastructure for virtually every local LLM tool in existence: Ollama, LM Studio, Jan, and AnythingLLM all use llama.cpp as their core inference backend. The project is MIT-licensed, has over 108,000 GitHub stars, and has shipped more than 5,000 versioned builds. It runs on Apple Silicon (Metal), NVIDIA GPUs (CUDA), AMD GPUs (HIP), Vulkan for cross-vendor GPU support, and pure CPU with AVX/AVX2/AVX512 instruction set optimizations.

The engine supports 50+ model architectures, including LLaMA variants, Mistral, Mixtral, Gemma, Phi, DeepSeek, and Qwen. It uses the GGUF file format, which Gerganov introduced in August 2023 and which has since become the universal standard for local LLM distribution. Quantization options range from 1.5-bit to 8-bit, letting a 7-billion-parameter model fit in under 4 GB of RAM. The project ships a full OpenAI-compatible HTTP server (llama-server), a command-line chat interface (llama-cli), benchmarking tools, and multimodal inference support via the libmtmd library added in April 2025. The install footprint is under 90 MB. There is no cloud dependency, no account required, and no data leaves the machine running it.

What llama.cpp actually does in May 2026

The current build (b9012, May 3, 2026) ships a complete local inference stack. The primary entry point for most developers is llama-server, which launches an OpenAI-compatible API at localhost:8080 with a single command. Any client built against the OpenAI SDK works without modification, which means llama.cpp plugs directly into VS Code plugins, Python scripts, or any application using /v1/chat/completions.

Quantization is the core technical capability that makes llama.cpp viable on consumer hardware. The GGUF format packages the quantized model, vocabulary, and architecture config in a single file. The default recommendation for most use cases is Q4_K_M (about 4.5 GB for a 7B model), which delivers negligible quality loss versus the full-precision version while running on a MacBook with 8 GB of RAM at 40+ tokens per second. For constrained devices, the April 2026 Q1_0 format fits capable models under 1 GB. For users prioritizing quality, Q8_0 or Q5_K_M are available.

Hardware offloading is handled automatically. When a model exceeds available GPU VRAM, llama.cpp splits it: the layers that fit go to the GPU, and the remainder processes on CPU RAM. Performance degrades gracefully rather than failing. On Apple Silicon, Metal acceleration is native. On Linux with NVIDIA hardware, CUDA is the default. A single binary can be compiled with Vulkan support to run on AMD, Intel Arc, or any Vulkan-capable GPU without vendor-specific drivers.

The April 2026 builds added backend-agnostic tensor parallelism via NCCL and RCCL, which splits individual operations across multiple GPUs simultaneously rather than just dividing model layers. For multi-GPU configurations, this produces 3-4x throughput gains compared to the older layer-partitioning approach and makes llama.cpp increasingly viable for small production deployments, not just personal use. The same month, Walsh-Hadamard KV cache rotation shipped, lifting AIME25 reasoning benchmark scores from 0.0% to 21.7% under Q4_0 quantization, a significant improvement for quantized inference on multi-step reasoning tasks.

Model downloading integrates directly with Hugging Face and Docker Hub. Passing a Hugging Face model ID to llama-server pulls the GGUF file and starts the server. Since Hugging Face natively indexes GGUF files, this works for thousands of community-quantized models without any conversion step.

"I managed the 13B dataset on a single Pi4 8gig..the current reality of reproducible LLM at home on anything you've got" - cameron_b, Hacker News, September 2023

Where llama.cpp sits versus vLLM and MLX

llama.cpp vs. vLLM: These two tools serve different ends of the deployment spectrum and rarely compete for the same use case. vLLM is a Python-based, GPU-server-focused inference framework built around continuous batching and PagedAttention, a mechanism for managing KV cache memory at the page level. On an NVIDIA H200 at peak load, vLLM delivers 35x the request throughput and 44x the token output per second compared to llama.cpp, but that comparison only holds at 10+ concurrent users on enterprise-grade hardware. For single-user or low-concurrency workloads, llama.cpp's inter-token latency is dramatically lower because it does not batch requests. vLLM requires CUDA and does not run meaningfully on consumer hardware, Windows desktops, or Apple Silicon. llama.cpp runs everywhere. Compare vLLM if you are deploying a production API on a dedicated GPU server with 5+ concurrent users; llama.cpp is the right choice for everything else.

llama.cpp vs. MLX: MLX is Apple's own machine learning framework, optimized for the unified memory architecture of Apple Silicon chips. Because the CPU and GPU share the same physical memory on Apple Silicon, MLX achieves zero-copy tensor operations, eliminating the data transfer overhead that llama.cpp incurs when moving data between Metal shaders. On an M2 Pro running Llama 3.1 8B at Q4_K_M, MLX achieves approximately 45-58 tokens per second versus llama.cpp's 38-48. For Mac-only development workflows, that gap matters. MLX also supports native LoRA and QLoRA fine-tuning, which llama.cpp does not. The critical limitation: MLX is Apple Silicon only. It does not run on Windows, Linux, or Android. It also requires model conversion from GGUF format, and community models often appear in GGUF first with MLX conversions lagging by hours or days. llama.cpp has day-one support for new releases from the community and runs on any hardware.

"Just switched from ollama and the speed token generation and efficiency gain has been outstanding." - user review, itsfoss.com, 2025

What the daily inference workflow looks like

The minimal path to a running model is three steps: download a GGUF file from Hugging Face, run llama-server -m model.gguf, and hit localhost:8080/v1/chat/completions. The server starts in under five seconds on fast storage. Stopping and restarting with a different model takes the same amount of time. No daemon management, no model registry, no background service eating resources when idle.

For interactive use, llama-cli -m model.gguf --interactive opens a terminal chat session. Grammar constraints let you force JSON output or restrict responses to a defined schema, which is useful for structured data extraction pipelines. Context window length is configurable via command-line flag, limited by available RAM.

The web UI, introduced in 2024, provides a browser-based chat interface hosted by llama-server. It supports multi-turn conversation, system prompt configuration, and parameter adjustment without touching the command line. It is not as polished as LM Studio's interface, but it works in any browser and requires no installation beyond the server binary.

For users building applications on top of llama.cpp, VS Code and Vim/Neovim plugins ship as part of the project, routing code completion requests through the local llama-server endpoint. The OpenAI compatibility layer means any tool that accepts a custom base URL, such as AnythingLLM or Continue.dev, connects without configuration changes.

Who llama.cpp is built for

llama.cpp targets developers and technical users who want full control over their inference stack and are willing to spend 30-60 minutes on initial setup to avoid cloud dependency, per-token costs, or the abstraction layers that higher-level tools add. Privacy-conscious users who cannot or will not send data to external APIs rely on it. Researchers running experiments on consumer hardware use it to access the same model weights served by cloud providers at zero marginal cost per query. Embedded and edge developers compile it for Raspberry Pi, Android (native acceleration via Qualcomm Hexagon since April 2026), and ChromeOS devices.

The ecosystem significance is harder to overstate: even users who never touch llama.cpp directly are likely running it if they use any local LLM frontend tool. Understanding what llama.cpp is helps users understand where their inference actually happens, why certain GGUF quantization levels work better on their hardware, and how to debug performance problems that their GUI wrapper does not expose.

Users who pair llama.cpp directly with Meta's Llama models get the fullest compatibility, as the project was originally built for Llama and the two remain closely aligned on new architecture support. The community model ecosystem on Hugging Face, where thousands of GGUF quantizations exist for every major open-weight release, effectively makes llama.cpp the on-ramp for any new model within hours of its release.

What llama.cpp is not

llama.cpp is not a graphical application. Users expecting point-and-click model management will find the CLI unfamiliar. LM Studio and Jan exist precisely to add that GUI layer on top of llama.cpp's inference engine.

It is not a multi-user inference server at scale. The engine processes requests sequentially. At five or more concurrent users, queue latency grows exponentially. For production API serving with real concurrency requirements, vLLM or a hosted endpoint is more appropriate.

It is not a fine-tuning tool. There is no training loop, no LoRA implementation, and no dataset pipeline. If you need to fine-tune a model, that work happens in a Python framework (PyTorch, MLX, Axolotl) and the resulting weights are then converted to GGUF for inference.

It is not a model hub or discovery tool. llama.cpp does not curate or recommend models. Users find models on Hugging Face or through community resources like r/LocalLLaMA, then download GGUF files directly.

It is also worth being clear about attribution in the ecosystem. Many users of Ollama or LM Studio do not realize those tools depend on llama.cpp for their inference. A persistent community frustration, surfaced in GitHub issue #3185 and discussed repeatedly on Hacker News, is that Ollama does not clearly credit llama.cpp in its documentation or user-facing interface. This matters because performance problems, quantization behavior, and supported model architectures are all properties of llama.cpp, not of the wrapper tool. Knowing that llama.cpp is the actual engine gives users a meaningful vocabulary for debugging and optimization.

The practical implication: if you are choosing between Ollama and llama.cpp directly, you are not choosing between different inference engines. You are choosing between different levels of abstraction over the same inference engine. For most users, Ollama's simplicity wins. For developers who need to tune parameters, build custom pipelines, or run models that Ollama's registry does not carry, direct llama.cpp access is the right layer to work at.