⚙️ llama.cpp — Tool Hub: Facts, Setup & Our Verdict
Everything about llama.cpp in one place: what the engine does, GGUF basics, the server flags that matter, and when to drop down from Ollama.
Derek Holt · Local AI & Hardware Writer
· 2 min read
⚡ TL;DR — quick answers
- What is llama.cpp?
- A C/C++ inference engine for running LLMs on your own hardware — CPU, CUDA, Metal or Vulkan — reading the GGUF model format. It's the engine underneath most local-AI tools, including Ollama.
- Do I need llama.cpp if I already use Ollama?
- Not usually. Ollama wraps llama.cpp's engine with a friendlier interface. Drop to raw llama.cpp when you need exotic quant formats, speculative decoding, or the last 10-20% of throughput that Ollama's wrapper overhead costs you — see the full comparison in Ollama vs llama.cpp.
- What's GGUF?
- The single-file model format llama.cpp reads — packing weights, tokenizer and metadata together, with support for quantization down to 2-bit. It replaced the older GGML format and is now the de facto standard for local LLM distribution.
At a glance
| What it is | C/C++ inference engine for local LLMs (CPU, CUDA, Metal, Vulkan) |
| License / price | MIT, free, 100K+ GitHub stars (crossed that mark faster than PyTorch or TensorFlow) |
| Platforms | Windows, macOS, Linux — plus edge/ARM builds |
| Signature strengths | Direct control over quantization, batching, and hardware-specific build flags |
| Weak spots | No GUI, manual flag tuning, you own the troubleshooting |
| Best for | Power users squeezing max throughput or running new GGUF quant formats day one |
Why it's the engine under everything
llama.cpp is what Ollama, LM Studio and most self-hosted local-AI stacks actually run on underneath. It reads the GGUF format — a single portable file packing weights, tokenizer and metadata, with quantization down to 2-bit for squeezing large models onto small cards. New quant formats and inference tricks land here first, before wrapper tools catch up.
The 2026 server rewrite added real production features: OpenAI- and Anthropic-compatible chat routes, function calling, speculative decoding, experimental multimodal input, embeddings/reranking endpoints, and router mode for serving several models behind one process. CUDA kernel rewrites also meaningfully sped up inference on Ada Lovelace and Blackwell-class cards.
Getting it running
git clone --depth 1 https://github.com/ggml-org/llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
./build/bin/llama-server --model model.gguf --port 8080 --n-gpu-layers 99
Drop -DGGML_CUDA=ON for a CPU-only build, or use -DGGML_METAL=ON on macOS. You can also load models directly from Hugging Face with -hf <user>/<model>:<quant> instead of downloading a GGUF file manually first.
Key flags that matter
--n-gpu-layers 99— offload every layer to GPU (lower it if you don't have enough VRAM)--ctx-size 32768— set your context window--flash-attn— memory-efficient attention, nearly always worth enabling--cache-type-k q8_0— quantize the KV cache to fit longer contexts in less VRAM--tensor-split— distribute layers across multiple GPUs by ratio
When to drop down from Ollama
Most people never need raw llama.cpp — Ollama's wrapper overhead costs roughly 10-20% throughput, invisible for single-user chat. Drop down when you're chasing that last margin at scale, need a quant format Ollama's registry hasn't picked up yet, or want direct control over batching for a specific workload. The full trade-off table is in Ollama vs llama.cpp.
Official resources
- llama.cpp on GitHub — source, releases, build docs
- llama-server README — full flag reference
Go deeper
- Ollama vs llama.cpp — the same-hardware showdown with real numbers
- How to run Llama locally — three on-ramps, VRAM table per model size
- Quantization explained — what Q4 vs Q8 actually costs you
Frequently asked questions
▸What is llama.cpp?
A C/C++ inference engine for running LLMs on your own hardware — CPU, CUDA, Metal or Vulkan — reading the GGUF model format. It's the engine underneath most local-AI tools, including Ollama.
▸Do I need llama.cpp if I already use Ollama?
Not usually. Ollama wraps llama.cpp's engine with a friendlier interface. Drop to raw llama.cpp when you need exotic quant formats, speculative decoding, or the last 10-20% of throughput that Ollama's wrapper overhead costs you — see the full comparison in Ollama vs llama.cpp.
▸What's GGUF?
The single-file model format llama.cpp reads — packing weights, tokenizer and metadata together, with support for quantization down to 2-bit. It replaced the older GGML format and is now the de facto standard for local LLM distribution.
▸What are the most important llama-server flags?
--n-gpu-layers 99 for full GPU offload, --ctx-size for context window, --flash-attn for memory-efficient attention, and --cache-type-k q8_0 for KV-cache quantization. The 2026 server also ships OpenAI- and Anthropic-compatible chat routes, function calling, and router mode for serving multiple models.
The 5 best AI video finds, every week
New models, tested prompts, and what actually worked in our production — one short email a week. No spam, unsubscribe anytime.
Written by Derek Holt
Local AI & Hardware Writer
Runs the site's local-inference rig and benchmarks every GPU, quant, and speed-stack claim on it personally before it goes in a guide. Will not shut up about VRAM bandwidth.
Explore these topics
Every guide, comparison and prompt library we have on each.
Keep learning
Local LLMs · Ollama & LM Studio
GuidesLocal LLMs on Apple Silicon: Unified Memory Wins
A 64GB MacBook loads models a 24GB RTX 4090 cannot touch. What unified memory buys you, where MLX still beats everything, and which stack to actually run.
