🧮 LLM Quantization Explained: Q4 vs Q8 in Practice

Quantization is the single trick that makes local AI possible — it's why a 14B model fits in 10GB instead of 28GB — and it's also the setting most people cargo-cult without understanding. We run quantized models daily on a 16GB rig, so this is the practical version: what the numbers mean, what quality you actually lose, and the two rules that answer 90% of quant decisions.

By the numbers

• FP16 weights cost ~2 bytes per parameter — a 14B model is ~28GB before quantization, ~10GB at Q4 (LLM Configurator)
• The GGUF ecosystem these quants live in has grown to ~135,000 models on Hugging Face
• Rule-of-thumb VRAM at Q4: 7-8B ≈ 5GB · 13-14B ≈ 10GB · 30B-class ≈ 20GB · 70B ≈ 40GB

What quantization actually does

A model's "intelligence" lives in billions of numeric weights. Quantization stores each with fewer bits — 16 → 8 → 4 → lower — like re-encoding a photo at higher compression. The surprise that makes local AI work: neural networks are redundant enough that 4-bit storage keeps almost all behavior intact. The loss isn't uniform, though — it concentrates in the model's most precision-hungry behaviors: long reasoning chains, exact code syntax, and rare factual recall.

Decoding the alphabet soup

Tag	Meaning	When you'd pick it
Q8_0	~8-bit, near-lossless	You have VRAM to burn
Q6_K	~6-bit K-quant	Quality-first when it fits
Q5_K_M	~5-bit, protected sensitive layers	Sweet spot on roomy cards
Q4_K_M	~4-bit, protected sensitive layers	The default. Start here.
Q3_K_M	~3-bit	Last resort before quality visibly sags
IQ2/IQ1	Ultra-low importance-quants	Fitting whales on small cards; expect trade-offs

The K family varies precision by layer sensitivity, and _M (medium) variants guard the layers that hurt most when crushed — which is why Q4_K_M punches above its bit count and became the community default. When a tool like Ollama pulls a model, this class of quant is usually what you're getting.

The two rules that settle most decisions

Rule 1 — Parameters beat precision. A 14B at Q4 outperforms an 8B at Q8 on nearly everything we throw at them. If extra VRAM lets you choose between a fatter quant or a bigger model, take the bigger model.

Rule 2 — Don't pay for precision you can't feel. Step up from Q4 only when you observe a Q4-typical failure — broken code syntax, derailing logic chains — and the next quant up still fits. Stepping up "for quality" you never measured is how people end up with slower models and no visible gain.

Where the quality loss actually shows

From our daily use: casual chat and summarization are indistinguishable Q4-vs-Q8. Code generation shows the gap first (a missed bracket, a hallucinated argument), followed by multi-step math and niche factual recall. If your workload is code-heavy, favor Q5/Q6 when the VRAM math allows; if it's drafting and chat, Q4 is genuinely free lunch.

Context: the quant nobody mentions

Weights aren't the only VRAM consumer — long contexts grow the KV cache fast, and some runtimes let you quantize that too. Practical effect: a "fits fine" model can OOM the moment you paste a long document. When it happens, shrink context before shrinking the model; a smarter model with adequate context beats a dumber one with a huge window you rarely fill.

Try before you buy (hardware, that is)

If you're weighing a GPU upgrade around quant trade-offs, the honest experiment costs a few dollars: rent a 24GB card for an evening, run your real workload at Q4/Q5/Q6, and see which differences you can actually blind-taste. That test has talked us out of more upgrades than into them — the same logic behind our run Llama locally tiers.

How we know

Everything above comes from running quantized models in daily production on our own 16GB RTX 4080 — same prompts across quant levels, failures logged, plus the published VRAM guidance cited above. When new quant families change these rules, this page's updated date will move.