Guide · Layer 03 · Models
Quantization explained: GGUF, AWQ, GPTQ, and what 4-bit actually costs
Quantization is the reason a 70B model runs on a workstation and an 8B model runs on a phone. It is also the least understood decision in a local stack, usually made by downloading whichever file was smallest. This guide explains what the formats do, what they cost, and how to measure that cost on the work you actually do instead of trusting a leaderboard.
- Series
- 03 of 09
- Sections
- 06
- Updated
- Sep 14, 2026
- Reading time
- 9 min
- Written for
- developers
What quantization is
A model's weights are trained as 16-bit or 32-bit floating-point numbers. Quantization stores them in fewer bits, typically 8, 5, 4, or fewer, by mapping each group of weights to a small set of levels plus a scale factor that recovers the original range. The compute during inference dequantizes on the fly or uses kernels that work directly on the packed integers. The model gets smaller by a factor of the bit ratio, memory bandwidth per token drops by the same factor, and generation gets faster because bandwidth is the ceiling, as the hardware guide covers.
The loss is precision. Every weight is rounded to its nearest level, and the rounding error accumulates through the network. Good quantization schemes spend their bits where the error matters most: on the weights with the largest activations, on attention layers over feed-forward layers, on the first and last layers over the middle. The difference between formats is mostly how cleverly they do that.
The families
GGUF (llama.cpp)
The format of llama.cpp, Ollama, and LM Studio, and the one you will meet first. A GGUF file is a single self-describing archive: weights, tokenizer, chat template, and metadata. Its quantization types come in three generations. The legacy Q4_0/Q8_0 types use one scale per block of 32 weights. The K-quants (Q3_K_M, Q4_K_M, Q5_K_M, Q6_K) use super-blocks with two levels of scales and mix precisions across tensors so the important ones keep more bits. The IQ types (IQ2_XXS, IQ3_XS, IQ4_XS) use non-uniform codebooks and an importance matrix, computed from a calibration text, to reach lower bit rates at the same quality.
The letters after K are size variants: S small, M medium, L large. Q4_K_M is the default recommendation for a reason: it is the point on the curve where quality is nearly indistinguishable from 8-bit on most tasks and size is a little under a third of FP16.
AWQ and GPTQ
GPU-native 4-bit formats served by vLLM, SGLang, and the transformers library. GPTQ quantizes layer by layer using second-order information about how each weight affects the output, correcting the rounding error of each weight against the ones already quantized. AWQ (activation-aware weight quantization) observes which weight channels see the largest activations during calibration and scales them so they keep more precision. Both use a group size (128 is typical) that sets how many weights share a scale. Both have fast fused kernels on NVIDIA hardware, which is why they are the throughput choice for a batching server.
EXL2
The ExLlama format assigns a different bit rate to each layer and tensor to hit an overall target, so a model can be quantized to 4.65 or 5.2 bits per weight rather than a fixed integer. It is NVIDIA-only and single-user oriented, and it is what you reach for when a model almost fits a card and a fractional bit rate is the difference.
FP8
Not a quantization scheme in the integer sense but an 8-bit floating-point format with hardware support on recent NVIDIA and AMD datacenter and workstation parts. It halves memory against FP16 with negligible measured loss and keeps the model in a float representation that behaves like the original. Where the hardware supports it, it is the least controversial way to fit a model that is one size too large.
NF4 and INT8 (bitsandbytes)
Training-time formats rather than serving formats. NF4 is the 4-bit normal-float used by QLoRA to hold a frozen base model while adapters train on top; LLM.int8() is an 8-bit matrix multiply with outlier handling. They are what make fine-tuning on a consumer GPU possible, and they are slow for inference compared to the serving formats above.
Bits per weight, and what they buy
| Type | Approx. bits / weight | Size vs FP16 | Where it sits |
|---|---|---|---|
| FP16 / BF16 | 16 | 1.00× | Reference; what everything is measured against |
| Q8_0 / FP8 / INT8 | 8–8.5 | ~0.53× | Effectively lossless on most tasks |
| Q6_K | ~6.6 | ~0.41× | Very close to 8-bit; the safe compromise |
| Q5_K_M | ~5.7 | ~0.36× | Small measurable loss; still comfortable |
| Q4_K_M / AWQ / GPTQ 4-bit | ~4.5–4.9 | ~0.30× | The default; most models hold up well here |
| IQ4_XS | ~4.25 | ~0.27× | K-quant quality at a smaller size, needs an importance matrix |
| Q3_K_M / IQ3_XS | ~3.5–3.9 | ~0.23× | Noticeable loss; acceptable on large models, risky on small ones |
| IQ2 family | ~2.1–2.6 | ~0.15× | Last resort for fitting a huge model; expect degradation |
What degrades first
Quality loss is not uniform, and the pattern is consistent enough to plan around:
- Small models suffer more than large ones. A 70B at 3 bits often outperforms an 8B at 8 bits; an 8B at 3 bits is usually not worth running. The rule of thumb is that the larger the model, the more redundancy it has to absorb rounding error.
- Precise tasks break before fluent ones. Arithmetic, code that must compile, exact-format extraction, and multi-step reasoning show loss at bit rates where chat and summarization still look fine. Instruction-following on strict output formats is an early casualty.
- Long-context recall degrades. Retrieval of a fact from deep in a long prompt is sensitive to attention precision. If the workload is long documents, test at the context length you will use, not at a short one.
- Low-resource languages and rare tokens go first. The weights that handle them are the ones a scheme is most willing to sacrifice.
- The KV cache is separate. Quantizing the cache to 8-bit is a different, milder decision than quantizing the weights, and it should be made first when memory is the constraint. Cache at 4-bit is where measurable loss appears on recall-heavy tasks.
Measuring the loss on your own task
The only evaluation that matters is the one that looks like your workload. It does not need to be large; it needs to be representative and held out.
- Collect 100–300 real inputs from the task: tickets, documents, prompts, code changes. Strip anything sensitive, or run the evaluation entirely on the local box so nothing leaves it.
- Write the expected output for each, or a rubric that a script can score: exact match for extraction, a compile-and-test pass for code, a checklist for summaries.
- Run the same inputs through the model at 8-bit, at the 4-bit you plan to ship, and at any lower rate you are tempted by. Same prompt, same sampling settings, same context length.
- Score, and look at the failures, not just the aggregate. A two-point drop that is concentrated in one category is a different decision from a two-point drop spread evenly.
- Keep the set. It becomes the regression test for every model swap from here on.
# The shape of the loop, not a library. Any OpenAI-compatible local
# server works — the base URL is the only thing that changes per engine.
import json, httpx
cases = [json.loads(l) for l in open("eval/cases.jsonl")]
servers = {"q8": "http://localhost:8080/v1", "q4": "http://localhost:8081/v1"}
def ask(base, prompt):
r = httpx.post(f"{base}/chat/completions", timeout=120, json={
"model": "local", "temperature": 0,
"messages": [{"role": "user", "content": prompt}],
})
return r.json()["choices"][0]["message"]["content"].strip()
for name, base in servers.items():
hits = sum(ask(base, c["input"]) == c["expected"] for c in cases)
print(f"{name}: {hits}/{len(cases)} exact match")Rules of thumb
- Start at
Q4_K_M(or AWQ/GPTQ 4-bit on a batching server). Move up toQ6_Kor 8-bit if the eval shows loss on your task; move down only if the model does not fit. - Prefer a bigger model at 4-bit to a smaller model at 8-bit for the same memory. This holds far more often than it fails.
- Use an importance-matrix quantization for anything below 4 bits. Without it, the low-bit types are noticeably worse.
- Quantize the KV cache to 8-bit before dropping the weights a tier.
- Never mix results from two quantizations of a model in one evaluation. They are different models for the purpose of the numbers.
Quantization sets what fits; the fine-tuning guide covers the other lever on model quality, and the hardware guide turns the bits into gigabytes.