Guide · Layer 03 · Models

Quantization explained: GGUF, AWQ, GPTQ, and what 4-bit actually costs

Quantization is the reason a 70B model runs on a workstation and an 8B model runs on a phone. It is also the least understood decision in a local stack, usually made by downloading whichever file was smallest. This guide explains what the formats do, what they cost, and how to measure that cost on the work you actually do instead of trusting a leaderboard.

Bits per weight
The same weights at three precisions. Fewer bits, less memory, and a cost in quality worth measuring.
Series
03 of 09
Sections
06
Updated
Sep 14, 2026
Reading time
9 min
Written for
developers

What quantization is

A model's weights are trained as 16-bit or 32-bit floating-point numbers. Quantization stores them in fewer bits, typically 8, 5, 4, or fewer, by mapping each group of weights to a small set of levels plus a scale factor that recovers the original range. The compute during inference dequantizes on the fly or uses kernels that work directly on the packed integers. The model gets smaller by a factor of the bit ratio, memory bandwidth per token drops by the same factor, and generation gets faster because bandwidth is the ceiling, as the hardware guide covers.

The loss is precision. Every weight is rounded to its nearest level, and the rounding error accumulates through the network. Good quantization schemes spend their bits where the error matters most: on the weights with the largest activations, on attention layers over feed-forward layers, on the first and last layers over the middle. The difference between formats is mostly how cleverly they do that.

The families

GGUF (llama.cpp)

The format of llama.cpp, Ollama, and LM Studio, and the one you will meet first. A GGUF file is a single self-describing archive: weights, tokenizer, chat template, and metadata. Its quantization types come in three generations. The legacy Q4_0/Q8_0 types use one scale per block of 32 weights. The K-quants (Q3_K_M, Q4_K_M, Q5_K_M, Q6_K) use super-blocks with two levels of scales and mix precisions across tensors so the important ones keep more bits. The IQ types (IQ2_XXS, IQ3_XS, IQ4_XS) use non-uniform codebooks and an importance matrix, computed from a calibration text, to reach lower bit rates at the same quality.

The letters after K are size variants: S small, M medium, L large. Q4_K_M is the default recommendation for a reason: it is the point on the curve where quality is nearly indistinguishable from 8-bit on most tasks and size is a little under a third of FP16.

AWQ and GPTQ

GPU-native 4-bit formats served by vLLM, SGLang, and the transformers library. GPTQ quantizes layer by layer using second-order information about how each weight affects the output, correcting the rounding error of each weight against the ones already quantized. AWQ (activation-aware weight quantization) observes which weight channels see the largest activations during calibration and scales them so they keep more precision. Both use a group size (128 is typical) that sets how many weights share a scale. Both have fast fused kernels on NVIDIA hardware, which is why they are the throughput choice for a batching server.

EXL2

The ExLlama format assigns a different bit rate to each layer and tensor to hit an overall target, so a model can be quantized to 4.65 or 5.2 bits per weight rather than a fixed integer. It is NVIDIA-only and single-user oriented, and it is what you reach for when a model almost fits a card and a fractional bit rate is the difference.

FP8

Not a quantization scheme in the integer sense but an 8-bit floating-point format with hardware support on recent NVIDIA and AMD datacenter and workstation parts. It halves memory against FP16 with negligible measured loss and keeps the model in a float representation that behaves like the original. Where the hardware supports it, it is the least controversial way to fit a model that is one size too large.

NF4 and INT8 (bitsandbytes)

Training-time formats rather than serving formats. NF4 is the 4-bit normal-float used by QLoRA to hold a frozen base model while adapters train on top; LLM.int8() is an 8-bit matrix multiply with outlier handling. They are what make fine-tuning on a consumer GPU possible, and they are slow for inference compared to the serving formats above.

Bits per weight, and what they buy

TypeApprox. bits / weightSize vs FP16Where it sits
FP16 / BF16161.00×Reference; what everything is measured against
Q8_0 / FP8 / INT88–8.5~0.53×Effectively lossless on most tasks
Q6_K~6.6~0.41×Very close to 8-bit; the safe compromise
Q5_K_M~5.7~0.36×Small measurable loss; still comfortable
Q4_K_M / AWQ / GPTQ 4-bit~4.5–4.9~0.30×The default; most models hold up well here
IQ4_XS~4.25~0.27×K-quant quality at a smaller size, needs an importance matrix
Q3_K_M / IQ3_XS~3.5–3.9~0.23×Noticeable loss; acceptable on large models, risky on small ones
IQ2 family~2.1–2.6~0.15×Last resort for fitting a huge model; expect degradation
Table 01 Bits-per-weight are approximate effective values from the llama.cpp quantization documentation and the AWQ/GPTQ papers; a specific file's ratio depends on which tensors the scheme leaves at higher precision. Size ratios follow from the bits.

What degrades first

Quality loss is not uniform, and the pattern is consistent enough to plan around:

  • Small models suffer more than large ones. A 70B at 3 bits often outperforms an 8B at 8 bits; an 8B at 3 bits is usually not worth running. The rule of thumb is that the larger the model, the more redundancy it has to absorb rounding error.
  • Precise tasks break before fluent ones. Arithmetic, code that must compile, exact-format extraction, and multi-step reasoning show loss at bit rates where chat and summarization still look fine. Instruction-following on strict output formats is an early casualty.
  • Long-context recall degrades. Retrieval of a fact from deep in a long prompt is sensitive to attention precision. If the workload is long documents, test at the context length you will use, not at a short one.
  • Low-resource languages and rare tokens go first. The weights that handle them are the ones a scheme is most willing to sacrifice.
  • The KV cache is separate. Quantizing the cache to 8-bit is a different, milder decision than quantizing the weights, and it should be made first when memory is the constraint. Cache at 4-bit is where measurable loss appears on recall-heavy tasks.

Measuring the loss on your own task

The only evaluation that matters is the one that looks like your workload. It does not need to be large; it needs to be representative and held out.

  1. Collect 100–300 real inputs from the task: tickets, documents, prompts, code changes. Strip anything sensitive, or run the evaluation entirely on the local box so nothing leaves it.
  2. Write the expected output for each, or a rubric that a script can score: exact match for extraction, a compile-and-test pass for code, a checklist for summaries.
  3. Run the same inputs through the model at 8-bit, at the 4-bit you plan to ship, and at any lower rate you are tempted by. Same prompt, same sampling settings, same context length.
  4. Score, and look at the failures, not just the aggregate. A two-point drop that is concentrated in one category is a different decision from a two-point drop spread evenly.
  5. Keep the set. It becomes the regression test for every model swap from here on.
eval.pypython
# The shape of the loop, not a library. Any OpenAI-compatible local
# server works — the base URL is the only thing that changes per engine.
import json, httpx

cases = [json.loads(l) for l in open("eval/cases.jsonl")]
servers = {"q8": "http://localhost:8080/v1", "q4": "http://localhost:8081/v1"}

def ask(base, prompt):
    r = httpx.post(f"{base}/chat/completions", timeout=120, json={
        "model": "local", "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    })
    return r.json()["choices"][0]["message"]["content"].strip()

for name, base in servers.items():
    hits = sum(ask(base, c["input"]) == c["expected"] for c in cases)
    print(f"{name}: {hits}/{len(cases)} exact match")

Rules of thumb

  • Start at Q4_K_M (or AWQ/GPTQ 4-bit on a batching server). Move up to Q6_K or 8-bit if the eval shows loss on your task; move down only if the model does not fit.
  • Prefer a bigger model at 4-bit to a smaller model at 8-bit for the same memory. This holds far more often than it fails.
  • Use an importance-matrix quantization for anything below 4 bits. Without it, the low-bit types are noticeably worse.
  • Quantize the KV cache to 8-bit before dropping the weights a tier.
  • Never mix results from two quantizations of a model in one evaluation. They are different models for the purpose of the numbers.

Quantization sets what fits; the fine-tuning guide covers the other lever on model quality, and the hardware guide turns the bits into gigabytes.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Is Q4_K_M good enough for production?

    For most chat, summarization, classification, and retrieval-augmented workloads on models of 7B and up, yes: the measured loss against 8-bit is small and often within noise. For tasks that demand exactness (code that must compile, strict structured extraction, arithmetic), evaluate before assuming. On models under 7B, treat 4-bit as a trade rather than a free lunch, and consider 6-bit or 8-bit if they fit.

  2. What is the difference between Q4_0 and Q4_K_M?

    Q4_0 is the legacy 4-bit type with a single scale per block of 32 weights and the same precision everywhere. Q4_K_M is a K-quant: super-blocks with hierarchical scales, and a mix of precisions that keeps attention and output tensors at higher bit rates. At nearly the same file size Q4_K_M is measurably better, and there is no reason to choose Q4_0 today except compatibility with a very old runtime.

  3. Does quantization make a model faster?

    Usually, because generation is bandwidth-bound and a smaller model moves fewer bytes per token. The exception is compute-bound regimes such as long prompt processing or heavy batching, where dequantization overhead can cost more than the bandwidth saves. On a batching server, FP8 or AWQ with fused kernels tends to keep the speed advantage; on a single-user machine, GGUF at 4-bit is almost always faster than FP16.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.