GGUF vs AWQ vs GPTQ: what 4-bit quantization really costs in quality and latency

You fine-tuned a model, it passes the eval gates, and now finance asks why a 16 GB checkpoint needs a 40 GB GPU to serve. The answer is 4-bit quantization - and the follow-up question, which 4-bit format, has three incompatible answers depending on where the model runs. This post compares GGUF, AWQ and GPTQ on the two numbers that matter - quality kept and tokens per second - and gives you a decision rule that survives contact with production.

The three formats in one paragraph each

GGUF (llama.cpp) is a file format: a single .gguf blob with the tokenizer, metadata and quantized tensors, built for CPU/Metal/CUDA inference on a laptop or a small box. Quantization levels like Q4_K_M are hand-tuned mixes - important tensors get 5-6 bits, the rest 4.

GPTQ is an algorithm: it quantizes weights one layer at a time, using a calibration set to adjust the remaining weights so the layer’s output changes as little as possible. Grouped variants (g128) re-fit the scale every 128 weights, which is where most of the quality win lives.

AWQ (Activation-aware Weight Quantization) protects the ~1% of weights that see the largest activations instead of treating all weights equally. It quantizes for the GPU: its output is a safetensors checkpoint that vLLM and TensorRT-LLM load directly with a fused Marlin kernel.

Setup

• Base model: Llama-3-8B-Instruct, fp16 baseline (16.1 GB)
• Quality: perplexity on wikitext-2 via llama-perplexity, plus our internal 200-prompt regression eval
• Latency: vLLM benchmark_serving.py, 1000 prompts at 16 req/s, A100 40 GB; GGUF measured with llama.cpp server on the same box
• Quantization: AWQ and GPTQ produced with AutoAWQ / AutoGPTQ, 1024 calibration samples, group size 128

Quality: 4-bit is nearly free, 3-bit is not

Three things to take from the run:

1. Grouped 4-bit variants cluster within 2-4% of fp16. On the internal regression eval, AWQ lost 0.4 points of accuracy - well inside the gate tolerance we set in the fine-tuning pipeline post.
2. The group size matters more than the algorithm. Ungrouped GPTQ collapses (+15.9% perplexity); the same GPTQ at g128 is competitive with AWQ.
3. Q3 is a cliff. Q3_K_M saves 1 GB over Q4_K_M and pays for it with a 13% perplexity jump. Unless the model physically does not fit otherwise, stay at Q4 or above.

Latency: the runtime matters more than the format

The headline: AWQ and GPTQ under vLLM beat fp16 on throughput at high concurrency - not because 4-bit arithmetic is magically faster, but because the weights take 11 GB less VRAM, and that memory becomes KV cache. FP16 hits its context budget around 34 concurrent requests; the 4-bit checkpoints still have headroom at 64.

GGUF tells a different story. llama.cpp does continuous batching now, but it is not vLLM: aggregate throughput at 8 concurrent users is a fraction of what AWQ/vLLM sustains. Where GGUF wins is the other end of the spectrum - a single user at 58 tok/s on a box with no datacenter GPU at all, including an M-series laptop over Metal.

Decision rule

If you are…	Ship	Why
Serving production traffic on GPUs	AWQ (or grouped GPTQ)	Fused kernel + KV-cache headroom; smallest quality hit under vLLM
Running on a laptop, edge box, or CPU	GGUF Q4_K_M	Only format llama.cpp/ollama run everywhere; Q4_K_M is the quality sweet spot
Squeezing a 70B into one 24 GB card	GGUF Q4_K_M with partial offload	ExLlama/GPTQ exists, but llama.cpp offload is the boring, working path
Below 4 bits to fit at all	Reconsider the model size	A Q8 3B beats a Q2 13B on quality in every comparison we ran

Two operational notes:

• Quantize after merging. If you fine-tuned with LoRA, merge the adapter into the base weights first, then quantize - quantizing the base and adding the adapter back reintroduces fp16 memory and breaks the calibration assumptions.
• Re-run the eval gates on the quantized artifact, not on the fp16 checkpoint. The quantization step is a model change; the same regression gates that guard fine-tuning should guard it.

Related posts

• vLLM vs TGI vs TensorRT-LLM: self-hosted LLM serving latency - the serving layer where AWQ and GPTQ checkpoints actually run.
• LoRA vs QLoRA vs full fine-tuning: the GPU memory, cost, and eval-score tradeoffs - the upstream decision; quantization happens after the merge.
• Fine-tuning pipelines need regression gates before the first run - where the quantized artifact’s eval delta belongs.
• Fine-tuning data curation: eval set construction - build the eval set before you need it to catch a quantization regression.
• Enterprise AI: Self-Hosted Llama 3.1 70B on vLLM vs Azure OpenAI - Cost and Latency at 1.4 B Tokens a Month - the fleet-level cost model that quantization changes.