GGUF, GPTQ, or AWQ? Pick the Right Quantization Format Before You Download

TL;DR: Pick GGUF for local inference (Ollama, llama.cpp, any hardware), GPTQ for legacy NVIDIA checkpoints in vLLM, and AWQ for new production GPU deployments — AWQ preserves quality better at the same bit depth.

When you browse HuggingFace for an open-weight model, you see filenames like Llama-3.3-70B-Instruct-Q4_K_M.gguf, Mistral-7B-v0.3-AWQ, or Qwen3-8B-GPTQ. Pick wrong and you'll download the wrong file, fail to load it, or get worse quality than necessary.

What you'll be able to do after this:

• Read a HuggingFace model card and pick the right format for your hardware without trial and error
• Know when Q4_K_M is good enough vs. when to reach for Q8_0 or AWQ
• Swap formats correctly when moving from local laptop dev to GPU production serving

The three formats

GGUF (the local inference format) — a single self-contained file (weights + tokenizer + config). Runs on CPU, Apple Silicon, and consumer NVIDIA/AMD GPUs. The format for Ollama and llama.cpp. Common quality tiers:

• Q4_K_M — best size/quality balance; ~4 GB for a 7B model; start here
• Q8_0 — near-lossless; ~8 GB for a 7B model; use when quality matters more than speed
• Q2_K / IQ variants — smallest, but noticeable quality loss; avoid unless you're severely VRAM-constrained

GPTQ — GPU-native quantization that finds the optimal per-layer compression. Requires CUDA. Stored as HuggingFace safetensors. Has a massive back-catalog and wide support in vLLM and transformers, but largely superseded by AWQ for new releases.

AWQ (Activation-aware Weight Quantization) — identifies the ~1% of weights that matter most for quality (via activation patterns) and keeps them at higher precision. Better output quality than GPTQ at the same bit depth. The new default for production 4-bit serving on NVIDIA GPUs.

Quickstart for each

# GGUF: pull directly via Ollama
ollama pull llama3.3:70b-instruct-q4_K_M

# AWQ: serve via vLLM
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-8B-Instruct-AWQ \
  --quantization awq \
  --max-model-len 32768

# GPTQ: serve via vLLM (legacy checkpoint)
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Mistral-7B-Instruct-v0.3-GPTQ \
  --quantization gptq

Decision table

Goal	Hardware	Format
Local dev, any hardware	CPU / Apple Silicon / any GPU	GGUF Q4_K_M
High quality local	GPU 8 GB+	GGUF Q8_0
Production GPU (new model)	NVIDIA	AWQ
Production GPU (existing checkpoint)	NVIDIA	GPTQ
Max throughput, no quality loss	A100 / H100	fp16 or fp8

The video below covers the math behind why AWQ beats GPTQ and walks through loading each format in a real session — the clearest 20-minute intro to quantization formats available.

Sources: LLM Quantization Explained: GPTQ, AWQ, QLoRA, GGUF and More (YouTube) · A Visual Guide to Quantization (Maarten Grootendorst) · GGUF vs GPTQ vs AWQ 2026 (Local AI Master) · vLLM Quantization Docs