Run Any Open-Weight Model Locally: What GGUF Quantization Tags Actually Mean

Chris Harper

4 min read

Aug 10, 2026 · 04:15 UTC

Tutorial

Self-Hosting

LLM

Q4_K_M is Ollama's default and covers 99% of local setups: 4.5GB for a 7B model, roughly 96% of full-precision quality. Here's what the letters and numbers in every GGUF model tag actually mean.

What you'll be able to do after this:

Read any GGUF model tag (Q4_K_M, Q8_0, IQ4_XS, Q5_K_M) and know what it means for RAM and quality
Pull a specific quantized variant from Ollama or HuggingFace to match your hardware budget
Make an informed decision about when to trade quality for size — and understand why bigger-at-lower-precision often wins

What is GGUF?

When you run ollama pull llama3.1, Ollama downloads a GGUF file — a single portable bundle containing model weights, the tokenizer, and metadata. GGUF (GGML Unified Format) is the universal format for local LLM inference, used by Ollama, llama.cpp, LM Studio, and GPT4All.

GGUF files come in different quantization levels that compress the weights to reduce RAM and improve inference speed, at some quality cost. Think of it like JPEG compression: a Q4 quantization file is roughly 70% smaller than full-precision (fp16), while retaining ~96% of the original quality.

Reading the tags

Ollama and HuggingFace model tags encode the quantization level directly. Here's a reference table for 7B-class models:

Tag	Bits	~RAM (7B model)	Quality
`fp16`	16	~14 GB	Reference (100%)
`Q8_0`	8	~8 GB	≈99%
`Q5_K_M`	5 (mixed)	~5.5 GB	≈98%
`Q4_K_M`	4 (mixed)	~4.5 GB	≈96%
`IQ4_XS`	4 (i-quant)	~4.2 GB	≈95%
`Q3_K_M`	3 (mixed)	~3.5 GB	≈91%
`Q2_K`	2	~2.8 GB	≈82%

K-quants (Q4_K_M, Q5_K_S, Q3_K_L): "K" means mixed precision. Attention layers (sensitive to precision loss) get slightly higher bit-width; feed-forward layers (less sensitive) get lower. M/S/L stand for Medium/Small/Large within the k-quant family — K_M is the balanced sweet spot.

I-quants (IQ4_XS, IQ3_XS): "I" means importance-based quantization, which allocates bits according to weight importance rather than uniformly across all layers. Slightly smaller files than k-quants at the same nominal bit level, with similar quality.

How to pull a specific variant

Ollama defaults to Q4_K_M for official models. To override:

# Default pull — gets Q4_K_M automatically
ollama pull llama3.1

# Explicit variant by tag
ollama pull llama3.1:8b-instruct-q8_0       # higher quality, needs 8GB
ollama pull llama3.1:70b-instruct-q4_K_M    # 70B at Q4, ~40GB

# Pull GGUF models directly from HuggingFace
# bartowski maintains regularly updated, high-quality GGUF quants
ollama run hf.co/bartowski/Qwen3-8B-GGUF:Q4_K_M
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Choosing a quantization level

Match the quantization to your available RAM:

≤8 GB RAM (laptop, Pi): Q4_K_M or IQ4_XS for 7B models
16 GB RAM: Q5_K_M or Q8_0 for 7B/8B; Q4_K_M for 13B
24 GB GPU (RTX 3090/4090): Q8_0 for 13B; Q4_K_M for 70B
Apple Silicon (48 GB+): Q8_0 for 70B models is practical

Counter-intuitive rule: A larger model at lower quantization almost always beats a smaller model at higher quantization. A 70B at Q4_K_M (~40 GB) will outperform an 8B at Q8_0 (~8 GB) for most reasoning tasks, if you have the RAM to run it at GPU speed.

The quality degradation below Q4 (Q3, Q2) becomes noticeable on reasoning-heavy tasks — responses get less coherent, facts less reliable. Q4_K_M is the floor for anything you'd put in front of a user.

Why this matters for your stack

Every AI application that runs local inference — for cost, privacy, offline capability, or regulatory reasons — starts with this choice. Pick Q2 and you'll save RAM but degrade reasoning noticeably. Pick Q8 without enough RAM and the model spills to CPU, killing inference speed. Q4_K_M is the rational default because most hardware can run it fully on GPU, at a quality level where the degradation is invisible for real tasks.