You Fine-Tuned It — Now Run It: Export Your Unsloth Model to GGUF and Load It Into Ollama

After QLoRA fine-tuning with Unsloth, two commands export your model to GGUF and load it into Ollama — your fine-tune becomes a named local model callable from any app.

What you'll be able to do after this:

• Export any Unsloth fine-tuned model to a portable GGUF file in one Python call, choosing quantization level (q4_k_m, q8_0, or f16)
• Load that GGUF into Ollama using an auto-generated Modelfile that preserves your chat template — the most common failure point after export
• Call your custom model from any OpenAI-compatible client, LM Studio, Open WebUI, or MCP tool that uses Ollama

Why this step is the missing link

Fine-tuning with Unsloth produces HuggingFace safetensors — 10–30 GB files that require the full HF stack to serve. GGUF is a portable, compact format that llama.cpp and Ollama load directly. A Q4_K_M quantized 7B model runs in ~4 GB RAM on any laptop with no GPU required.

Step 1: Export to GGUF (at the end of your training loop)

# After your SFTTrainer finishes:
model.save_pretrained_gguf("gguf_output", tokenizer, quantization_method="q4_k_m")
# q4_k_m: recommended — balanced size vs. quality
# "q8_0": faster export, larger file   "f16": lossless, ~14 GB for 7B

This writes two files to gguf_output/:

• unsloth.Q4_K_M.gguf — the quantized model weights
• Modelfile — Ollama config with your training chat template already embedded

The export takes 5–10 minutes on a T4 GPU (free Colab tier). On CPU-only machines, use the manual llama.cpp convert path.

Step 2: Create the Ollama model

# Transfer gguf_output/ to your local machine (scp, Drive, Hub)
# Or push directly to HuggingFace Hub from Colab:
# model.push_to_hub_gguf("your-hf-username/my-tune", tokenizer, "q4_k_m")

ollama create my-fine-tune -f gguf_output/Modelfile

Unsloth's auto-generated Modelfile contains the correct chat template — this eliminates the most common post-export failure: models that perform well during training but generate garbage in Ollama because the template doesn't match.

Step 3: Run it

ollama run my-fine-tune

# OpenAI-compatible API (any client, LM Studio, Open WebUI):
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"my-fine-tune","messages":[{"role":"user","content":"Hello"}]}'

Skip the file transfer: pull direct from HuggingFace Hub

If you pushed to Hub, Ollama pulls directly — no scp needed:

ollama run hf.co/your-username/my-fine-tune:Q4_K_M

Ollama downloads the GGUF at first run and caches it locally. This works from any machine that has Ollama installed and internet access.

Sources: Saving to Ollama — Unsloth Docs · Saving to GGUF — Unsloth Docs · Llama 3 + Ollama — Official Unsloth Colab Notebook · Import models — Ollama Docs