Instruction-Tune a Llama 8B on Your Own Data for Free: QLoRA and Unsloth on Colab's T4 GPU

TL;DR: QLoRA + Unsloth fit an 8B instruction fine-tune into ~5 GB of VRAM and about 30 minutes on Colab's free T4 — you get a portable LoRA adapter you can export as GGUF or push to HuggingFace Hub.

What you'll be able to do after this: fine-tune an open-weight model on domain-specific data without a paid GPU, then run the adapter locally in Ollama or deploy it on your own server.

Three things to know before you open the notebook:

1. QLoRA quantizes the base model to 4-bit NF4, cutting Llama 3.1 8B from ~16 GB to ~5 GB VRAM — that's what fits it on a free T4 (16 GB total). The LoRA adapter trains on top; only the adapter weights update during training. Unsloth claims 2× training speed and ~60% less VRAM versus vanilla HuggingFace TRL, measured on Llama/Mistral on NVIDIA GPUs.
2. LoRA rank controls the adapter size. At rank 16 (the default), about 1.5% of the model's parameters train, saving to a 50–100 MB adapter file. Lower rank = faster and smaller; higher rank = more capacity but slower.
3. 100 training steps ≈ 30 minutes on the free T4. Enough to verify your data pipeline and see whether the loss is dropping. Not enough for production — plan 500–1,000 steps for real tasks.

Walk-through (Llama 3.1 8B Alpaca notebook):

1. Open the official Unsloth Llama 3.1 8B Alpaca Colab. Runtime → Change runtime type → T4 GPU.
2. Run the install cell (~2 min). Unsloth handles all dependencies.
3. The notebook loads unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit — a pre-quantized checkpoint, so no on-the-fly quantization overhead.
4. max_seq_length = 2048 is the default; drop to 1024 if you hit out-of-memory. r = 16, lora_alpha = 16 for the adapter.
5. Swap the dataset cell for your own JSONL with instruction / input / output fields, or leave it on the Alpaca dataset for a baseline run.
6. SFTTrainer uses sequence packing — it fills each batch to the max length instead of padding, roughly doubling effective throughput for short examples.
7. Run all. Loss should drop by step 20; if it doesn't, check your chat-template formatting.
8. Export: model.save_pretrained_gguf("adapter", tokenizer, quantization_method="q4_k_m") produces a GGUF you can load directly in Ollama.

The limits that matter:

• 100 steps is a diagnostic run. Production use typically requires 500–1,000 steps on 1k–10k examples — 3–6 hours on the free T4, or ~30 minutes on a rented A100.
• The free T4 tops out around 7–8B parameters in 4-bit. A 13B model requires Colab Pro or a rented GPU.
• Saving the merged model (adapter + base at full precision) requires more RAM than the T4 has. Push only the adapter to HuggingFace Hub and merge locally if you need the full checkpoint.
• Unsloth's speed and VRAM claims are measured on NVIDIA GPUs; results vary on other hardware.

Sources: Unsloth Llama 3.1 8B Alpaca Colab notebook · Unsloth notebooks overview · Teach a 3B Model to Call Functions with QLoRA + Unsloth on Free Colab T4 — datahacker.rs · [LoRA] Unsloth Fine-Tuning Guide (Jan 2026, YouTube)