Stop Forcing One Tool to Do Both Jobs: A RAG vs Fine-Tuning Decision Framework

RAG handles "what the model knows" — fine-tuning handles "how it responds." Mixing up these two jobs is the most common LLM engineering mistake and the easiest to fix.

What you'll be able to do after this:

• Identify in under five minutes whether your problem needs RAG, fine-tuning, prompt engineering, or none of the above
• Understand why most production apps use both, and how to divide the work cleanly between them
• Start a standard supervised QLoRA fine-tune on a free Colab T4 using Unsloth

The decision frame

RAG and fine-tuning solve different problems. Confusing them burns weeks on the wrong tool.

Problem	Right tool
Model doesn't know recent or private information	RAG
Answers need to cite sources or be auditable	RAG
Model consistently uses wrong format or tone	Fine-tuning
Classification labels are inconsistent	Fine-tuning
Policy adherence is unstable	Fine-tuning

RAG is the default. For roughly 80% of production use cases, it's the right starting point:

• Update source data without retraining
• Cite which document generated each answer
• Switch base models freely at any time

Fine-tune when behavior is the failure mode, not knowledge. If your pipeline retrieves the right documents but the model formats the answer wrong, that's a behavior problem — and behavior lives in the weights, not in retrieval. Fine-tuning teaches the model how to respond; RAG teaches it what to say.

Prompt engineering first. Before reaching for either tool, try few-shot examples in the system prompt. Many "we need to fine-tune" problems dissolve with 3–5 well-chosen examples. Only move to fine-tuning when the examples need to be so numerous or consistent that they'd overflow every context window.

The hybrid pattern (most production apps)

Most mature applications use both:

• RAG layer: current documents, product data, user history, tickets — anything that changes
• Fine-tuned layer: consistent output structure, domain vocabulary, classification behavior, stable tone

They compose cleanly: the retrieval system passes relevant chunks into context; the fine-tuned model knows how to turn those chunks into correctly structured responses.

Start with supervised QLoRA on a free Colab T4

Once fine-tuning is the right tool, the standard path is supervised fine-tuning (SFT) with QLoRA. Maxime Labonne's canonical HuggingFace guide walks through the full loop:

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps trl peft accelerate bitsandbytes

Key steps:

1. Load with 4-bit quantization — QLoRA compresses the base model from 32-bit to 4-bit, so an 8B model fits on a free T4 GPU (16 GB VRAM)
2. Add LoRA adapters — only ~1% of parameters are trained; the rest are frozen, preserving the base model's capabilities
3. Format in ChatML — instruction datasets need <|im_start|>system / <|im_start|>user / <|im_end|> wrapping for the model to generalize correctly
4. SFTTrainer from TRL — handles the training loop, prompt-token masking, and packing
5. Export — save to GGUF or push to the Hub for inference in Ollama or vLLM

Result: an 8B model fine-tuned on your task in under 2 hours, at zero GPU cost.

Sources: Fine-tune Llama 3.1 Ultra-Efficiently with Unsloth — Maxime Labonne, HuggingFace · RAG vs Fine-Tuning for LLMs: A Comprehensive Guide — HuggingFace · Fine-Tuning vs RAG in 2026: A Decision Guide — codercops.com