Beyond SFT and DPO: Fine-Tune for Reasoning With GRPO and Unsloth on a Free Colab GPU

Chris Harper

4 min read

Jul 8, 2026 · 04:15 UTC

Tutorial

Fine-Tuning

HuggingFace

TL;DR: GRPO goes beyond SFT and DPO by using custom reward functions — not labels or preference pairs — to reinforce reasoning behavior; run a complete training loop on a free Colab T4 GPU with TRL's GRPOTrainer and Unsloth in under an hour.

What you'll be able to do after this:

Understand how GRPO differs from SFT (label imitation) and DPO (preference pairs) — it scores groups of model outputs and updates the policy based on which ones performed better relative to the group average
Write custom reward functions that signal correctness, format adherence, and reasoning quality without any human-labeled data
Run a complete GRPO training loop on a free Google Colab T4 GPU using Unsloth + TRL's GRPOTrainer

What GRPO is and why it's different

Supervised fine-tuning (SFT) teaches the model to imitate labeled examples. DPO teaches it to prefer one output over another via chosen/rejected pairs. GRPO (Group Relative Policy Optimization) does something fundamentally different: it generates multiple candidate outputs per prompt, scores each with a reward function, and updates the model to favor outputs that scored higher than the group average.

No labels. No preference pairs. Just a reward function — pure Python code. This is the same core technique behind DeepSeek-R1.

Why GRPO fits on a free T4. PPO (the older RL fine-tuning approach) requires a second "critic" model the same size as the policy, running in parallel to estimate value. GRPO eliminates that critic entirely. On a 1B model, this cuts memory by 40–60%, making free Colab T4 training practical.

Setup

pip install unsloth trl

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-3-1b-instruct",
    max_seq_length=2048,
    load_in_4bit=True,     # 4-bit quant — fits on free T4
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    use_gradient_checkpointing="unsloth",
)

Writing reward functions

This is the new skill GRPO adds. Each function takes a batch of completions and returns a list of float scores.

import re

def correctness_reward(prompts, completions, answer, **kwargs):
    """1.0 if the extracted answer matches ground truth."""
    responses = [c[0]["content"] for c in completions]
    extracted = [re.search(r"<answer>(.*?)</answer>", r, re.DOTALL)
                 for r in responses]
    return [1.0 if e and e.group(1).strip() == str(a) else 0.0
            for e, a in zip(extracted, answer)]

def format_reward(prompts, completions, **kwargs):
    """0.5 if output uses <think>...</think><answer>...</answer> format."""
    responses = [c[0]["content"] for c in completions]
    return [0.5 if re.search(
                r"<think>.*</think>.*<answer>.*</answer>", r, re.DOTALL)
            else 0.0 for r in responses]

Stack as many reward functions as you like. The trainer combines per-completion scores into group-relative advantages that drive the policy update.

Training loop

from trl import GRPOConfig, GRPOTrainer
from datasets import load_dataset

dataset = load_dataset("gsm8k", "main", split="train")   # grade-school math

training_args = GRPOConfig(
    output_dir="./grpo-gemma3",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    num_generations=8,       # completions per prompt for comparison
    max_new_tokens=512,
    learning_rate=5e-6,
)

trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[correctness_reward, format_reward],
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

The full notebook (Gemma 3 1B on GSM8K math) is runnable on the free Colab T4. After training, the model generates <think>...</think><answer>...</answer> traces and improves on grade-school math accuracy over the base checkpoint.

When to use GRPO vs SFT vs DPO

Method	What you need	Best for
SFT	Labeled examples (input → output)	Style, format, domain knowledge
DPO	Preference pairs (chosen vs rejected)	Tone, safety alignment
GRPO	A reward function (Python code)	Reasoning, math, verified tasks

Use GRPO when you have a verifiable objective — a math answer, code that compiles, correctly structured output — because evaluating is easy but labeling at scale is expensive.