Teach Any Model to Reason Step-by-Step: GRPO, the RL Algorithm Behind DeepSeek-R1, on a Free Colab T4

GRPO is the RL algorithm behind DeepSeek-R1: score groups of responses, reward the better-than-average ones — no value model, no preference pairs. Unsloth's free Colab notebook trains a reasoning model on a T4 in under 2 hours.

What you'll be able to do after this:

• Explain why GRPO is cheaper and simpler than PPO for training reasoning models — and when it beats supervised fine-tuning
• Write a verifiable reward function for your domain (math, code, SQL, structured extraction) and plug it into Unsloth's GRPO trainer
• Run a full GRPO training run on a free Colab T4 to build a model that generates <think>…</think> traces before answering

What GRPO is

When you do supervised fine-tuning (SFT) on a reasoning dataset, the model learns to imitate the reasoning traces in your training data. GRPO (Group Relative Policy Optimization) does something fundamentally different: it explores by sampling multiple responses, evaluates each one against a reward function, and reinforces the above-average ones.

The key insight: instead of needing a value model to estimate expected reward (like PPO does), GRPO uses the group mean reward as the baseline. Sample G=8 completions for one prompt, score each, subtract the mean — the ones that did better than average get positive gradients, the rest get negative.

Prompt: "Solve for x: 3x + 7 = 22"

G=8 sampled completions:
  C1: "x = 5"          reward=1.0  (correct)
  C2: "x = 5"          reward=1.0  (correct)
  C3: "x = 5"          reward=1.0  (correct)
  C4: "x = 4"          reward=0.0  (wrong)
  C5: "x = 5"          reward=1.0  (correct)
  C6: "x = 6"          reward=0.0  (wrong)
  C7: "x = 5"          reward=1.0  (correct)
  C8: "x = 4"          reward=0.0  (wrong)

Group mean = 0.625
Advantages:
  C1: 0.375  (above mean → reinforce)
  C4: -0.625 (below mean → discourage)

No reference model scoring. No separate value head. No paired preference data. Just: sample, score, update.

Running GRPO with Unsloth on a free Colab T4

Open Unsloth's GRPO notebook on Colab. The key moving parts:

from unsloth import FastLanguageModel
from trl import GRPOConfig, GRPOTrainer

# Load a 1.5B model in 4-bit — fits in 16GB VRAM with GRPO overhead
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-1.5B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=16)

# Your reward function: the only thing GRPO needs you to write
def reward_correct_answer(completions, ground_truth, **kwargs):
    """Return 1.0 if the completion contains the correct answer, else 0.0."""
    return [
        1.0 if str(gt).strip() in c else 0.0
        for c, gt in zip(completions, ground_truth)
    ]

trainer = GRPOTrainer(
    model=model,
    reward_funcs=reward_correct_answer,  # plug in your domain's verifier
    args=GRPOConfig(
        num_generations=8,       # G — completions per prompt
        max_new_tokens=512,
        max_steps=500,
        learning_rate=5e-6,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        output_dir="grpo-reasoning-model",
    ),
    train_dataset=dataset,       # needs a "prompt" column
)
trainer.train()

Unsloth patches the attention kernels to cut VRAM by ~2×, which is what makes GRPO feasible on a free T4.

When to use GRPO instead of SFT

Situation	Better choice
You have correct reasoning traces (CoT)	SFT on those traces
You can verify correctness automatically (math, code, SQL)	GRPO
You want the model to discover better reasoning strategies	GRPO
You're adapting writing style or domain vocabulary	SFT
Your task doesn't have a clear correct/incorrect signal	SFT or DPO

GRPO excels at tasks with verifiable outcomes: the reward function is just a unit test. Code that passes tests, math answers that match, SQL that returns the right rows, structured output that satisfies a regex — all clean GRPO targets.

Sources: Train Your Own Reasoning Model with GRPO — Unsloth Docs · Train your own R1 reasoning model locally — Unsloth Blog · GRPOTrainer — TRL / Hugging Face · DeepSeek-R1 Technical Report — arXiv