You've SFT'd Your Model. Now Tell It What "Better" Means: DPO With TRL in 8 Lines

TL;DR: DPO trains a model on preferred/rejected response pairs — no reward model, no RL loop needed. TRL's DPOTrainer runs on LoRA adapters in 8 lines and works on any HuggingFace causal LM.

What you'll be able to do after this:

• Run preference alignment (DPO) on any HuggingFace causal LM with a LoRA adapter
• Build a minimal preference dataset from your own domain
• Understand when DPO adds value on top of SFT and when SFT alone is enough

Why DPO

SFT teaches the model "here is the correct output" — it requires labeled ground-truth answers. DPO teaches the model "here is what's preferred" — it requires pairs of responses to the same prompt: a better one (chosen) and a worse one (rejected). DPO fits quality judgements (writing style, tone, instruction following) where you can compare two responses but can't easily produce a perfect one from scratch.

The mechanism: DPO directly optimizes the policy to widen the log-likelihood margin between chosen and rejected responses, relative to a reference model, without training a separate reward model first. Computationally cheaper than RLHF; on par or better in most alignment tasks.

The dataset format

Three columns — prompt, chosen, rejected:

# Conversational format (recommended)
example = {
    "prompt": [{"role": "user", "content": "Explain gradient descent."}],
    "chosen": [{"role": "assistant", "content": "Gradient descent minimizes loss by moving parameters in the direction of the negative gradient..."}],
    "rejected": [{"role": "assistant", "content": "It's when the model learns stuff..."}],
}

The trl-lib/ultrafeedback_binarized dataset on HuggingFace Hub is a solid public starting point (~60k pairs across diverse prompts).

8 lines to train

from trl import DPOTrainer, DPOConfig
from datasets import load_dataset
from peft import LoraConfig

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    "Qwen/Qwen3-0.6B",            # any causal LM by name or path
    train_dataset=dataset,
    peft_config=LoraConfig(),      # LoRA keeps VRAM manageable on consumer GPUs
    args=DPOConfig(
        beta=0.1,                  # controls deviation from reference model
        learning_rate=1e-5,        # higher than full fine-tune (adapter params only)
        num_train_epochs=1,
    ),
)
trainer.train()

Building your own preference dataset

Domain	chosen	rejected	How to label
Code	passes tests	fails tests	automated test runner
Writing	follows house style	ignores it	LLM-as-judge with your rubric
Factual QA	verified answer	plausible but wrong	human review or retrieval

For code, test results are a free labeling signal: generate 4 responses per prompt from your SFT model, run tests, label passing as chosen and failing as rejected. Phil Schmid's walk-through on using synthetic preference data shows ~2,000 pairs yielding a 5% benchmark improvement on reasoning tasks.

When DPO beats SFT alone

DPO is most useful after SFT, not instead of it. SFT gets the model producing plausible outputs; DPO refines the distribution toward your quality standard. DPO without SFT often produces artifacts because the reference model (the pre-trained base) is too far from the target distribution.

Limits

• Double VRAM: DPO keeps a reference model alongside the policy during training. Use precompute_ref_log_probs=True to pre-compute reference log-probs and free the reference model, or Unsloth for ~70% less VRAM overall.
• Preference quality beats quantity. 1,000 clean pairs typically outperform 100,000 noisy ones. Contradictory labels degrade alignment instead of improving it.
• Won't rescue a bad base. DPO shifts the distribution; it doesn't rebuild a model that wasn't pre-trained or SFT'd adequately.

Sources: DPO Trainer — HuggingFace TRL docs · How to align open LLMs with DPO and synthetic data — philschmid.de · Preference alignment — smol-course (HuggingFace)