Skip the Second Training Pass: ORPO Bakes Preference Alignment Into Supervised Fine-Tuning

ORPO combines supervised fine-tuning and preference alignment in a single training pass — no reference model, no separate DPO step, and a single beta scalar to tune instead of a full KL penalty.

The standard two-step alignment pipeline: fine-tune on instruction data (SFT), then run a preference pass with a frozen reference model (DPO). Two training runs, twice the VRAM, twice the checkpointing. ORPO (Hong et al., 2024) folds both into one pass: it adds an odds-ratio penalty to the next-token prediction loss so the model learns task behavior and preference alignment simultaneously.

Why it needs less memory: DPO keeps a frozen copy of the base model in memory throughout training (the reference for KL divergence). ORPO drops the reference model entirely. On a 7B model that's ~14 GB saved — enough to go from needing an A100 to running on a free Kaggle T4.

What you'll be able to do after this:

• Align a fine-tuned model to human preferences in one training run instead of two
• Run full ORPO training on a free T4 (Qwen2 0.5B fits in under 8 GB VRAM)
• Reuse your existing DPO preference dataset — the format is identical (prompt, chosen, rejected)

Minimal working example

pip install trl datasets transformers accelerate bitsandbytes

from datasets import load_dataset
from trl import ORPOConfig, ORPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "Qwen/Qwen2-0.5B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Same format as DPO: prompt / chosen / rejected
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")

config = ORPOConfig(
    output_dir="qwen2-orpo",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    learning_rate=8e-6,
    beta=0.1,        # preference weight (lambda in the paper)
    max_length=512,
)

trainer = ORPOTrainer(
    model=model, args=config,
    train_dataset=dataset, tokenizer=tokenizer
)
trainer.train()
trainer.save_model()

The one hyperparameter that matters

beta (called lambda in the original paper) controls how strongly the model penalizes rejected responses. Too high: the model becomes overly conservative. Too low: preference alignment is weak. 0.1 is the TRL default and a robust starting point for instruction-following.

Dataset compatibility

If you already have a DPO preference dataset, ORPO uses the exact same prompt/chosen/rejected schema — no reformatting needed. trl-lib/ultrafeedback_binarized (60k preference pairs) works out of the box.

Or use the CLI

accelerate launch examples/scripts/orpo.py \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
  --dataset_name trl-lib/ultrafeedback_binarized \
  --num_train_epochs 1 \
  --output_dir qwen2-orpo

For a complete walkthrough with Llama 3 8B, QLoRA configuration, chat template setup, and benchmark evaluation, see mlabonne's Fine-tune Llama 3 with ORPO — it includes a free Colab notebook that runs end-to-end on a T4.

Sources: ORPO Trainer — TRL Docs, Fine-tune Llama 3 with ORPO — mlabonne / HuggingFace Blog, ORPO: Monolithic Preference Optimization without Reference Model (arXiv 2403.07691)