Build a DPO Training Dataset From Scratch: Synthetic Preference Data With Distilabel and Two Open Models

What you'll be able to do after this:

• Generate a synthetic preference dataset (chosen/rejected pairs) for DPO, ORPO, or RLHF fine-tuning from raw prompts — no hand-labeling
• Run two open models against the same instructions via HuggingFace serverless inference and use a third model as the judge, all without provisioning a GPU
• Push the finished dataset to the HuggingFace Hub ready to drop into trl.DPOTrainer in your next session

TL;DR: Distilabel generates DPO preference pairs by prompting two open models on the same instructions, then scoring with a third judge model — from prompt list to DPO-ready dataset, no hand-labeling required.

Before you can fine-tune with Direct Preference Optimization (DPO), you need rows where every entry says "prefer response A over response B and here's why." The standard path is human annotation — slow and expensive. Distilabel automates the annotation by using one model to judge another.

The pipeline in three steps

Step 1 — Generate two responses per prompt

from distilabel.llms import InferenceEndpointsLLM
from distilabel.steps.tasks import TextGeneration

# Two different models respond to each prompt
llm_a = InferenceEndpointsLLM(model_id="meta-llama/Meta-Llama-3-8B-Instruct")
llm_b = InferenceEndpointsLLM(model_id="mistralai/Mixtral-8x7B-Instruct-v0.1")

gen_a = TextGeneration(llm=llm_a, num_generations=1)
gen_b = TextGeneration(llm=llm_b, num_generations=1)

Llama 3 8B and Mixtral 8x7B both answer every prompt. Their responses are grouped into one row: one prompt, two candidate answers.

Step 2 — Score with a judge model (UltraFeedback)

from distilabel.steps.tasks import UltraFeedback

judge = InferenceEndpointsLLM(model_id="meta-llama/Meta-Llama-3-70B-Instruct")
evaluate = UltraFeedback(llm=judge)

A 70B model scores each response on helpfulness, honesty, and instruction-following. The higher scorer becomes chosen, the lower becomes rejected.

Step 3 — Format and push to Hub

from distilabel.steps import FormatTextGenerationSFT, PushToHub

format_step = FormatTextGenerationSFT()
push = PushToHub(repo_id="your-org/my-preference-dataset")

with Pipeline(name="pref-dataset") as pipeline:
    load >> [gen_a, gen_b] >> combine >> evaluate >> format_step >> push

pipeline.run(dataset=dataset)

Output: a dataset with prompt, chosen, rejected, and per-dimension scores — ready for trl.DPOTrainer.

No GPU required for data generation

All inference goes through HuggingFace's serverless Inference API. For datasets under 10K rows the free tier handles it. For larger runs (100K+), switch to a dedicated endpoint or route through Together AI or Fireworks.

Why this step matters

Data quality is what separates a fine-tune that improves behavior from one that degrades it. The UltraFeedback rubric that Distilabel applies is the same pipeline used to build alignment datasets for Zephyr and Notus. You're not inventing a scoring rubric — you're reusing one validated at research scale, in a 30-line script.

Sources: Generate a Preference Dataset with distilabel — HuggingFace Cookbook · Run in Colab · Distilabel docs — Argilla · Distilabel GitHub