Your Fine-Tune Learns the Wrong Thing Without This: Chat Templates and TRL’s SFTTrainer

TL;DR: Pass your dataset as {"messages": [...]} rows and set assistant_only_loss=True. Without that flag, your model trains on user and system turns too — a silent mistake that shows up as nonsense outputs, not an error.

Fine-tuning tutorials almost always skip straight to training. The dataset format step gets one sentence. That’s backwards — format mistakes are the most common cause of a fine-tuned model that behaves wrong from the first token.

What you’re building toward

After this tutorial you’ll be able to take any Q&A or conversation dataset, format it correctly for TRL’s SFTTrainer, and confirm your model is training only on the response tokens — not on the question or system prompt.

The wrong-format problem

Standard next-token training computes loss on every token in the sequence. If your training example is:

[SYSTEM] You are a helpful assistant.
[USER] What is a context manager?
[ASSISTANT] A context manager handles setup and teardown...

...and you pass it as a flat string, the model learns to predict the system prompt, the user turn, and the answer. During inference it has no system turn to predict — so it invents one, typically by echoing the structure it saw in training.

The correct format

TRL’s SFTTrainer accepts a messages column with standard role/content pairs:

# Each row in your dataset
{
    "messages": [
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user",   "content": "Explain what a context manager does."},
        {"role": "assistant", "content": "A context manager handles setup and teardown automatically using Python’s `with` statement..."}
    ]
}

TRL applies the model’s chat template automatically — it handles special tokens, turn boundaries, and the generation markers that tell the loss function which tokens to train on.

Training with assistant_only_loss

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B-Base",
    args=SFTConfig(
        output_dir="./my-model",
        assistant_only_loss=True,   # Only train on assistant turns
        packing=True,               # Pack multiple short examples per sequence
        max_seq_length=2048,
    ),
    train_dataset=load_dataset("your-org/your-dataset", split="train"),
)
trainer.train()

assistant_only_loss=True uses the chat template’s {% generation %} / {% endgeneration %} markers to mask loss on every token except the assistant’s response. TRL auto-patches this for Qwen3, Llama 3, SmolLM3, and most HuggingFace-canonical models — check the TRL model compatibility list before using a custom checkpoint.

Three-bullet takeaway

1. Training on a flat string trains on everything, including the prompt structure — the model learns to hallucinate context it won’t have at inference.
2. messages format + assistant_only_loss=True tells TRL exactly which tokens to learn from.
3. TRL handles tokenization, special tokens, and turn masking automatically once the format is right — you don’t write any of that yourself.

Real limits

• assistant_only_loss requires a chat template with generation markers. TRL patches the most common models; for others you may need to add {% generation %} markers to the template manually.
• Default max_seq_length=1024 truncates longer examples silently — raise it (or use packing) if your responses are long.
• packing=True concatenates short examples into full sequences for efficiency; disable it if your dataset mixes very long and very short responses (padding artifacts can appear at sequence boundaries).

Sources: TRL SFTTrainer — huggingface.co/docs/trl/sft_trainer · smol-course Ch. 1: Instruction tuning — github.com/huggingface/smol-course · AMD ROCm TRL fine-tuning tutorial (independent)