Before You Instruction-Tune: Teach Your Model a New Domain With Continued Pre-Training in Unsloth

Continued pre-training adapts a base model's vocabulary and patterns to your domain BEFORE instruction fine-tuning — Unsloth's free Colab does it in minutes, and the gains compound through every downstream SFT task.

What you'll be able to do after this:

• Know when CPT is worth doing (niche vocabulary, specialized formats, low-resource languages)
• Run Unsloth's QLoRA-based continued pre-training pipeline on raw domain text with a free Colab GPU
• Chain CPT → SFT for a model that understands your domain AND follows instructions

---

Every instruction fine-tune (SFT) fights the base model's priors. If your domain — medical notes, legal filings, an internal codebase, a language the model rarely saw at scale — has vocabulary and sentence patterns that are out-of-distribution for the base model, your SFT data has to teach both "what this domain is" AND "how to follow instructions." That's two jobs for one fine-tune.

Continued pre-training solves this with a "step 0": expose the base model to raw domain text as a next-token-prediction task. The model absorbs domain vocabulary and structure without changing output format. Then SFT your adapted model — you need fewer instruction pairs to hit the same quality, and domain-specific hallucinations drop significantly.

Walk-through (Unsloth, free Colab T4):

from unsloth import FastLanguageModel, UnslothTrainer, UnslothTrainingArguments
from datasets import load_dataset

# 1. Load a base (not instruction-tuned) model
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.2-3B-bnb-4bit",
    max_seq_length=2048,
)

# 2. Prepare your domain corpus as raw text
dataset = load_dataset("text", data_files={"train": "my_domain_corpus.txt"})

# 3. CPT: raw text as next-token prediction, no instruction template
trainer = UnslothTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    dataset_text_field="text",
    args=UnslothTrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_ratio=0.1,
        num_train_epochs=1,
        learning_rate=5e-5,            # lower than SFT (2e-4)
        embedding_learning_rate=5e-6,  # 1/10th LR — keeps vocab stable
        output_dir="cpt-adapter/",
    ),
)
trainer.train()
model.save_pretrained("cpt-adapter/")

Key settings: The lower learning rate (5e-5 vs SFT's typical 2e-4) prevents catastrophic forgetting of general capabilities. embedding_learning_rate at 1/10th of the main LR keeps vocabulary embeddings stable while the transformer layers adapt. After CPT, merge the adapter and use the merged checkpoint as your SFT base — you'll need less instruction data to reach the same output quality.

When CPT is worth it: Medical/legal/scientific text, a low-resource language, a proprietary codebase with unique APIs or conventions. For general-purpose assistants built on general-domain data, skip CPT and go straight to SFT.

Sources: Continued LLM Pretraining — Unsloth Blog · Unsloth CPT Documentation · Continuing Pre-Training on Raw Text — Chris McCormick