Your Fine-Tune's Report Card: Catch Catastrophic Forgetting with lm-evaluation-harness in 5 Commands

Fine-tuning without evaluation is flying blind -- lm-evaluation-harness runs MMLU, GSM8K, and 200+ other benchmarks on your base and fine-tuned model in minutes so you see what got better and what broke.

What you'll be able to do after this:

• Run standardized benchmarks on any HuggingFace checkpoint in under 10 minutes using the same tool that powers the Open LLM Leaderboard
• Spot catastrophic forgetting before it breaks your fine-tune in production
• Generate a base-to-fine-tune score delta that makes model quality reviewable by your whole team

Resource: EleutherAI lm-evaluation-harness -- the open-source evaluation framework behind the HuggingFace Open LLM Leaderboard

Why this step gets skipped

Most developers fine-tune a model, try a few prompts, see plausible outputs, and ship. The hidden risk is catastrophic forgetting: your model gets 15% better at your task and quietly loses 8% on everything else. lm-evaluation-harness makes this visible in 5 commands.

Install

pip install lm-eval

No dataset downloads upfront -- task configs are pulled from YAML files managed by the project.

Step 1: Baseline the base model

lm_eval \
  --model hf \
  --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct \
  --tasks mmlu,gsm8k \
  --num_fewshot 5 \
  --device cuda:0 \
  --batch_size 8 \
  --output_path ./results/base/

The --num_fewshot 5 flag is critical: it matches how the Open LLM Leaderboard scores MMLU. Without it your scores are 3-8 points lower and incomparable to any published number.

Step 2: Run the same eval on your fine-tune

lm_eval \
  --model hf \
  --model_args pretrained=./checkpoints/my-fine-tune \
  --tasks mmlu,gsm8k \
  --num_fewshot 5 \
  --device cuda:0 \
  --batch_size 8 \
  --output_path ./results/finetuned/

Step 3: Compare the delta

import json
base = json.load(open("results/base/results.json"))
ft   = json.load(open("results/finetuned/results.json"))
for task in ["mmlu", "gsm8k"]:
    b = base["results"][task]["acc,none"]
    f = ft["results"][task]["acc,none"]
    print(f"{task}: {b:.3f} -> {f:.3f}  (delta {f-b:+.3f})")

A healthy fine-tune holds MMLU within ~2 points of the base while improving on your target task. A drop of more than 3 points on MMLU or more than 4 points on GSM8K is a red flag for catastrophic forgetting -- the dataset is too narrow or the learning rate is too high.

Faster eval with the vLLM backend

Add --model vllm instead of --model hf for 3-5x faster evaluation on a Colab T4 or A100:

lm_eval \
  --model vllm \
  --model_args pretrained=./checkpoints/my-fine-tune,dtype=float16 \
  --tasks mmlu,gsm8k \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path ./results/finetuned/

Add it to CI

Once you have a baseline, a simple pass/fail check prevents regressions automatically:

python -c "
import json, sys
b = json.load(open('results/base/results.json'))['results']['mmlu']['acc,none']
f = json.load(open('results/finetuned/results.json'))['results']['mmlu']['acc,none']
if (f - b) < -0.02:
    print(f'FAIL: MMLU dropped {f-b:+.3f}', file=sys.stderr); sys.exit(1)
print(f'PASS: MMLU delta {f-b:+.3f}')
"

Sources: EleutherAI lm-evaluation-harness · HuggingFace Open LLM Leaderboard · Evaluating fine-tuned LLMs guide · lm-eval-harness 2026 tutorial