Your Fine-Tune Might Be Lying to You: Measure It With lm-evaluation-harness

Chris Harper

4 min read

Aug 15, 2026 · 12:04 UTC

Tutorial

Fine-Tuning

HuggingFace

Training loss drops — but your fine-tuned model can still quietly break on everything it wasn't tested on. Three layers of evaluation catch regressions before production does.

What you'll be able to do after this:

Detect catastrophic forgetting before shipping a fine-tuned model
Run standard benchmarks (MMLU, ARC, GSM8K) on any HuggingFace model with one command using lm-evaluation-harness
Know when to use benchmark evals vs task-specific evals vs LLM-as-judge

The hidden trap

Training loss goes down. The model responds fluently. But did it improve on your task — without quietly regressing on everything else? Most devs skip evaluation and ship on vibes. The risk is catastrophic forgetting: a model fine-tuned on customer support data might score 10–15 points lower on math reasoning (GSM8K) after just two epochs, with no warning in the training logs.

Three evaluation layers catch this before users do.

Layer 1: Your held-out eval set (task accuracy)

Before training, set aside 10–20% of your dataset as a held-out test set. Score your fine-tuned model against it after training:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "./outputs/my-fine-tuned-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)

test_data = load_dataset("json", data_files="test.jsonl")["train"]

correct = 0
for example in test_data:
    inputs = tokenizer(example["prompt"], return_tensors="pt").to("cuda")
    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=256)
    response = tokenizer.decode(output[0], skip_special_tokens=True)
    if example["expected"] in response:
        correct += 1

print(f"Task accuracy: {correct/len(test_data)*100:.1f}%")

For structured outputs (JSON, SQL, code), use exact-match. For freeform text, use ROUGE or BERTScore. This is your primary signal.

Layer 2: Benchmark regression with lm-evaluation-harness

lm-evaluation-harness (EleutherAI) is the framework behind the HuggingFace Open LLM Leaderboard — the standard infrastructure for evaluating models at benchmark scale. One command runs your model on 200+ tasks. Run it on the base model first, save the results, then compare after fine-tuning.

pip install lm-eval

# Step 1: baseline the base model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
  --tasks mmlu,gsm8k,arc_challenge \
  --device cuda:0 --batch_size 8 \
  --output_path ./eval-results/base

# Step 2: eval your fine-tuned model with the same tasks
lm_eval --model hf \
  --model_args pretrained=./outputs/my-fine-tuned-model \
  --tasks mmlu,gsm8k,arc_challenge \
  --device cuda:0 --batch_size 8 \
  --output_path ./eval-results/fine-tuned

Then diff the result JSON files. What to watch:

MMLU: general knowledge across 57 subjects — a broad regression signal
GSM8K: grade-school math — catches reasoning degradation fast
ARC-Challenge: science questions requiring multi-step reasoning

Interpreting the delta:

GSM8K / MMLU drop	Verdict
0–3%	Normal — fine-tuning always trades some generality
3–10%	Investigate — check training epochs, learning rate, data mix
10%+	Catastrophic forgetting — roll back; revisit LoRA rank, dataset balance, or add a regularization term

Layer 3: LLM-as-judge for generative quality

Automated metrics can't assess response quality for open-ended tasks. For that, use a frontier model as a judge:

import anthropic, json

client = anthropic.Anthropic()

def judge_response(prompt: str, response: str, rubric: str) -> dict:
    result = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": f"""Rate this response on a 1–5 scale.

Rubric: {rubric}

User prompt: {prompt}
Model response: {response}

Return JSON only: {{"score": 1-5, "reason": "one sentence"}}"""
        }]
    )
    return json.loads(result.content[0].text)

# Run on 50–100 held-out examples
results = [
    judge_response(ex["prompt"], generate(model, ex["prompt"]), rubric)
    for ex in eval_set
]
avg_score = sum(r["score"] for r in results) / len(results)
print(f"LLM-judge avg score: {avg_score:.2f}/5.0")

Run this against your base model too to get a baseline. A fine-tune that scores below the base on the judge is a regression.

Which layer for which task

Task type	Layer 1	Layer 2	Layer 3
Structured output (JSON, SQL, code)	Exact-match	Always	Optional
Freeform generation (chat, summaries)	ROUGE	Always	Required
Any fine-tune	Required	Required	Recommended