Scale Beyond One GPU: DeepSpeed ZeRO + Accelerate for Multi-GPU LLM Fine-Tuning

TL;DR: DeepSpeed ZeRO-3 shards optimizer states, gradients, and model weights across GPUs — HuggingFace Accelerate wires it into any Trainer workflow with a config file and zero code changes.

What you'll be able to do after this:

• Fine-tune models too large to fit on a single GPU by sharding parameters across N GPUs with ZeRO-3
• Add DeepSpeed to any HuggingFace Trainer or SFTTrainer job by running accelerate config once
• Choose the right ZeRO stage (1, 2, or 3) based on your memory constraints and model size

Resource: DeepSpeed ZeRO Tutorial: Fine-Tune LLMs Across Multiple GPUs (Nov 2025, YouTube) — walks through fine-tuning a 3B model across two GPUs with working Accelerate config files.

Why you need it

Single-GPU QLoRA gets you surprisingly far, but when your model or batch doesn't fit — even with 4-bit loading — you need distributed training. DeepSpeed ZeRO eliminates memory redundancy across GPUs by sharding three heavy objects:

Stage	What it shards	When to use
ZeRO-1	Optimizer states	Training fits; want throughput gains
ZeRO-2	+ Gradients	Need 30–50% more headroom
ZeRO-3	+ Model params	Model doesn't fit on one GPU at all

A 7B model in bf16 uses ~14 GB of parameters alone; ZeRO-3 splits that across N GPUs so each holds 14/N GB.

Three-step setup

# 1. Install
pip install accelerate deepspeed

# 2. Generate config interactively (run once per machine or cluster)
accelerate config
# → choose: multi-GPU, DEEPSPEED, ZeRO stage 3, bf16

# 3. Launch (replaces "python train.py")
accelerate launch train.py

No changes to your training script. Accelerate patches PyTorch's training loop transparently when you use Trainer or SFTTrainer.

Your generated accelerate_config.yaml will include:

compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED
deepspeed_config:
  zero_stage: 3
  offload_optimizer_device: none   # set "cpu" to trade GPU VRAM for RAM
  offload_param_device: none       # set "cpu" for very tight VRAM budgets
  gradient_accumulation_steps: 4
mixed_precision: bf16
num_processes: 2  # number of GPUs

Set offload_optimizer_device: cpu (ZeRO-Offload) to trade GPU memory for CPU RAM — useful on consumer GPUs with ≤24 GB VRAM. ZeRO-Infinity adds NVMe offload for models that don't even fit in RAM. Combine with QLoRA (4-bit base + LoRA adapters) for the most memory-efficient multi-GPU setup.

Sources: DeepSpeed ZeRO Tutorial (YouTube) · HuggingFace Accelerate + DeepSpeed docs · Accelerate DeepSpeed usage guide