TRL's GRPO Trainer Gets Continuous Batching: ~1.25x Throughput for Reasoning RL With No Extra Infrastructure

TRL's GRPOTrainer now supports continuous batching for rollout generation, cutting idle GPU time during reasoning RL training with no server setup.

GRPO training is generation-heavy: producing the N rollouts (completions the model judges itself on) is typically the bottleneck. Standard batching waits for all N sequences to finish; continuous batching fills idle GPU slots as sequences complete, keeping utilization high. On an A100 80GB with Llama-3.2-1B and GSM8K at N=32: ~1.25x throughput versus standard batching, with lower peak memory at the same time.

It's a drop-in upgrade — set use_vllm=False in GRPOConfig (continuous batching is now the in-process default). TRL pre-allocates max_memory_percent=0.5 of free VRAM for rollouts, leaving headroom for the backward pass. Async continuous batching — overlapping generation with the gradient update — is in progress (PR #5781).

Why it matters: Reasoning RL fine-tuning (GRPO, DAPO) is the fastest-growing method for building capable open models. Any free throughput gain in the rollout phase directly reduces GPU-hours per training run.

Sources: Continuous batching for GRPO, now in TRL — HuggingFace Blog · GRPOTrainer — HuggingFace