vLLM Hits 25K TPS/GPU and Ships 3x Long-Context Throughput with Decode Context Parallelism

Two vLLM performance milestones this week signal how fast the inference stack is moving for long-context and agentic workloads.

25K total TPS/GPU on Qwen3.5 (Aug 6): vLLM reached 25,000 tokens per second per GPU running Qwen3.5-397B-A17B-NVFP4 on NVIDIA GB200 NVL72 hardware. The result uses disaggregated prefill/decode serving (separate GPU pools for the two phases), Blackwell GDN kernels, HMA cache transfer, and async scheduling improvements. Most teams won't have GB200s yet — but disaggregated serving is available in vLLM today on standard hardware and the architectural pattern applies regardless of GPU generation.

Decode Context Parallelism for long-context workloads (Aug 7): For inference on 16k–128k token contexts, vLLM's new Decode Context Parallelism (DCP) shards the KV cache across GPUs by sequence dimension rather than by tensor layer. Result: 3× higher throughput on long-context agentic tasks compared to standard tensor parallelism. This matters directly if you're serving retrieval-augmented or multi-step agentic pipelines that regularly hit large context windows.

Why it matters: If you're designing a production vLLM deployment for an agentic system, both of these techniques reduce per-request latency and cost at scale. DCP is the more immediately practical one for teams already running vLLM — it activates for long-context workloads without requiring next-generation hardware.

Sources: vLLM Reaches 25K TPS/GPU on Qwen3.5 — vLLM Blog · Efficient Decode Context Parallelism with vLLM — vLLM Blog