Get Up to 2.8x Faster Token Generation Without Changing Your Model: Enable Speculative Decoding in vLLM

TL;DR: Add --speculative-model and --num-speculative-tokens 5 to your vLLM serve command and get 2-3x faster output on low-temperature workloads like code generation and summarization — with zero change to output quality.

What you'll be able to do after this:

• Enable speculative decoding in vLLM with two extra CLI flags, no model changes, no quality trade-off
• Know when it cuts time-per-output-token by 2-3x vs when it slows you down (and why)
• Pick a draft model for your target and track acceptance rate to tune --num-speculative-tokens

How speculative decoding works

Standard LLMs generate one token per forward pass through the full model — each 70B-parameter pass produces exactly one token. Speculative decoding inserts a small, fast draft model (1-8B) before the expensive target:

1. Draft phase: the small model proposes K tokens in a single cheap pass (default K=5)
2. Verify phase: the large target model evaluates all K drafts in one parallel forward pass — the same compute cost as generating one token — and accepts those that match its own distribution

When acceptance rates are high (0.65+), you get 3-5 tokens per expensive target pass instead of 1, cutting time-per-output-token (TPOT) by 2-3x. The accepted tokens are mathematically identical to what the target would have generated alone — no quality trade-off.

Enable it in vLLM

The draft model must share the same vocabulary as the target (same model family works best):

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --dtype bfloat16

Key flags:

• --speculative-model — HF hub name or local path of the draft model
• --num-speculative-tokens — tokens proposed per round; 5 is a good start, raise to 8 for long low-temp outputs
• --speculative-draft-tensor-parallel-size 1 — keep draft on one GPU; target uses --tensor-parallel-size

The draft model lives in VRAM alongside the target. On 2x H100, Llama-3.1-8B fits comfortably next to Llama-3.1-70B with --gpu-memory-utilization 0.92.

When to enable it — and when to skip it

Workload	Expected result
Code generation, temperature 0.5 or below	2-3x faster TPOT
Summarization, long structured outputs	up to 2.8x faster
Chat with temperature above 0.7	minimal gain; draft acceptance drops
GPU already saturated at high QPS	1.4-1.8x slower — skip it

At high temperatures the target's distribution becomes too wide for the draft to predict, so most proposals get rejected — you pay for extra compute with no speedup.

Monitor acceptance rate

After launch, check the metrics endpoint:

curl http://localhost:8000/metrics | grep spec_decode_draft_acceptance_rate
# vllm:spec_decode_draft_acceptance_rate_perc{...} 0.72

Acceptance rate above 0.65 means real speedup. Below 0.5, try a draft model from the same model family or reduce --num-speculative-tokens.

Sources: Speculative Decoding on vLLM: A Configuration and Decision Framework (DigitalOcean) · vLLM Speculative Decoding Docs · Speculative Decoding: 2-3x LLM Inference Speedup (Introl)