Zero Code Changes, 2.5x More Throughput: Enable vLLM's Automatic Prefix Cache

Add one flag to vLLM and get 254% throughput improvement and 78% faster time-to-first-token on any workload where requests share a prefix — zero client code changes required.

What you'll be able to do after this:

• Cut time-to-first-token by up to 78% on any vLLM deployment where requests share a system prompt or document context
• Enable the optimization with one flag — no client-side code changes required
• Know when APC helps (and the one scenario where it adds overhead, so you don't deploy it blindly)

The problem: your system prompt is recomputed on every request

When vLLM processes a request, it computes attention keys and values (the KV cache) for every prompt token from scratch. If 100 concurrent users all send a request that starts with your 2,000-token system prompt, that prefix is processed 100 separate times — even though the result is identical each time.

Automatic Prefix Caching (APC) fixes this. vLLM hashes each block of prompt tokens and stores the resulting KV tensors. The next request that shares the same prefix skips the prefill step for that cached portion entirely.

Workloads that benefit most

• Shared system prompts — a customer-service bot with a 2,000-token briefing prepended to every request
• RAG retrieval context — same document chunks retrieved for many similar questions about a shared corpus
• Multi-turn conversations — chat history grows each turn; APC avoids recomputing the full history on every new message
• Few-shot templates — fixed examples at the start of every prompt

APC provides no benefit when requests have no shared prefix, or when decoding time dominates (very long generated outputs).

Enable it in one command

# vLLM server mode — any model:
vllm serve meta-llama/Llama-3-8b-instruct --enable-prefix-caching

On vLLM V1 (the default engine since v0.7), APC is already on by default — no flag needed.

# Python offline inference:
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3-8b-instruct", enable_prefix_caching=True)

Verify it's working

vllm bench serve --model meta-llama/Llama-3-8b-instruct \
  --dataset-name sonnet --sonnet-prefix-len 100 \
  --sonnet-input-len 120 --sonnet-output-len 100 \
  --request-rate 10

Look for cache_hit_tokens in per-request metrics after the warmup period.

What to expect

Jarvis Labs benchmarked APC on Qwen3-32B with 20 queries about the same shared technical document:

• Output throughput: 427 → 1,513 tokens/sec (254% improvement)
• Time to first token: 4.3s → 0.97s (78% reduction)

The first request in a session always pays the full prefill cost (cold start). Every subsequent request with the same prefix is nearly instant.

One caveat: mixed-workload overhead

When APC is enabled but no requests actually share a prefix, there's roughly a 37% throughput overhead from the hashing and cache-lookup machinery. If your workload consists mostly of unique prompts with no shared content, benchmark your traffic pattern first before enabling it in production.

The runnable example

The official vLLM repo has a complete offline inference example at examples/offline_inference/prefix_caching.py. It demonstrates a hiring/interview scenario with a long shared context, runs a warmup generation, then compares cached vs. non-cached outputs. Drop it into Colab with !pip install vllm to see the speedup yourself.

Sources: Automatic Prefix Caching — vLLM Docs · prefix_caching.py — vLLM GitHub · vLLM Optimization Techniques (benchmarks) — Jarvis Labs · KV-Cache Wins at Production Scale — llm-d Blog