From pip install vllm to a Production OpenAI-Compatible API: The Five Commands That Matter

TL;DR: vLLM turns any HuggingFace open-weight model into an OpenAI-compatible HTTP API in one command — continuous batching and PagedAttention give you production throughput; here are the flags and metrics that matter before you put it in front of traffic.

What you will be able to do after this:

• Launch a vLLM server that speaks the OpenAI chat completions API
• Configure it for production with auth, memory limits, and context capping
• Monitor KV cache pressure and queue depth before something breaks

vLLM is the standard GPU serving framework for open-weight deployments. Its two core mechanisms: PagedAttention — a paged virtual memory manager for the KV cache, which eliminates memory fragmentation that wastes GPU RAM in naive serving — and continuous batching, which adds new incoming requests to an in-flight batch rather than waiting for a batch boundary. Together they turn the same GPU into substantially higher effective throughput than a simple request-per-slot server. The trade-off: NVIDIA GPU required; AMD is in beta; Apple Silicon (MPS) is not supported.

Install

pip install vllm==0.27.1   # pin the version; minor releases break flag names
python -c "import torch; print(torch.cuda.get_device_name(0))"  # verify GPU

Start the dev server

export HF_TOKEN="hf_..."   # for gated models like Llama
vllm serve meta-llama/Llama-3.1-8B-Instruct     --host 127.0.0.1     --port 8000     --gpu-memory-utilization 0.85

--gpu-memory-utilization 0.85 leaves 15% headroom for CUDA context overhead. Skip it and you will hit OOM under concurrent load.

Production configuration

vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4     --host 0.0.0.0     --port 8000     --gpu-memory-utilization 0.88     --max-model-len 4096     --max-num-seqs 64     --api-key "$(openssl rand -hex 32)"

Set --api-key before binding to 0.0.0.0 — unauthenticated access returns HTTP 401 once the key is set. --max-model-len caps context per request; set it to what your use case actually needs, not the model maximum. Shorter context = more requests fit in KV cache simultaneously. --max-num-seqs caps concurrent requests.

Call it from any OpenAI-compatible client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in one paragraph."}],
)
print(response.choices[0].message.content)

Three metrics to watch from day one

curl localhost:8000/health          # 200 OK = server up; check before traffic
curl localhost:8000/metrics | grep -E "kv_cache|requests_waiting|requests_running"

• vllm:kv_cache_usage_perc — when this approaches 1.0, new requests queue rather than start
• vllm:num_requests_waiting — sustained > 0 means you are over capacity; the fix is fewer concurrent users or a smaller --max-model-len
• vllm:num_requests_running — active in-flight requests

Wire these to Grafana before you route production traffic. KV cache pressure is the first failure mode, not CPU.

When to pick vLLM vs. alternatives

Use case	Pick
NVIDIA production serving, multi-LoRA adapter hot-swapping	vLLM
Long shared prefixes — agent system prompts, RAG context	SGLang (29% faster than vLLM on H100 in independent benchmarks)
Local chat, quick experiments, Apple Silicon	Ollama

Where it breaks: MoE models with large expert pools (e.g. 320B total parameters) require multi-GPU tensor parallelism — set --tensor-parallel-size N to match your GPU count; a single GPU cannot hold the full model. AMD ROCm support is in beta and not production-ready. vLLM serves plain HTTP — add TLS termination at your reverse proxy layer, not inside vLLM.

Sources: vLLM Quickstart — docs.vllm.ai · How to Deploy vLLM in Production 2026 — Markaicode · Install vLLM on Linux for Production — computingforgeeks.com · vLLM Production Deployment 2026 — Spheron Blog