Serve Any Open-Source LLM as an OpenAI-Compatible API in 3 Commands: vLLM Quickstart

vLLM turns any HuggingFace model into a drop-in OpenAI-compatible endpoint with one command; PagedAttention and continuous batching give it 2-4x the throughput of naive serving.

What you'll be able to do after this:

• Serve a HuggingFace model locally with a real /v1/chat/completions endpoint
• Route your existing OpenAI SDK code to a self-hosted model by changing one URL
• Understand PagedAttention and continuous batching — the techniques behind vLLM's speed advantage

The 3-command quickstart

# 1. Install (Python 3.9+; CUDA 12.1+ for GPU)
pip install vllm

# 2. Serve — downloads from HuggingFace on first run
vllm serve meta-llama/Llama-3.1-8B-Instruct

# 3. Hit the OpenAI-compatible endpoint
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is RAG?"}]
  }'

The server starts at http://localhost:8000 and speaks the full OpenAI API: chat completions, completions, embeddings, and streaming. To point your existing code at it:

from anthropic import Anthropic  # or openai.OpenAI
client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

Why vLLM is faster than a naive generate() loop

Two techniques work together:

PagedAttention — the KV cache (the memory holding each token's context) is managed like OS virtual memory: allocated in non-contiguous pages rather than a single reserved block. Naive serving pre-allocates a worst-case block per sequence and wastes 20–30% of GPU RAM to fragmentation. PagedAttention eliminates that waste, so more sequences fit on the GPU simultaneously.

Continuous batching — instead of waiting for all requests in a batch to finish before processing the next, vLLM adds new requests to the running batch the moment a slot opens. GPU utilization stays high even with mixed request lengths. Together, the two techniques yield 2–4x throughput over a naive implementation.

Useful serve flags

# 4-bit GPTQ quantization: fits an 8B model on ~8 GB VRAM
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct-GPTQ-Int4 --quantization gptq

# Multi-GPU tensor parallelism
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

# Serve a local fine-tune under a custom name
vllm serve ./my-fine-tune --served-model-name my-model

No GPU? CPU-only inference works for testing:

vllm serve facebook/opt-125m --device cpu

For a free Colab T4 (15 GB VRAM), meta-llama/Llama-3.2-3B-Instruct fits comfortably without quantization.

The one-URL swap test

If you want to verify your existing OpenAI client code is self-host-ready before investing in a GPU, run vLLM with facebook/opt-125m on CPU, point your SDK at http://localhost:8000/v1, and run your integration. If it works there, it will work on a real GPU with a real model.

Sources: vLLM Quickstart — official docs · PagedAttention and Continuous Batching explained — RunPod · vLLM GitHub