One GPU, Dozens of Fine-Tuned Customers: Serve Multiple LoRA Adapters Simultaneously With vLLM

Chris Harper

3 min read

Jul 14, 2026 · 04:15 UTC

Tutorial

Self-Hosting

Fine-Tuning

LLM

vLLM's multi-LoRA mode holds one copy of the base model in VRAM and applies adapter layers per request — so dozens of fine-tuned variants share a single GPU at near-zero marginal cost per adapter.

What you'll be able to do after this:

Launch a single vLLM server that routes requests across dozens of named LoRA adapters
Configure memory limits so adapters evict cleanly when VRAM fills
Route any OpenAI-compatible client to a specific fine-tune by swapping one parameter

The problem: serving many fine-tuned models is GPU-expensive

A Llama 3.2 3B base model takes ~6GB of VRAM. Ten separate fine-tuned copies would take 60GB. With vLLM multi-LoRA, those ten adapters add roughly 100–200MB total — a rounding error.

vLLM holds one copy of the base model weights in VRAM. Each LoRA adapter is a small delta (10–100MB, depending on rank). When a request arrives for a specific adapter, vLLM injects the adapter's low-rank matrices during the forward pass. Adapters not actively in use are evicted to CPU memory and reloaded on demand.

Launch a multi-adapter server

vllm serve meta-llama/Llama-3.2-3B-Instruct     --enable-lora     --lora-modules sql=./adapters/sql-specialist                   legal=./adapters/legal-summarize                   code=./adapters/code-reviewer     --max-loras 4     --max-lora-rank 64

Key flags:

--enable-lora — activates LoRA support on the server
--lora-modules name=path ... — registers each adapter as a named model alias
--max-loras N — adapters that can be active in VRAM simultaneously; increase if you have headroom
--max-lora-rank N — must be >= the highest rank used by any adapter you load

Route requests by adapter name

Each adapter appears as a separate model in the OpenAI-compatible API:

# Use the SQL fine-tune
curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "sql",
    "messages": [{"role": "user", "content": "SELECT users created in the last 7 days"}]
  }'

# Use the base model directly
curl http://localhost:8000/v1/chat/completions   -d '{"model": "meta-llama/Llama-3.2-3B-Instruct", ...}'

In Python, swap the model= parameter in your AsyncOpenAI client — the rest of the code is unchanged.

Add adapters at runtime

curl -X POST http://localhost:8000/v1/load_lora_adapter   -H "Content-Type: application/json"   -d '{"lora_name": "medical", "lora_path": "./adapters/medical-qa"}'

Memory math

Llama 3.2 3B base: ~6GB VRAM
Each LoRA adapter at rank 64: ~50MB
20 adapters: ~1GB extra — fits comfortably alongside the base on a single A100

At scale: Llama 3.1 70B with 100 rank-16 adapters runs on a single A100 80GB. Without multi-LoRA, 100 separate 70B instances would require roughly 1,750 A100s.