Scale Any vLLM-Backed Model to Production with Ray Serve LLM

Ray Serve LLM wraps vLLM in a production harness — queue-depth autoscaling, multi-model routing, and an OpenAI-compatible API endpoint in ~25 lines of Python.

You've loaded a model with Ollama or spun up a raw vLLM server. That works for one machine and modest load. When you need autoscaling, rolling upgrades, multi-model routing, or true production reliability, Ray Serve LLM is the missing layer.

What you'll be able to do after this:

• Deploy any Hugging Face model behind an OpenAI-compatible REST endpoint using Ray Serve LLM and vLLM
• Enable autoscaling that responds to request queue depth (not CPU) — spinning up replicas under load, scaling back down at idle
• Serve multiple models from one cluster and route traffic by model name, with no changes to the client

Install

pip install "ray[serve]" vllm

Deploy a model in ~20 lines

# serve_llm.py
from ray import serve
from ray.serve.llm import LLMServer, LLMConfig

config = LLMConfig(
    model_loading_config={
        "model_id": "meta-llama/Llama-3.1-8B-Instruct",
    },
    deployment_config={
        "autoscaling_config": {
            "min_replicas": 1,
            "max_replicas": 4,
            "target_ongoing_requests": 5,
        }
    },
    accelerator_type="A10G",   # or "T4", "A100", etc.
)

llm_server = LLMServer.from_llm_config(config)
serve.run(llm_server)

# Start
serve run serve_llm.py

# Query — standard OpenAI API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain KV caching in one paragraph."}]
  }'

Multi-model serving

Add a second LLMConfig entry with a different model_id — Ray Serve routes requests by the model field in each API call. No load balancer config needed; Ray handles it.

Why queue depth, not CPU

Traditional autoscalers trigger on CPU or memory. LLMs are I/O-bound: a GPU can be 100% utilized but still have headroom for more requests in the batch. Ray Serve autoscales on target_ongoing_requests — the number of requests currently being processed per replica. This correctly captures backpressure and scales before the queue grows too long.

What this covers beyond raw vLLM

Feature	Raw vllm serve	Ray Serve LLM
OpenAI-compatible API	Yes	Yes
Autoscaling	No	Yes (queue-depth)
Multi-model routing	No	Yes
Rolling upgrades	No	Yes
Multi-node tensor parallel	Yes	Yes
Health checks	Basic	Enterprise-grade

Sources: Serving LLMs — Ray 2.56.0 docs · Ray Serve LLM Quickstart · Announcing Native LLM APIs in Ray Data and Ray Serve — Anyscale blog