Serve Any LLM at 5× the Throughput: SGLang and RadixAttention in 10 Minutes

TL;DR: SGLang is a high-performance LLM serving framework that caches shared prompt prefixes across requests via RadixAttention — giving 4–6× throughput gains on agentic workloads, with a fully OpenAI-compatible API.

What you'll be able to do after this:

• Install SGLang and serve any HuggingFace model with a single sglang serve command
• Point any OpenAI SDK client at your local SGLang server with zero code changes
• Understand why RadixAttention makes multi-agent and RAG workloads dramatically faster when requests share the same system prompt or retrieved context

Why SGLang?

If you self-host open models, you've probably tried vLLM first. SGLang (from LMSYS, the Chatbot Arena team) is a newer serving framework built around RadixAttention — a globally shared radix-tree KV cache that reuses attention states across concurrent requests that share a common prefix.

In practice: every agentic request typically starts with a 1,000–3,000 token system prompt (tools, persona, instructions). Without prefix caching, that's 1,000–3,000 tokens recalculated for every single call. SGLang's radix tree recognizes the shared prefix and serves it from cache — only the new tokens are computed. Across a multi-agent run with 100+ calls, that gap compounds fast.

Benchmarks from LMSYS show up to 6.4× higher throughput on workloads with 60%+ shared prefixes, and the framework now powers production inference on 400,000+ GPUs worldwide.

Installation

# Python 3.10+ and CUDA 12.1+ (Ampere or later GPU)
pip install "sglang[all]"

# CPU-only / development (no GPU required — slow but functional)
pip install "sglang[all]" --extra-index-url https://download.pytorch.org/whl/cpu

Serve your first model

# Serve Llama 3.1 8B (downloads from HuggingFace Hub on first run)
sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct              --host 0.0.0.0 --port 30000

# You'll see: "The server is fired up and ready to roll!" when ready

Swap meta-llama/Llama-3.1-8B-Instruct for any HuggingFace model path — Qwen3, Mistral, Gemma, DeepSeek, etc. The --model-path accepts local paths too.

Test with curl

curl http://localhost:30000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}]
  }'

Drop-in replacement for the OpenAI SDK

from openai import OpenAI

# Change only base_url — everything else stays the same
client = OpenAI(base_url="http://localhost:30000/v1", api_key="none")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a code reviewer."},
        {"role": "user", "content": "Review this function: def add(a, b): return a - b"},
    ]
)
print(response.choices[0].message.content)

Enable prefix caching (on by default, verify it's working)

SGLang enables RadixAttention prefix caching by default. You can verify it's active by checking the server logs — look for "prefix_cache_hit_rate" in periodic stats output. On workloads with a fixed system prompt, expect to see cache hit rates of 80–95% after the first few requests warm the tree.

To explicitly enable (or tweak chunk prefill):

sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct              --port 30000              --chunked-prefill-size 512   # tune for your workload

Sources: Introduction to LLM Serving with SGLang — YouTube (Philip Kiely & Yineng Zhang, Baseten) · SGLang Quickstart — docs.sglang.io · Fast and Expressive LLM Inference with RadixAttention — LMSYS Blog