Serve Open-Source Models Faster Than vLLM: SGLang's RadixAttention and Strict JSON Output

SGLang is an OpenAI-compatible LLM inference server that shares KV cache across requests with the same prefix (system prompts, tool lists) — giving 3–5x throughput on agent workloads — and enforces JSON Schema output at the token level.

What you'll be able to do after this:

• Serve any HuggingFace model locally with a drop-in OpenAI-compatible API in under 5 minutes
• Get 3–5× higher throughput on agent workloads where multiple requests share a long system prompt or tool definition list
• Enforce strict JSON Schema output so your pipeline receives valid structured data every time — no retries, no post-processing

Why SGLang instead of vLLM or Ollama?

Each tool has a lane:

Tool	Best for
Ollama	Developer convenience; one command, great for local chat
vLLM	Production serving; strong multi-LoRA adapter support
SGLang	Agent-first workloads; RadixAttention reuses KV cache across requests sharing a prefix

The last row matters for agent loops: a 2,000-token system prompt plus tool definitions sent with every tool call is computed once in SGLang and amortized across every request in the batch. In benchmarks, SGLang runs 29% faster than vLLM on H100 GPUs and up to 6× faster in RAG scenarios with long shared context.

Note: HuggingFace retired TGI (their older inference toolkit) in 2026 and now recommends SGLang alongside vLLM.

Install and launch a model (5 minutes)

# GPU machine with CUDA 12.1 + Python 3.10+
pip install "sglang[all]" \
  --find-links https://flashinfer.ai/whl/cu121/torch2.3/flashinfer/

# Start the server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --tp 1 \
  --mem-fraction-static 0.88

Or via Docker (no CUDA install needed):

docker run --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 30000:30000 \
  lmsysorg/sglang:latest \
  python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --port 30000 --tp 1

Smoke-test it:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct",
       "messages": [{"role": "user", "content": "Hello!"}],
       "max_tokens": 32}'

Point any existing OpenAI client at http://localhost:30000/v1 — it's a drop-in replacement.

Enforce strict JSON Schema output

This is where SGLang earns its place in an agent pipeline. Pass response_format with json_schema and strict: true, and the model is constrained at the token level — it physically cannot produce output that violates the schema:

import requests, json

schema = {
    "type": "object",
    "properties": {
        "intent": {"type": "string", "enum": ["buy", "sell", "hold", "unknown"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1}
    },
    "required": ["intent", "confidence"],
    "additionalProperties": False
}

resp = requests.post("http://localhost:30000/v1/chat/completions", json={
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Classify: 'NVIDIA beat earnings estimates by 40%'"}],
    "max_tokens": 64,
    "response_format": {
        "type": "json_schema",
        "json_schema": {"name": "signal", "schema": schema, "strict": True}
    }
})

result = json.loads(resp.json()["choices"][0]["message"]["content"])
# guaranteed to match schema — no try/except, no retries
print(result)  # {"intent": "buy", "confidence": 0.87}

When to choose SGLang

Choose SGLang when your workload has long shared prefixes (agent system prompts, RAG context, few-shot examples) and you need structured output guarantees. Choose vLLM when you need production-grade multi-LoRA adapter hot-swapping. Use Ollama when you just need a fast local chat interface.

Sources: SGLang official docs — docs.sglang.io, SGLang fast inference guide — markaicode.com, SGLang vs vLLM review — chatforest.com