Guaranteed JSON from Any Local Model: vLLM Structured Outputs with XGrammar

TL;DR: vLLM's guided decoding constrains generation at the token level — XGrammar masks invalid tokens so the output is always a valid JSON object, enum member, or regex match, with zero post-processing.

What you'll be able to do after this:

• Serve any HuggingFace model with vLLM and get guaranteed JSON Schema-conformant output via guided_json
• Classify text into a fixed enum with guided_choice — zero prompt engineering, zero retries
• Understand how constrained decoding differs from Instructor's prompt-then-retry approach — and when each wins

The earlier post on Instructor covers the retry-based approach: ask the model for JSON, validate with Pydantic, retry if invalid. Constrained decoding skips the retry loop entirely. Invalid tokens are masked before sampling, so the model can't produce malformed output. This is available server-side in vLLM via the extra_body parameters in the OpenAI-compatible client.

vLLM supports four constraint modes via extra_body:

Mode	Use when
guided_json	Extracting nested structured data — JSON Schema or Pydantic model
guided_choice	Classification over a fixed label set — fastest, clearest
guided_regex	Extracting a typed string (version, date, email) from free text
guided_grammar	Generating valid SQL, code, or other formal languages

XGrammar is the default backend (auto mode in vLLM). It partitions the vocabulary by schema at startup and caches token masks, adding under 40 microseconds of overhead per token at serving time — effectively free.

Walk-through

1. Start a vLLM server

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0
# XGrammar is default; explicit: --guided-decoding-backend xgrammar

2. guided_json — extract structured data with a Pydantic schema

from openai import OpenAI
from pydantic import BaseModel
from enum import Enum

client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")

class Sentiment(str, Enum):
    positive = "positive"
    negative = "negative"
    neutral = "neutral"

class ReviewAnalysis(BaseModel):
    sentiment: Sentiment
    confidence: float
    key_phrases: list[str]

schema = ReviewAnalysis.model_json_schema()

result = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{
        "role": "user",
        "content": "Analyze: 'The API latency is excellent but the docs are thin.'"
    }],
    extra_body={"guided_json": schema}
)
import json
analysis = ReviewAnalysis(**json.loads(result.choices[0].message.content))
# Always a valid ReviewAnalysis — no try/except needed

3. guided_choice — zero-shot classification

result = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Triage this: 'App crashes on startup after login'"}],
    extra_body={"guided_choice": ["bug", "feature_request", "question"]}
)
label = result.choices[0].message.content
# Exactly one of the three strings — no parse step at all

4. guided_regex — extract a typed string

result = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Extract the version from: 'Fixed in release 2.14.3-beta'"}],
    extra_body={"guided_regex": r"\d+\.\d+\.\d+(-[a-z]+)?"}
)

These same parameters work with the offline LLM class too: pass a GuidedDecodingParams inside SamplingParams (json=, choice=, regex=, grammar=).

Sources: vLLM Structured Outputs docs · Structured Decoding in vLLM — BentoML · XGrammar paper