Run Any HuggingFace Model in Production: TGI's Continuous Batching and Flash Attention in One Docker Command

HuggingFace Text Generation Inference (TGI) deploys any of 100,000+ open models as a production-grade, OpenAI-compatible server in one docker run — continuous batching, Flash Attention, and Paged Attention included.

What you'll be able to do after this:

• Launch TGI from any HuggingFace model ID with a single Docker command
• Understand how continuous batching, Flash Attention, and Paged Attention let TGI serve far more concurrent users than naive inference on identical hardware
• Query your running server with Python, curl, or any existing OpenAI-SDK client

TGI is HuggingFace's production serving framework — the same engine that powers the HuggingFace Inference API. While vLLM excels for fine-tuned checkpoints, TGI's native Hub integration makes it the fastest path from "model card" to "running HTTP endpoint."

The one-command launch

model=teknium/OpenHermes-2.5-Mistral-7B
volume=$PWD/data   # cache weights between restarts

docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $volume:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id $model

TGI pulls and caches weights on first run. Swap the model ID for any architecture TGI supports: Llama, Mistral, Qwen, Gemma, Falcon, and more.

Query with the OpenAI-compatible API

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="-")

response = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "What is continuous batching?"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

# Or with curl
curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"tgi","messages":[{"role":"user","content":"Hello"}]}'

Any tool that speaks the OpenAI API (LangChain, LlamaIndex, Open WebUI, your existing SDK clients) works against TGI unchanged.

Why TGI is faster than naive inference

Continuous batching — TGI's router slots new requests into active batches as soon as a slot frees up, instead of waiting for an entire batch to finish. This eliminates idle GPU cycles and increases requests-per-second at the same latency percentiles.

Flash Attention — Variable-length sequences are computed without padding, cutting VRAM consumption and speeding attention on long prompts.

Paged Attention — The KV cache is split into fixed-size pages allocated on-demand, preventing memory bloat under concurrent load. Pages can be shared between requests (e.g., a shared system prompt).

Combined: TGI typically serves 5–10× more concurrent users than a plain model.generate() loop on the same GPU.

Useful flags

# Tensor parallelism across 2 GPUs (for 70B+ models)
--num-shard 2

# 4-bit NF4 quantization (fits a 13B model into ~8 GB VRAM)
--quantize bitsandbytes-nf4

# List all configuration options
docker run ghcr.io/huggingface/text-generation-inference:3.3.5 --help

Production tip: Pin to a specific version tag (3.3.5, not latest) so your inference container doesn't change unexpectedly on restart. The 3.3.x patch line is backward-compatible.

Sources: TGI Quicktour — HuggingFace Docs · LLM Inference at Scale with TGI — HuggingFace Blog · TGI GitHub