From Model Card to Private HTTPS Endpoint in Minutes: HuggingFace Inference Endpoints With Autoscale-to-Zero

HuggingFace Inference Endpoints gives you a private, dedicated GPU API for any Hub model in minutes — HuggingFace manages TGI, the container, and the load balancer; you pick hardware, set min_replica=0 to pay nothing when idle, and query via the OpenAI-compatible API.

What you'll be able to do after this:

• Deploy any model from the HuggingFace Hub to a dedicated GPU instance (T4, A10G, A100) via four lines of Python — no Docker, no Kubernetes, no infrastructure management
• Configure autoscale-to-zero so the endpoint costs nothing when idle and wakes automatically within 20–30 seconds on the next request
• Query your endpoint with the standard OpenAI client, the huggingface_hub library, or any HTTP client using the same API shape as Fireworks or Together AI

---

HuggingFace Inference Endpoints fills the gap between the free Inference API (shared, rate-limited, no SLA) and full self-hosting with TGI or vLLM (you manage the container). HuggingFace manages TGI under the hood — you just pick a model, pick hardware, and pay per GPU-hour while the endpoint is running (zero when scaled to zero).

Step 1: Install and authenticate

pip install huggingface_hub

from huggingface_hub import login
login(token="hf_...")   # or set HF_TOKEN env var

Step 2: Create an endpoint

from huggingface_hub import HfApi

api = HfApi()

endpoint = api.create_inference_endpoint(
    name="llama-3b-endpoint",
    repository="meta-llama/Llama-3.2-3B-Instruct",  # any Hub model ID
    framework="pytorch",
    task="text-generation",
    accelerator="gpu",
    vendor="aws",
    region="us-east-1",
    instance_size="x4",      # T4 16GB ($0.60/hr); x8 = A10G 24GB ($1.30/hr)
    instance_type="protected",
    min_replica=0,           # scale to zero when idle — key for cost control
    max_replica=1,
    type="protected",        # "protected" = private; "public" = open URL
)

endpoint.wait()              # blocks until running (~2–5 min on first deploy)
print(endpoint.url)
# https://<id>.us-east-1.aws.endpoints.huggingface.cloud

Step 3: Query with the OpenAI client

Every Inference Endpoint is OpenAI-compatible (TGI serves /v1/chat/completions):

from openai import OpenAI
import os

client = OpenAI(
    base_url=f"{endpoint.url}/v1",
    api_key=os.environ["HF_TOKEN"],
)

response = client.chat.completions.create(
    model="tgi",                 # TGI ignores the model field; it serves whatever was deployed
    messages=[{"role": "user", "content": "What is gradient descent?"}],
    max_tokens=256,
)
print(response.choices[0].message.content)

Or use the native HuggingFace client:

from huggingface_hub import InferenceClient

client = InferenceClient(model=endpoint.url, token=os.environ["HF_TOKEN"])
result = client.text_generation("What is gradient descent?", max_new_tokens=256)
print(result)

Step 4: Scale to zero (or let autoscaling handle it)

The endpoint scales to zero automatically after ~15 minutes of no traffic. You can also trigger it manually:

endpoint.scale_to_zero()   # pauses immediately
# later:
endpoint.resume()           # wakes back up (~20–30s cold start)

Compare with other options

Option	Best for	Latency	Cost model
HF Free Inference API	Prototyping	Shared, variable	Free (rate-limited)
HF Inference Endpoints	Private prod, any model	Dedicated, consistent	Per GPU-hour
vLLM / TGI self-hosted	Maximum control	Dedicated	Your cloud bill
Together / Fireworks	High-throughput, multi-tenant	Dedicated, fast	Per token

Cold starts take ~20–30s from zero-replica state. If your use case can't tolerate that, set min_replica=1 (one replica always warm) — you pay the GPU-hour rate continuously but get sub-second response times.

Sources: Inference Endpoints documentation — HuggingFace · Getting Started with Inference Endpoints — HuggingFace Blog · Autoscaling — HuggingFace Docs