Zero to Serverless GPU in 30 Lines: Deploy Any HuggingFace Model with Modal + vLLM

Chris Harper

3 min read

Aug 14, 2026 · 12:12 UTC

Tutorial

Self-Hosting

HuggingFace

Modal is Python-native serverless GPU: annotate a function, name a model, and modal deploy gives you an OpenAI-compatible inference endpoint on a real H100 that scales to zero when idle.

What you'll be able to do after this:

Deploy a vLLM-backed LLM endpoint to a real GPU with ~30 lines of Python and no Dockerfile, Kubernetes, ECR push, or cluster management
Get an OpenAI-compatible /v1/chat/completions endpoint that scales to zero when idle — you pay only for inference time (Modal bills per second of container uptime)
Cache model weights in a Modal Volume so cold starts drop from several minutes to 10–30 seconds for a 7–8B model

Modal fills a gap between "local Ollama" (great for your machine, not shareable) and "Together/OpenRouter" (someone else's model). With Modal you control the model, the GPU tier, and the serving configuration — but you still write plain Python and let Modal handle provisioning.

Install and authenticate

pip install modal
modal setup   # opens browser to authenticate with your modal.com account

Modal gives you $30/month of free credits — enough for several hours of H100 time per month for experiments.

Write the inference server (inference.py)

import modal
import subprocess

app = modal.App("vllm-server")

MODEL = "mistralai/Mistral-7B-Instruct-v0.3"   # swap any vLLM-supported HF model

vllm_image = (
    modal.Image.from_registry("nvidia/cuda:12.8.0-devel-ubuntu22.04", add_python="3.12")
    .uv_pip_install(
        "vllm==0.10.2",
        "huggingface_hub[hf_transfer]==0.35.0",
    )
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})   # faster HF downloads
)

weights = modal.Volume.from_name("llm-weights", create_if_missing=True)  # persists across deploys

@app.cls(
    image=vllm_image,
    gpu="H100",                    # or "A10G" for smaller/cheaper experiments
    volumes={"/models": weights},
    concurrency_limit=2,           # scale up replicas beyond this threshold
    timeout=3600,
)
@modal.web_server(8000)            # expose vLLM's HTTP server as a Modal web endpoint
class VLLMServer:
    @modal.enter()
    def run_server(self):
        subprocess.Popen([
            "vllm", "serve", MODEL,
            "--download-dir", "/models",
            "--host", "0.0.0.0",
            "--port", "8000",
        ])

Deploy and call it

modal deploy inference.py
# → Deployed! URL: https://your-workspace--vllm-server.modal.run

from openai import OpenAI

client = OpenAI(
    base_url="https://your-workspace--vllm-server.modal.run/v1",
    api_key="not-needed",
)
response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "What is PagedAttention and why does it matter?"}],
)
print(response.choices[0].message.content)

The endpoint is identical to the OpenAI API — any code that calls openai.chat.completions.create works without modification, just swap the base_url.

What happens when you hit the endpoint

If no container is running, Modal cold-starts one: CUDA image pulls (cached after the first deploy), vLLM process starts, model weights load from the Volume.
vLLM serves the request using PagedAttention for efficient KV-cache management.
After idle timeout, the container shuts down — zero cost while it's not running.

The weight Volume is the key optimization: model files are downloaded once on the first request and stored in Modal's networked storage, so subsequent cold starts skip the HuggingFace download and load directly from the Volume (10–30 seconds for a 7B model vs. 2–10 minutes for a fresh download).

Swap models with one line change

Change MODEL to any model vLLM supports — meta-llama/Llama-3.1-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, google/gemma-2-9b-it — and redeploy. For gated models (Llama), add a HuggingFace token as a Modal Secret and pass it to the container.