GPU in a Decorator: Run Any Open-Weight LLM on a Serverless H100 With Modal

TL;DR: Add @app.function(gpu="h100") to any Python function and Modal runs it on a cloud GPU — no cluster, no idle costs, sub-5-second cold starts, scales to zero between requests.

What you'll be able to do after this:

• Deploy an open-weight LLM (Llama, Qwen, Mistral) to a GPU-backed serverless endpoint in under 10 minutes, with no infrastructure to provision or maintain
• Use Modal Volumes to cache model weights so cold starts don't re-download multi-gigabyte checkpoints on every scale-up
• Ship a production-grade OpenAI-compatible inference endpoint using vLLM on Modal with modal deploy, billed per second of GPU time with zero idle cost

---

Why Modal exists

After you fine-tune a model or want to self-host an open-weight LLM, the deployment question hits: where does it run? The two common answers are managed API providers (OpenRouter, Fireworks, Together — fast but you're sharing infrastructure and someone else controls pricing) or a self-managed vLLM server (full control but you pay for the GPU whether traffic is flowing or not, and you handle cold starts, capacity, and restarts yourself).

Modal is a third option: serverless GPU functions. You write Python, decorate it with Modal annotations, and the cloud handles container builds, GPU allocation, sub-5-second cold starts, autoscaling, and teardown when idle. You pay per second of compute — an H100 at ~$0.001/sec, an A10G at ~$0.0003/sec. An idle endpoint costs nothing.

The minimal example

Three commands to get started:

pip install modal
modal setup        # opens browser, authenticates with your account
modal run inference.py

inference.py:

import modal

app = modal.App("llm-inference")

# Define the container image — installs dependencies once, cached
image = modal.Image.debian_slim().uv_pip_install("transformers[torch]")

@app.function(gpu="h100", image=image)
def generate(prompt: str) -> str:
    from transformers import pipeline
    pipe = pipeline(
        "text-generation",
        model="Qwen/Qwen3-1.7B",
        device_map="cuda",
        max_new_tokens=512,
    )
    result = pipe([{"role": "user", "content": prompt}])
    return result[0]["generated_text"][-1]["content"]

@app.local_entrypoint()
def main():
    print(generate.remote("Explain attention mechanisms in one paragraph."))

modal run inference.py sends the function to Modal, allocates an H100, and returns the output. modal deploy inference.py makes it a persistent endpoint.

Production: vLLM + Modal Volume

For an OpenAI-compatible endpoint that handles concurrent requests, Modal's vLLM example is the reference implementation. The key additions over the minimal example:

Model weight caching with modal.Volume — without caching, every cold start re-downloads the model (multi-gigabyte for anything useful). A Volume persists the weights on Modal's infrastructure:

model_cache = modal.Volume.from_name("llm-model-cache", create_if_missing=True)

vllm_image = (
    modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12")
    .uv_pip_install("vllm==0.21.0")
)

@app.function(gpu="A100", image=vllm_image, volumes={"/cache": model_cache})
@modal.web_server(8000)
def serve():
    import subprocess
    subprocess.Popen([
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "meta-llama/Llama-3.1-8B-Instruct",
        "--download-dir", "/cache",
    ])

modal deploy vllm_inference.py publishes a public URL at https://your-org--llm-inference-serve.modal.run with a /v1/chat/completions endpoint. Use it with the openai Python client by setting base_url.

Cost model

GPU	Cost/sec	Cost per 10-sec call
H100	~$0.001	~$0.01
A100 (40GB)	~$0.0006	~$0.006
A10G	~$0.0003	~$0.003

Idle endpoints (no requests in flight) cost nothing. Compare to a dedicated A10G instance running 24/7: ~$26/day whether you use it or not.

Sources: Modal quickstart · vLLM on Modal example · How Modal achieves truly serverless GPUs