Run Any Open-Weight Model Locally in Two Commands: Ollama and Its OpenAI-Compatible API

What you'll be able to do after this: pull and run any open-weight model locally (Qwen, Llama, Gemma, DeepSeek, Mistral) and call it from Python using the exact same code you'd use for the OpenAI API — one URL change, no cloud cost, no data leaving your machine.

• Pull a model in one command. ollama pull qwen3:8b downloads the model, selects the right quantization for your GPU or RAM, and makes it ready to serve.
• Drop in as an OpenAI replacement. Ollama starts a server at localhost:11434/v1 with an OpenAI-style /chat/completions endpoint — your existing openai.ChatCompletion code works unchanged.
• Function calling and structured JSON output are stable. Tool-call support arrived in Ollama 0.3; structured format output is solid in 0.32.x for models that support it.

Walk-through:

# Install (macOS, Linux, Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model — starts the local server automatically on first run
ollama pull qwen3:8b        # ~5 GB at Q4_K_M; fits 8 GB VRAM
ollama pull llama4:8b-q4    # Llama 4 8B quantized
ollama pull gemma4:4b       # Gemma 4 4B (text + audio)

from openai import OpenAI

client = OpenAI(
    api_key="ollama",                     # any non-empty string
    base_url="http://localhost:11434/v1"
)

response = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Explain vector search in two sentences."}]
)
print(response.choices[0].message.content)

ollama list shows downloaded models. ollama rm <name> removes one. ollama ps shows what is currently loaded and using GPU memory.

Where it breaks: a 4-bit quantized 8B model is meaningfully weaker than a frontier model — use it for classification, format conversion, local RAG, and privacy-sensitive tasks, not for complex reasoning or production code you won't review. Ollama's server processes requests serially; concurrent requests queue, so high-throughput production serving belongs on vLLM or SGLang instead. Also: models with a :cloud suffix in the Ollama library run on Ollama's own servers, not yours — check before assuming local.

Resources: Ollama Tutorial for Beginners (2026) — YouTube · Ollama official site — ollama.com · Ollama OpenAI-Compatible API: Setup Steps and Limits — TokenMix

Sources: What is Ollama: Run AI Models Locally — Thunder Compute · Local LLM Inference 2026: Ollama, Python, and the Open Model Ecosystem — Programming Helper Tech