Your First Local LLM: Install Ollama and Run Open-Weight Models in Five Minutes

TL;DR: Ollama gives you a one-command install, a library of 100+ models, an OpenAI-compatible REST API, and Python/JS SDKs — everything you need to run LLMs locally, free, offline.

What you'll be able to do after this:

• Install Ollama and pull any open-weight model — Llama, Gemma, Qwen, Phi — with a single command
• Query it via curl or Python exactly like you'd call a hosted API, with a drop-in-compatible response format
• Create a Modelfile to bake in a system prompt, temperature, and context length — your own named local model in two commands

Install Ollama (30 seconds)

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows (PowerShell)
irm https://ollama.com/install.ps1 | iex

# Docker
docker run -d -p 11434:11434 ollama/ollama

Pull a model

ollama pull llama3.2          # Meta 3B chat model — 2.0 GB
ollama pull qwen3:4b          # Alibaba 4B, strong on coding — 2.5 GB
ollama pull phi4-mini         # Microsoft 3.8B, very fast — 2.5 GB

Browse 100+ models at ollama.com/library.

Run interactively

ollama run llama3.2
>>> What is a vector embedding?

Type /bye to exit.

Query via REST API

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Explain embeddings in one paragraph"}],
  "stream": false
}'

The response matches the OpenAI chat/completions format — swap the base URL in any OpenAI client and it works without code changes.

Python integration

pip install ollama

from ollama import chat

resp = chat(model='llama3.2', messages=[
    {'role': 'user', 'content': 'Explain embeddings in one paragraph'}
])
print(resp.message.content)

Build a named local model with a Modelfile

Create a file called Modelfile:

FROM llama3.2
SYSTEM "You are a terse code reviewer. Call out real bugs only, skip style."
PARAMETER temperature 0.3
PARAMETER num_ctx 8192

Then:

ollama create code-reviewer -f ./Modelfile
ollama run code-reviewer

FROM picks the base model. SYSTEM bakes in a system prompt that persists across every conversation. PARAMETER sets inference defaults. The result is a named local model you reference anywhere Ollama runs — your CI server, your home lab, your laptop on a flight.

This pattern — base model + Modelfile — is the local equivalent of a fine-tune without the training step. You're fixing the context, not the weights. Enough for most internal tooling where you want consistent behavior without an external API dependency.

Sources: Ollama official docs · Ollama GitHub README · Modelfile reference · Ollama Tutorial for Beginners — YouTube