Your GGUF Is Already a Server: Run llama-server for Local OpenAI-Compatible Inference Without the Cloud

TL;DR: llama-server -m model.gguf --port 8080 gives you a local /v1/chat/completions endpoint — change one base_url in your OpenAI SDK and inference stays local, no GPU or cloud required.

The Jul 8 GGUF formats post covered which quantization to pick. This one covers what to actually do with a .gguf file: serve it. llama.cpp is the C/C++ engine that powers Ollama, LM Studio, and Jan under the hood. Its built-in llama-server binary exposes a fully OpenAI-compatible REST API, so any code that calls /v1/chat/completions works locally by changing exactly one parameter.

What you'll be able to do after this:

• Download a quantized GGUF from HuggingFace Hub and serve it locally in under 5 minutes
• Query it with the OpenAI Python SDK by changing one base_url — zero other code changes
• Offload layers to GPU with -ngl 99 for 3–5× faster generation when hardware is available

Step 1: Install

macOS (no compile):

brew install llama.cpp

Linux/Windows (pip, no compile):

pip install llama-cpp-python[server]
# run as: python -m llama_cpp.server --model model.gguf --port 8080

From source with CUDA:

git clone --depth 1 https://github.com/ggml-org/llama.cpp
cmake -B build -DGGML_CUDA=ON    # omit for CPU-only
cmake --build build --config Release -j$(nproc)
# binary: ./build/bin/llama-server

Step 2: Download a model

pip install huggingface-hub
huggingface-cli download bartowski/Llama-3.2-3B-Instruct-GGUF   --include "Llama-3.2-3B-Instruct-Q4_K_M.gguf"   --local-dir ./models

Q4_K_M is the safe default: 4-bit quantization with k-means correction, ~25% the size of BF16 with less than 1% quality loss. A 3B model weighs about 1.8 GB and runs at 15–25 tok/s on a modern laptop CPU.

Step 3: Start the server

llama-server   -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf   -c 4096   --port 8080   -ngl 0        # set -ngl 99 to offload all layers to GPU

When you see llama server listening at http://127.0.0.1:8080, it is ready.

Step 4: Query it — no code changes needed

curl:

curl http://localhost:8080/v1/chat/completions   -H "Content-Type: application/json"   -d '{"model":"local","messages":[{"role":"user","content":"What is LoRA?"}]}'

OpenAI Python SDK — one line changed:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
)
print(resp.choices[0].message.content)

Key flags

Flag	Effect
-c 4096	context window in tokens
-ngl 99	GPU layers to offload (0 = CPU-only)
-np 4	parallel request slots
--threads 8	CPU threads to use
--chat-template llama3	override auto-detected chat template

llama-server vs Ollama

Ollama is simpler for day-to-day use (auto model download, multi-model hot-swap). Use llama-server directly when you need exact GGUF control: a custom fine-tune, specific flags unavailable in Ollama, or when you are embedding llama.cpp into your own infrastructure and want the raw HTTP interface without Ollama's daemon layer.

Sources: llama-server README — ggml-org/llama.cpp · llama.cpp Quickstart — glukhov.org · Run llama.cpp Server: OpenAI API from GGUF — Markaicode · Unsloth llama-server deployment docs