One Command Launches Any Embedding Model as an OpenAI-Compatible API: Getting Started with Infinity

Self-host any sentence-transformer as an OpenAI-compatible embedding API in one command — Infinity adds dynamic batching and flash attention, your app changes nothing.

If your RAG pipeline calls openai.embeddings.create(), swapping to a self-hosted model costs you nothing in application code. Infinity speaks the OpenAI Embeddings wire format exactly — the only change is base_url.

What you'll be able to do after this:

• Replace text-embedding-ada-002 in any RAG pipeline with a free BAAI/bge or nomic-embed model — no code changes in your app
• Serve embedding and reranking models together on a single port
• Run offline on CPU, Apple MPS, or CUDA for near-zero cost per batch

Three commands to launch

Pick whichever fits your environment:

# Option 1: pip — fastest to try
pip install "infinity-emb[all]"
infinity_emb v2 --model-id BAAI/bge-en-icl --port 7997

# Option 2: Docker with a CUDA GPU
docker run --gpus all -p 7997:7997 \
  michaelf34/infinity:latest v2 \
  --model-id BAAI/bge-en-icl --port 7997

# Option 3: CPU / Apple Silicon — drop the --gpus flag
docker run -p 7997:7997 \
  michaelf34/infinity:latest v2 \
  --model-id nomic-ai/nomic-embed-text-v1.5 --port 7997

The model downloads from HuggingFace Hub on first run and caches in ~/.cache/huggingface/. Subsequent starts are instant.

Zero code changes in your existing app

from openai import OpenAI

client = OpenAI(
    api_key="local",                    # Infinity ignores this; any string works
    base_url="http://localhost:7997/v1"
)

response = client.embeddings.create(
    model="BAAI/bge-en-icl",           # must match --model-id above
    input=["How does attention work?", "What is a transformer?"]
)
print(len(response.data[0].embedding))  # 4096 — same structure as the OpenAI response

LangChain's OpenAIEmbeddings, LlamaIndex, and LiteLLM all work with this endpoint unchanged — pass openai_api_base="http://localhost:7997/v1" in whichever client you use.

Serve an embedder and a reranker on the same port

Add a second --model-id to serve both at once:

infinity_emb v2 \
  --model-id BAAI/bge-en-icl \
  --model-id BAAI/bge-reranker-v2-m3 \
  --port 7997

Embeddings arrive at /v1/embeddings; reranking at /v1/rerank — same server, same port, no extra process.

Why BAAI/bge-en-icl?

It sits above text-embedding-3-large on the MTEB leaderboard for most retrieval tasks, and it's free. With Infinity's dynamic batching, a 10k-token batch on a CPU-only t3.medium finishes in under two seconds. On a T4 GPU, it's milliseconds. At zero cost per call, the math is obvious once your evaluation dataset grows past a few thousand entries.

Infinity is MIT-licensed and actively maintained; it powers the embedding API behind Gradient.ai and several other hosted providers.

Sources: GitHub — michaelfeil/infinity · Infinity docs · MTEB Leaderboard — HuggingFace · Infinity on AMD — HuggingFace Blog