750 Tokens Per Second on Llama 70B: Get Started With Groq's LPU API in 10 Lines

TL;DR: Groq serves open-source LLMs on custom LPU chips at 700-900 tok/s — 20x faster than GPU inference. Free tier, OpenAI-compatible. Point your existing client at api.groq.com in two lines.

What you'll be able to do after this:

• Make your first Groq API call using either the native SDK or an OpenAI client with a base_url swap
• Understand why LPU inference is fundamentally different from GPU batching — and when that matters
• Drop Groq into latency-sensitive paths (streaming chat, real-time agents, voice pipelines) where time-to-first-token is perceptible

Why Groq is different from Fireworks or Together

Fireworks and Together run open models on GPU clusters and optimize through batching. Groq built custom silicon: the LPU (Language Processing Unit). LPUs use a static, fully pre-compiled execution schedule — no dynamic scheduling overhead at runtime. The result: time-to-first-token under 200ms and sustained throughput of 750-900 tokens/sec on Llama 3.3 70B, versus ~50-100 tok/s typical on GPU.

This matters most when:

• You're streaming and the user perceives the first token delay
• You're building voice pipelines (TTS latency + LLM latency must both be small)
• Your agent loop calls the LLM many times and total wall-clock matters
• You want to prototype fast before deciding whether to self-host

Setup: 3 minutes, no credit card

pip install groq
# Get a free key at console.groq.com → API Keys
export GROQ_API_KEY="gsk_..."

Native Groq SDK

from groq import Groq

client = Groq()   # reads GROQ_API_KEY automatically

completion = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain tensor parallelism in 3 sentences."}],
    temperature=0.6,
    max_tokens=512,
)
print(completion.choices[0].message.content)
# Typical TTFT: ~150ms; throughput: 750–900 tok/s

Already using OpenAI? Change two lines

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",  # ← change this
    api_key=os.environ.get("GROQ_API_KEY"),      # ← and this
)

# Everything else is identical
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "What is PagedAttention?"}],
)
print(response.choices[0].message.content)

What's available

The model catalog includes Llama 3.3 70B and 3.1 8B, Llama 4 Scout (750 tok/s) and Maverick, Qwen3 32B, GPT-OSS 20B (1,000 tok/s) and 120B, Mixtral, Gemma 3, Whisper (speech-to-text), and Orpheus (TTS). The free tier is generous: no credit card, 14,400 requests/day on most models.

Streaming

with client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Summarize the vLLM PagedAttention paper."}],
    stream=True,
) as stream:
    for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

At 750+ tok/s the stream feels instant — text arrives faster than a human reads it.

Sources: GroqDocs Quickstart · OpenAI Compatibility — GroqDocs · Groq LPU Inference Engine Tutorial — DataCamp