OpenAI's Custom Inference Chip Outperforms NVIDIA Blackwell on Throughput per Watt — and GPT-5.6 Luna Falls 80%

Two developments this week point in the same direction: inference is getting faster and cheaper, and the gap between "powerful" and "affordable" is narrowing.

Jalapeño results (Aug 25): OpenAI published first measured results from its custom inference chip, developed with Broadcom. On SemiAnalysis' InferenceX benchmark: 1.5–1.9× more AI work per watt at peak throughput, 1.7–3.6× lower end-to-end latency, and 2.1–4.1× better performance on interactive workloads — all versus current NVIDIA Blackwell hardware. The architecture keeps KV cache and model state local to cut data-movement overhead, which is the dominant inference bottleneck most hardware doesn't specifically address.

Deployment starts in "very small volumes" at the end of 2026, broader rollout in 2027. No API pricing or response-time changes are announced yet — these are internal infrastructure numbers.

GPT-5.6 Luna pricing: OpenAI cut Luna 80% this week to $0.20 input / $1.20 output per million tokens. Terra held at $2.50 / $15.

Why it matters: If your agent workflow routes everything through a single model tier, the infrastructure trend supports designing for model routing now — so switching to a cheaper tier for sub-tasks is a config change, not an architecture change. LiteLLM's router is the standard wire for that without coupling to a specific provider.

Sources: Jalapeño first results — OpenAI · OpenAI + Broadcom chip announcement · TechCrunch analysis · GPT pricing updates — llm-stats.com