Your Code and My Code Don't Cost the Same: How BPE Tokenization Shapes Every LLM API Bill

Chris Harper

4 min read

Aug 6, 2026 · 12:04 UTC

Tutorial

LLM

HuggingFace

BPE tokenization is why code often costs 2x more than prose, why different models see different context limits for the same text, and why fine-tuning on padding-heavy data wastes compute -- it is the one foundational concept that makes everything else in AI engineering make sense.

What you'll be able to do after this:

Explain how BPE builds a vocabulary from scratch and tokenizes any string into integers
Use the HuggingFace tokenizers library to inspect and compare tokenization across any open model
Predict when your prompts will be expensive or hit context limits before you send them

What tokens actually are

An LLM never sees your text. It sees a sequence of integers -- token IDs -- drawn from a fixed vocabulary. "Hello, world!" might become [9906, 11, 1917, 0]. The model learns probability distributions over these integers. Everything about cost, context limits, and generation speed is downstream of how many integers your text becomes.

How BPE builds a vocabulary

BPE (Byte-Pair Encoding) starts with a vocabulary of individual bytes (256 entries). It then iteratively finds the most frequent adjacent pair of tokens in a training corpus and merges them into a new single token. Repeat ~50,000 times and you have a 50,257-token vocabulary (GPT-2's size). Common English words become single tokens ("the" -> token 0) while rare combinations split into multiple tokens.

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer

# Train a BPE tokenizer from scratch on your own corpus
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(vocab_size=5000, special_tokens=["[UNK]", "[CLS]", "[SEP]"])
tokenizer.train(["my_corpus.txt"], trainer)

output = tokenizer.encode("Hello, world! def calculate():")
print(output.tokens)
# ['Hello', ',', 'world', '!', 'def', 'cal', 'culate', '(', ')', ':']
print(len(output.tokens))  # 10 tokens for 30 characters

Why this changes your cost math

Code tokenizes more expensively than English prose. Special characters like (, [, {, -> rarely appear adjacent to alphanumeric chars in English, so they do not get merged into efficient tokens. A Python function of 100 characters may cost 50-80 tokens. An English paragraph of 100 characters might be 25-35 tokens.

Different model vocabularies (GPT-4's cl100k_base vs Llama 3's SentencePiece vs Claude's BPE variant) tokenize the same string to different lengths. Before choosing a model for a code-heavy workload, compare token counts:

import tiktoken  # pip install tiktoken

enc = tiktoken.get_encoding("cl100k_base")
text = "def factorial(n):
    return 1 if n <= 1 else n * factorial(n-1)"
print(f"GPT-4 token count: {len(enc.encode(text))}")  # ~22

from transformers import AutoTokenizer
llama_tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
print(f"Llama-3 token count: {len(llama_tok(text)['input_ids'])}")  # may differ

Practical rules for AI engineers

Budget context windows per-token, not per-character. Rule of thumb: 1 token ~ 4 English chars, but 1 token ~ 2-3 code chars.
Check vocabulary coverage before fine-tuning. If your domain has many OOV-ish terms, they split into many subword tokens and inflate sequence lengths.
Padding adds compute cost. During fine-tuning, sequences are padded to the batch max length -- shorter, more uniform sequences mean less wasted GPU time.
JSON and YAML structures are expensive. Structural punctuation ({, ", :, }) resists merging and inflates token counts fast.
Use count_tokens before sending large prompts. Anthropic's SDK has client.messages.count_tokens() -- call it before a large structured prompt to confirm fit and estimate cost.

Go deeper

The interactive chapter at HuggingFace LLM Course Chapter 6 walks through BPE step by step with a live Colab notebook you can run in your browser -- no GPU needed. Chapter 6.8 builds a complete tokenizer from scratch on a small corpus so you can watch the merge algorithm work in real time.