Tokens and context windows
Models never see characters or words — they see tokens, small chunks of text mapped to integer ids. Tokens determine what you pay, how fast a response arrives, and how much the model can read at once. The context window is the hard limit on tokens per request. This lesson shows how tokenization works, how to count tokens precisely, and how to keep every request inside its budget.
Definitions
The basic unit of text a language model processes. A token can be a whole common word (" answers"), part of a word ("ieval"), a single character, punctuation, or whitespace. Each token has an integer id.
The deterministic program that converts text into a list of token ids (encoding) and back (decoding). Every model family has its own tokenizer, so the same sentence can be a different number of tokens on different models.
The fixed set of all tokens a tokenizer knows. Modern vocabularies contain roughly 100,000–260,000 tokens; OpenAI's o200k_base has about 200,000.
The maximum number of tokens a model can process in one request — the prompt (system message, history, documents, tool definitions) plus the generated output.
Why tokens, not characters or words?
| Approach | Problem |
|---|---|
| One token per character | Sequences become very long, so the model is slow and sees less text at once. |
| One token per word | The vocabulary explodes (every name, typo, and inflection), and unknown words cannot be represented at all. |
| Subword tokens (what LLMs use) | Common words are one token, rare words are built from a few pieces, and any text — including typos, code, and emoji — can be represented. |
How BPE builds a vocabulary
The most common tokenizer training algorithm. It starts with individual characters (or bytes) and repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches the target size. Frequent words end up as single tokens; rare words stay split.
Here is a miniature BPE training run on a tiny corpus. Watch frequent pairs merge into larger tokens:
from collections import Counter
words = ["low", "lower", "lowest", "newest", "widest", "low"]
seqs = [list(w) for w in words] # start from characters
def most_common_pair(seqs):
pairs = Counter()
for s in seqs:
for a, b in zip(s, s[1:]):
pairs[(a, b)] += 1
return pairs.most_common(1)[0][0]
def merge(seqs, pair):
merged = []
for s in seqs:
out, i = [], 0
while i < len(s):
if i < len(s) - 1 and (s[i], s[i + 1]) == pair:
out.append(s[i] + s[i + 1])
i += 2
else:
out.append(s[i])
i += 1
merged.append(out)
return merged
for step in range(1, 5):
pair = most_common_pair(seqs)
seqs = merge(seqs, pair)
print(f"merge {step}: {pair[0]!r} + {pair[1]!r}")
print(seqs)After only four merges, low and est have become reusable tokens. A real tokenizer runs hundreds of thousands of merges over terabytes of text.
Counting real tokens with tiktoken
tiktoken is OpenAI's fast tokenizer library. The o200k_base encoding is used by GPT-4o-era and newer OpenAI models. Install it with pip install tiktoken.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
text = "Retrieval-augmented generation grounds answers in your docs."
ids = enc.encode(text)
print(len(ids), "tokens")
print(ids)
print([enc.decode([i]) for i in ids])
print(enc.decode(ids) == text)Notice three things. Common words like " answers" are single tokens. Rarer words like Retrieval are split into pieces. And the leading space is part of the token — " generation" and "generation" are different tokens with different ids.
Different text, different token density
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
samples = {
"English": "Hello world",
"Rare word": "unbelievably",
"Brand name": "ChatGPT is great!",
"Hindi": "नमस्ते दुनिया",
"Code": "def add(a, b):\n return a + b",
}
for label, text in samples.items():
pieces = [enc.decode([i]) for i in enc.encode(text)]
print(f"{label:<11} {len(pieces):>2} tokens {pieces}")English prose averages roughly 4 characters or ¾ of a word per token. Code, numbers, and many non-English languages use noticeably more tokens for the same meaning — which means they cost more and fill the context window faster.
| Rule of thumb (English, OpenAI tokenizers) | Approximate value |
|---|---|
| 1 token | ≈ 4 characters ≈ ¾ of a word |
| 100 tokens | ≈ 75 words ≈ one short paragraph |
| 1,000 tokens | ≈ 750 words ≈ 1.5 pages |
| 100,000 tokens | ≈ 75,000 words ≈ a full-length novel |
Rules of thumb are fine for planning, never for billing or hard limits. Count with the provider's tokenizer, or read the exact usage numbers returned by the API. Other providers (Anthropic, Google, Meta) use different tokenizers and offer their own token-counting endpoints or libraries.
The context window
Everything the model sees in one request must fit in its context window, and the reply must fit too:
┌──────────────────────── context window (e.g. 128,000 tokens) ────────────────────────┐
│ system prompt │ tool definitions │ retrieved docs │ chat history │ user msg │ OUTPUT │
└──────────────────────────────────────────────────────────────────────────────────────┘
input tokens ─────────────────────────────────────────────────────────────▶ ◀ output ▶- Context windows range from about 8,000 tokens on small local models to 1,000,000+ tokens on some frontier models. Check the model card for the exact figure.
- Models also have a separate, smaller maximum output length (often 4,000–128,000 tokens). You cap it per request with
max_tokens(ormax_completion_tokens/max_output_tokens, depending on the API). - If input plus requested output exceeds the window, the API returns an error such as
context_length_exceeded— or, in some chat products, silently drops the oldest messages.
A well-documented effect where models recall information at the beginning and end of a long context better than information buried in the middle. A large window does not guarantee the model will use everything in it equally well.
Why not just use the biggest window?
| Cost of a long prompt | Explanation |
|---|---|
| Money | You pay for every input token on every request — a 100k-token prompt sent 1,000 times is 100 million tokens. |
| Latency | The model must process every input token before producing the first output token. |
| Quality | Irrelevant text distracts the model and dilutes attention on what matters. |
That is why retrieval-augmented generation exists: instead of pasting your whole knowledge base into every request, you retrieve the handful of chunks that matter.
Budgeting tokens in code
A chat application must keep its history inside a token budget. The standard strategy: always keep the system prompt, then add messages from newest to oldest until the budget is used, reserving room for the reply.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
def count_tokens(message: dict) -> int:
# ~4 tokens of per-message overhead for role and formatting
return len(enc.encode(message["content"])) + 4
def fit_history(messages, context_limit=60, reserve_for_output=20):
system, history = messages[0], messages[1:]
budget = context_limit - reserve_for_output - count_tokens(system)
kept = []
for msg in reversed(history): # newest first
cost = count_tokens(msg)
if cost > budget:
break
kept.append(msg)
budget -= cost
return [system] + list(reversed(kept))
messages = [
{"role": "system", "content": "You are a concise AI tutor."},
{"role": "user", "content": "What is a token?"},
{"role": "assistant", "content": "A token is a chunk of text, often part of a word."},
{"role": "user", "content": "And what is a context window?"},
{"role": "assistant", "content": "The maximum tokens the model can handle in one request."},
{"role": "user", "content": "How do I stay under it?"},
]
trimmed = fit_history(messages)
print(len(messages), "->", len(trimmed), "messages")
for m in trimmed:
print(f"{m['role']:>9}: {m['content']}")The oldest exchanges were dropped while the system prompt and the most recent turns survived. Production apps often go further and summarise dropped messages into a short memory note instead of discarding them entirely.
The exact per-message overhead depends on the model and API format, and tool definitions and images also consume tokens. Leave a safety margin, and use the usage.prompt_tokens value returned by the API to calibrate your estimates.
Tokens explain many “weird” behaviours
| Behaviour | Token explanation |
|---|---|
| Miscounting letters in a word (“how many r's in strawberry?”) | The model sees a few tokens like str + awberry, not individual letters. |
| Weak at reversing strings or exact character edits | Same reason — characters are hidden inside tokens. |
| Arithmetic errors on long numbers | Numbers are split into chunks of digits that do not align with place value. |
| Higher cost for some languages | Scripts that are less common in training data are split into more tokens. |
| Output stops mid-sentence | The reply hit max_tokens; finish_reason will be length. |
- Use
tiktokento count tokens for the same paragraph in English and in another language you know. Compute the ratio. - Tokenize a JSON object with and without
indent=2. How many tokens does pretty-printing cost? - Change
fit_historyso that, instead of dropping old messages, it replaces them with one assistant message:"(Earlier conversation omitted.)".
- Models read and write tokens — subword chunks with integer ids — produced by a model-specific tokenizer.
- BPE builds vocabularies by repeatedly merging the most frequent adjacent pairs.
- In English, 1 token ≈ 4 characters ≈ ¾ of a word; code and many languages use more.
- The context window covers input and output; long prompts cost more, run slower, and can lose detail in the middle.
- Count tokens with
tiktokenor API usage data, and trim or summarise history to stay within budget.