Next-token prediction

An LLM generates every answer one token at a time. At each step it scores every token in its vocabulary, turns those scores into probabilities, picks one token, appends it, and runs again. This lesson walks through that loop with runnable code, explains how the same objective is used in training, and shows how it accounts for both the power and the quirks of language models.

Last updated: 01 Nov, 2026

Definitions

One step: from logits to probabilities

Suppose the prompt is “The capital of France is” and, for simplicity, the vocabulary has only five tokens. The model outputs one logit per token; softmax turns them into a probability distribution.

import math

vocab = [" Paris", " Lyon", " a", " the", " beautiful"]
logits = [6.2, 2.1, 3.0, 2.4, 3.6]


def softmax(scores):
    m = max(scores)                       # subtract the max for numerical stability
    exps = [math.exp(s - m) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]


probs = softmax(logits)
for token, p in sorted(zip(vocab, probs), key=lambda x: -x[1]):
    print(f"{token!r:<13} {p:6.1%}  {'█' * round(p * 40)}")
print("sum:", round(sum(probs), 6))

The model is fairly confident in " Paris", but it assigns real probability to alternatives like " beautiful" (as in “…is a beautiful city”). Which one appears depends on the decoding strategy, covered in depth in the Temperature and sampling lesson.

The generation loop

1. Tokenize the prompt into ids.
2. Run the model on all ids so far to get logits for the next position.
3. Convert logits to probabilities and choose one token.
4. Append the chosen token to the sequence (and stream it to the user).
5. Stop if a stop condition is met; otherwise go back to step 2.

Here is that loop running on a toy model. The “model” is a lookup table of next-token probabilities, but the loop is structurally identical to what runs inside every LLM server.

TOY_MODEL = {
    "<start>":   {"LLMs": 0.7, "Models": 0.3},
    "LLMs":      {"predict": 0.8, "generate": 0.2},
    "Models":    {"predict": 0.6, "learn": 0.4},
    "predict":   {"the": 0.6, "tokens": 0.4},
    "generate":  {"text": 1.0},
    "learn":     {"patterns": 1.0},
    "the":       {"next": 0.9, "answer": 0.1},
    "next":      {"token": 1.0},
    "token":     {".": 0.7, "again": 0.3},
    "tokens":    {".": 1.0},
    "text":      {".": 1.0},
    "patterns":  {".": 1.0},
    "answer":    {".": 1.0},
    "again":     {".": 1.0},
    ".":         {"<eos>": 1.0},
}


def generate_greedy(max_tokens=10):
    tokens = ["<start>"]
    for step in range(1, max_tokens + 1):
        dist = TOY_MODEL[tokens[-1]]
        next_token = max(dist, key=dist.get)      # greedy: most likely token
        print(f"step {step}: {dist} -> {next_token!r}")
        if next_token == "<eos>":
            return tokens[1:], "stop"
        tokens.append(next_token)
    return tokens[1:], "length"


output, finish_reason = generate_greedy()
print(" ".join(output), "| finish_reason:", finish_reason)

A real model differs in three ways: its “lookup” is a neural network that considers the entire sequence, not just the last token; its vocabulary has ~200,000 entries; and it usually samples rather than always taking the top choice.

Stop conditions and finish_reason

Generation ends for one of a few reasons, which APIs report in a field such as finish_reason (OpenAI) or stop_reason (Anthropic):

How the same objective trains the model

import math

# probability the model assigned to the true next token " Paris"
for p in [0.9, 0.5, 0.1, 0.01]:
    print(f"p(correct) = {p:<5} -> loss = {-math.log(p):.2f}")

To predict the next token of a physics textbook, a legal contract, or a Python file well, the model must internally learn grammar, facts, conventions, and patterns of reasoning. That is why such a simple objective produces such broad abilities.

Prefill and decode: why output is slower than input

Practical consequence: shortening the answer speeds up a request far more than shortening the prompt by the same number of tokens.

What next-token prediction explains