Temperature and sampling
At every step, an LLM has a probability for every possible next token. Sampling settings decide how it turns those probabilities into one choice. Temperature reshapes the distribution, top-k and top-p cut off unlikely tokens, and penalties discourage repetition. This lesson shows the maths in plain language, simulates each setting in code, and gives you defaults that work for real tasks.
Definitions
Choosing the next token by a weighted random draw from the model's probability distribution, instead of always taking the most likely token. Sampling makes output varied and natural; it also makes it non-deterministic.
Always choosing the single most probable next token. It is predictable but can produce repetitive, flat text, and it is what temperature 0 approximates.
A positive number that divides the logits before softmax: p = softmax(logits / T). T < 1 sharpens the distribution (the favourite becomes even more likely); T > 1 flattens it (unlikely tokens get more chances); T → 0 becomes greedy.
Temperature, visualised
We reuse the five-token example from the previous lesson and apply three temperatures to the same logits:
import math
vocab = [" Paris", " beautiful", " a", " the", " Lyon"]
logits = [6.2, 3.6, 3.0, 2.4, 2.1]
def softmax_t(scores, temperature):
scaled = [s / temperature for s in scores]
m = max(scaled)
exps = [math.exp(s - m) for s in scaled]
total = sum(exps)
return [e / total for e in exps]
for t in [0.2, 1.0, 2.0]:
probs = softmax_t(logits, t)
row = " ".join(f"{tok.strip()}={p:.0%}" for tok, p in zip(vocab, probs))
print(f"T={t:<4} {row}")- At T = 0.2 the model is almost certain to say Paris — ideal for factual answers and extraction.
- At T = 1.0 you get the model's natural distribution.
- At T = 2.0 alternatives become competitive — more creative, but also more likely to drift into nonsense over a long answer, because one odd token changes the context for every token after it.
Seeing randomness in action
Let us draw 1,000 samples at each temperature and count how often each token is chosen. A fixed random seed makes the experiment reproducible.
import math
import random
from collections import Counter
vocab = [" Paris", " beautiful", " a", " the", " Lyon"]
logits = [6.2, 3.6, 3.0, 2.4, 2.1]
def softmax_t(scores, temperature):
scaled = [s / temperature for s in scores]
m = max(scaled)
exps = [math.exp(s - m) for s in scaled]
return [e / sum(exps) for e in exps]
rng = random.Random(42)
for t in [0.2, 1.0, 2.0]:
probs = softmax_t(logits, t)
draws = rng.choices(vocab, weights=probs, k=1000)
counts = Counter(tok.strip() for tok in draws)
print(f"T={t:<4}", dict(counts.most_common()))Top-k sampling
Keep only the k most probable tokens, set every other token's probability to zero, renormalise, and sample from what is left. It removes the long tail of very unlikely tokens.
Top-p (nucleus) sampling
Keep the smallest set of most-probable tokens whose probabilities add up to at least p (for example 0.9), then renormalise and sample. Unlike top-k, the number of candidates adapts: when the model is confident, very few tokens survive; when it is unsure, many do.
def top_k(probs: dict[str, float], k: int) -> dict[str, float]:
kept = dict(sorted(probs.items(), key=lambda x: -x[1])[:k])
total = sum(kept.values())
return {t: round(p / total, 3) for t, p in kept.items()}
def top_p(probs: dict[str, float], p: float) -> dict[str, float]:
kept, cumulative = {}, 0.0
for token, prob in sorted(probs.items(), key=lambda x: -x[1]):
kept[token] = prob
cumulative += prob
if cumulative >= p:
break
total = sum(kept.values())
return {t: round(q / total, 3) for t, q in kept.items()}
confident = {"Paris": 0.86, "beautiful": 0.06, "a": 0.04, "the": 0.03, "Lyon": 0.01}
uncertain = {"blue": 0.30, "grey": 0.25, "clear": 0.20, "dark": 0.15, "green": 0.10}
print("top-k=2 confident:", top_k(confident, 2))
print("top-p=.9 confident:", top_p(confident, 0.9))
print("top-p=.9 uncertain:", top_p(uncertain, 0.9))With a confident distribution, top-p 0.9 keeps just two tokens. With an uncertain one (“The sky is…”), it keeps four. That adaptivity is why top-p is more common than top-k in hosted APIs.
Providers generally recommend adjusting temperature or top-p, not both. Changing both at once makes the effect hard to reason about. Leave top_p at its default of 1 and tune temperature, unless you have a specific reason.
Penalties: fighting repetition
| Setting | What it does | Typical range |
|---|---|---|
frequency_penalty | Lowers a token's logit in proportion to how many times it has already appeared. Reduces repeated words and phrases. | 0 to 1 (API allows −2 to 2) |
presence_penalty | Lowers a token's logit once if it has appeared at all. Encourages new topics. | 0 to 1 (API allows −2 to 2) |
repetition_penalty | Multiplicative penalty used by Hugging Face and many local runtimes. | 1.0 (off) to 1.3 |
Leave penalties at 0 for most tasks. They help with long creative generations that loop; they hurt tasks where repetition is correct, such as code or JSON with repeated keys.
Determinism and seeds
Temperature 0 makes outputs much more consistent but not guaranteed identical. Tiny numerical differences in GPU computation, batching with other users' requests, and silent model updates can still change a token, after which the rest of the answer diverges. Some APIs accept a seed parameter that improves reproducibility on a best-effort basis.
For dependable behaviour in production:
- Pin a dated model version (for example
gpt-4.1-mini-2025-04-14) rather than an alias that moves. - Use low temperature plus structured outputs (a JSON schema) for machine-readable results.
- Validate outputs and retry on failure instead of assuming identical text.
- Evaluate on a test set — not by eyeballing a single run.
Many reasoning models (for example OpenAI's o-series and GPT-5-style reasoning modes) ignore or reject temperature and top_p, and instead offer a reasoning_effort setting. Check each model's documentation before tuning sampling parameters.
Recommended settings by task
| Task | Temperature | Why |
|---|---|---|
| Data extraction, classification, JSON output | 0 – 0.2 | You want the single most likely, consistent answer. |
| RAG question answering | 0 – 0.3 | Stay faithful to the retrieved documents. |
| Code generation | 0 – 0.3 | Correctness beats creativity. |
| General chat assistant | 0.5 – 0.8 | Natural, varied phrasing without drifting. |
| Brainstorming, marketing copy, stories | 0.8 – 1.2 | Diversity of ideas is the goal. |
| Generating many diverse samples (self-consistency, synthetic data) | 0.7 – 1.0 with n > 1 | Different samples explore different answers; vote or filter afterwards. |
Using the settings in an API call
from openai import OpenAI
client = OpenAI()
def ideas(topic: str, temperature: float) -> list[str]:
response = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[{"role": "user", "content": f"Give one startup idea about {topic}. One line."}],
temperature=temperature,
n=3, # three independent samples in one request
max_tokens=40,
)
return [c.message.content for c in response.choices]
print("T=0.0:", ideas("vector databases", 0.0))
print("T=1.0:", ideas("vector databases", 1.0))Your exact ideas will differ, but the pattern will not: low temperature collapses onto the same answer, high temperature spreads out.
- Extend the sampling simulation with temperatures 0.5 and 1.5 and plot or print the share of “Paris” at each.
- Combine
softmax_tandtop_p: apply temperature 1.5, then top-p 0.9. Which tokens survive? - With a real API, ask for a product name at temperatures 0, 0.7, and 1.2 with
n=5, and count unique answers.
- Sampling turns probabilities into a choice; greedy decoding always takes the top token.
- Temperature divides logits before softmax: low = focused and repeatable, high = diverse and riskier.
- Top-k keeps a fixed number of candidates; top-p keeps an adaptive set covering probability p.
- Tune temperature or top-p, rarely both; leave penalties at 0 unless output loops.
- Temperature 0 is not a determinism guarantee — pin model versions, use structured outputs, and validate.