Understand inference and generation controls
Turn logits into choices with temperature, top-k and top-p, and understand latency, cost and streaming.
- Explain greedy decoding and sampling
- Predict the effect of temperature, top-k and top-p
- Reason about tokens, latency and cost
At each step the model outputs logits for every token. How you pick from them is the decoding strategy:
- Greedy: always take the most likely token. Repeatable, but can get stuck in bland or repetitive text.
- Sampling: draw a token at random according to the probabilities.
- Temperature divides the logits before softmax: below 1 sharpens the distribution toward the top token; above 1 flattens it toward variety (and nonsense).
- Top-k keeps only the k most likely tokens. Top-p (nucleus sampling, Holtzman et al., 2019) keeps the smallest set of top tokens whose probabilities add up to p - adapting to how confident the model is.
Try it
Up to 15 XPSampling lab
These are a model’s scores for the next token. Sample 100 times at temperature 1, then try 0 (greedy), 0.3 and 2. Then set top-p to 0.9 and watch the unlikely tail disappear.
The capital of France is ▁
- “ Paris”78.5%
- “ a”9.6%
- “ the”5.8%
- “ known”3.9%
- “ Lyon”1.2%
- “ not”0.6%
- “ Berlin”0.3%
Struck-through tokens were filtered out by top-k or top-p; the remaining probabilities are rescaled to add up to 100%.
Tokens are time and money
Inference has two phases: prefill reads the whole prompt in parallel (fast per token), then decode produces output one token at a time (each needs a full pass through the model). So latency ≈ time-to-first-token + output tokens ÷ tokens per second, and providers price output tokens higher than input tokens.
Streaming shows tokens as they’re generated - the answer feels faster even though the total time is the same. Prompt caching reuses the processed prefix of repeated prompts, cutting cost and time-to-first-token.
first_token_seconds = 0.4
output_tokens, tokens_per_second = 600, 60
print(first_token_seconds + output_tokens / tokens_per_second, "seconds")10.4 seconds
Key takeaways
Greedy picks the top token; sampling draws from the distribution.
Temperature sharpens or flattens; top-k and top-p cut off the unlikely tail.
Output tokens dominate latency and cost; streaming and caching help.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Softmax with temperature (numpy)
Line 1 holds logits; line 2 holds temperatures. For each temperature, print T=t: p1 p2 ... - the softmax of logits / t, each with 3 decimals.
- Three temperatures
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Nucleus (top-p) filtering
Line 1 is p. Each following line is token probability. Sort by probability (highest first), keep tokens until their running total reaches at least p, renormalize, and print each kept token as token probability (3 decimals).
- p = 0.8 needs three tokens
- p = 0.5 keeps one
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…