0x00Lesson 1 of 16

Understand tokens and next-token prediction

See how text becomes tokens, how a model scores the next token, and how loss and perplexity measure it.

22 min 7-question quiz 2 code exercises

By the end of this lesson you can

Explain tokens and why they aren’t words
Describe autoregressive generation as repeated next-token prediction
Compute cross-entropy loss and perplexity

A large language model never sees letters or words. Text is first split into tokens - common words, pieces of words, punctuation, spaces - by a tokenizer such as byte-pair encoding (BPE). Each token becomes an integer ID. In English a token averages roughly ¾ of a word, but rare words, code and many other languages need more tokens per word.

The model’s whole job is one question, asked over and over: given these tokens, how likely is each possible next token? It outputs a score (a logit) for every token in its vocabulary - often 100,000 or more - which softmax turns into probabilities. To write a reply it picks a token, appends it, and asks again. That loop is autoregressive generation.

Try it

Up to 15 XP

Generate text one token at a time

This is a tiny next-word model trained on a few sentences. Build a sentence by picking each next word. Notice: the model only knows which words followed which in its training text - it has no idea whether the result is true.

the

After “the”, the corpus continues with… (choose one)

Measuring prediction: loss and perplexity

Training shows the model real text and checks the probability it gave each actual next token. The cross-entropy loss is the average of $-\ln p$ over those tokens: a confident correct guess (p = 0.9) costs 0.11; a bad one (p = 0.01) costs 4.6. Training adjusts billions of weights to push this number down.

Perplexity is $e^{\text{loss}}$ . It reads as “how many equally likely options the model was choosing between, on average”: a perplexity of 3 means it was about as uncertain as picking among 3 tokens.

loss.py

import math
for probability in [0.9, 0.5, 0.01]:
    print(probability, round(-math.log(probability), 2))

Output

0.9 0.11
0.5 0.69
0.01 4.61

Key takeaways

Text becomes tokens - often word pieces - before the model sees it.
An LLM predicts a probability for every possible next token, then repeats.
Cross-entropy loss is the average $-\ln p$ of the true tokens; perplexity is $e^{\text{loss}}$ .

Lesson quiz

7 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Next-token probabilities from counts

+25 XP

Line 1 is a training text; line 2 is a word. Count which words follow that word in the text, and print the top 3 as word probability (2 decimals), most likely first, ties alphabetically.

After “the”

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Loss and perplexity

+25 XP

The input lists the probabilities a model gave to each correct next token. Print loss=L perplexity=P, where loss is the mean of $-\ln p$ (3 decimals) and perplexity is $e^{\text{loss}}$ (2 decimals).

A mixed model
A confident model

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…