Model short sequences with n-grams
Count nearby word sequences, generate text from them, and meet their limits.
- Count bigrams and turn the counts into next-word probabilities
- Generate text by sampling from a bigram model
- Explain sparsity and why smoothing is needed
Your phone’s keyboard suggests the next word as you type. The oldest way to do that is to remember which words tend to follow which. An n-gram is a sequence of n consecutive tokens - a bigram has two - and an n-gram language model predicts the next word from counts of these short sequences.
words = "we learn language".split()
bigrams = list(zip(words, words[1:]))
print(bigrams)[('we', 'learn'), ('learn', 'language')]From bigram counts, the probability of a next word is P(next | previous) = count(previous next) / count(previous). If “I” is followed by “like” twice and “see” once, then P(like | I) = 2/3.
Try it
Up to 15 XPWrite with a bigram model
This model was trained on the short text you can reveal below. Build a sentence word by word:
- Roll the dice a few times and start over each time: does it ever produce a sentence that wasn’t in the training text?
- Always pick the most likely: what happens to the variety?
- Do the sentences make sense beyond two words at a time?
the
After “the”, the corpus continues with… (choose one)
The sparsity problem
Language is endlessly creative, so most possible word pairs never appear in any corpus. An unseen bigram gets probability zero, which means the model says a perfectly fine sentence is impossible.
Smoothing fixes this by giving unseen pairs a little probability. The simplest version, add-one (Laplace) smoothing, pretends every pair was seen one extra time: P(next | previous) = (count(previous next) + 1) / (count(previous) + V), where V is the vocabulary size.
Longer n-grams capture more context (“New York” vs “new car”) but make sparsity much worse. Neural language models replace counting with learned representations, which is how they handle contexts of thousands of tokens.
Key takeaways
A bigram model predicts the next word from counts:
count(previous next) / count(previous).Sampling gives variety; always taking the top word gives repetitive text.
Unseen sequences get zero probability without smoothing, and longer n-grams are ever sparser.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Count adjacent word pairs
Read a line, lowercase it, extract alphabetic words, count adjacent bigrams, and print each pair alphabetically as first second: count.
- Repeated pair
- Two pairs
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Predict the next word
The first line is a training text. The second line is a word. Lowercase the text and extract alphabetic words, then print every word that follows the given word, most likely first (ties alphabetically), as word: probability with the probability rounded to 2 decimals. If the word is never followed by anything, print unseen.
- Two continuations
- One continuation
- Unseen word
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…