Explore Transformers and attention
Compute self-attention by hand, see why LLMs mask the future, and follow a Transformer block.
- Compute scaled dot-product attention
- Explain the causal mask used by decoder-only LLMs
- Describe what a Transformer block contains
The Transformer (Vaswani et al., 2017) is the architecture behind essentially every modern LLM. Its key idea is self-attention: every token builds its new representation by looking at the other tokens and deciding which ones matter.
Each token is projected into three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?). The attention weights from token to every token are
where is the size of the vectors, and token ’s output is the weighted sum of the values: .
Try it
Up to 15 XPWho does “it” attend to?
Toy weights for one attention head. Click it, then light, and see which earlier words they draw on. In a real model, different heads learn different relationships - one tracks pronouns, another syntax, another nearby words.
Click a word to see where it looks
“it” pays the most attention to “ball” (66%). The weights add up to 100%.
- The2.0%
- robot14.7%
- picked2.0%
- up2.0%
- the2.0%
- ball65.9%
- because2.0%
- it5.4%
- was2.0%
- light2.0%
No peeking: the causal mask
An LLM is trained to predict each next token, so a token must not see the tokens after it - that would be cheating. Decoder-only models (GPT, Claude, Llama and most chat models) apply a causal mask: scores for future positions are set to −∞ before softmax, so their weights become exactly 0.
A Transformer block is then: attention (many heads in parallel) → add & normalize → a feed-forward network applied to each token → add & normalize. Models stack dozens of blocks. Because attention itself ignores order, positional information (such as rotary position embeddings) is mixed in so “dog bites man” differs from “man bites dog”.
1import numpy as np
2scores = np.array([2.0, 1.0, -np.inf])
3weights = np.exp(scores - scores.max())
4weights /= weights.sum()
5print(np.round(weights, 2))[0.73 0.27 0. ]
Key takeaways
Attention weights are ; outputs are weighted sums of values.
Decoder-only LLMs mask future tokens so each position only sees the past.
A Transformer stacks blocks of multi-head attention and feed-forward layers, with positional information.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Scaled dot-product attention with numpy
Read three JSON matrices, one per line: Q, K and V. Compute row by row ( = number of columns of Q) and the output . Print weights: and output: followed by each as a nested list rounded to 2 decimals (np.round(...).tolist()).
- Two queries, three keys
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Apply a causal mask
Read a square JSON matrix of attention scores. Set every score above the diagonal (future positions) to −∞, apply softmax to each row, and print each row’s weights with 2 decimals separated by spaces.
- Three tokens
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…