Add context: from word vectors to Transformers
See why one vector per word isn’t enough, and how Transformers build context-aware representations.
- Explain why a single fixed vector per word falls short
- Describe what a Transformer layer and self-attention add
- Explain how a language model generates text one token at a time
A word2vec-style model gives bank one vector. But “we sat on the river bank” and “I deposited cash at the bank” use very different meanings. A fixed (static) vector has to average them. Modern models produce contextual embeddings: the vector for bank changes depending on the words around it.
Try it
Up to 15 XPWhich bank?
Read each sentence and decide which meaning of bank it uses. Ask yourself which other words gave it away - those are exactly the words a contextual model pays attention to.
“We had a picnic on the bank of the river.”
“The bank approved her loan.”
“Fish gathered near the muddy bank.”
“He opened an account at the bank downtown.”
The Transformer in one picture
The Transformer (Vaswani et al., 2017) is the architecture behind today’s language models. Very roughly:
- Each token becomes an embedding, plus positional information (attention on its own doesn’t know word order).
- A stack of layers each runs self-attention (every token gathers information from the others) and a small feed-forward network.
- After many layers, each token’s vector reflects its whole context.
A decoder-style language model uses the final vectors to predict the next token, appends it, and repeats - the same next-word game as the bigram model, but with a context of thousands of tokens and learned representations instead of counts.
1import math
2a = [1, 0]
3b = [0.8, 0.6]
4dot = sum(x * y for x, y in zip(a, b))
5cosine = dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
6print(round(cosine, 2))0.8
Key takeaways
Static vectors give a word one meaning; contextual embeddings change with the surrounding words.
Transformers stack self-attention layers, plus positional information for word order.
Language models generate by repeatedly predicting the next token - fluent, but not guaranteed true.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Compare two vectors
Read two space-separated, equal-length, non-zero vectors on separate lines. Calculate cosine similarity and print it to two decimal places.
- Perpendicular vectors
- Similar vectors
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…