Connect embeddings, attention, and language models
Relate vector representations to context-aware Transformer models.
- Compare simple vectors and describe what self-attention contributes.
An embedding maps a token or text span to a vector of numbers. Learned embeddings can place items used in similar contexts near one another, though similarity does not guarantee identical meaning. Transformer layers use self-attention to combine information from different sequence positions; positional information helps represent order. Decoder-style language models use context to predict a next token, then repeat that step. See the original Transformer paper.
1import math
2a = [1, 0]
3b = [0.8, 0.6]
4dot = sum(x * y for x, y in zip(a, b))
5cosine = dot / (math.sqrt(sum(x*x for x in a)) * math.sqrt(sum(y*y for y in b)))
6print(round(cosine, 2))0.8
This vector comparison is a toy example, not an embedding model. Real embeddings are learned from data and may depend on context. Self-attention lets each position use information from other positions. Fluent generated text can still be incorrect, biased, or unsupported; verify important claims.
Key takeaways
Compare simple vectors and describe what self-attention contributes.
Simple baselines help make ideas concrete.
Interpret language tools in context and check important results.
Lesson quiz
4 questions · pass with 3 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Compare two vectors
Read two space-separated, equal-length, non-zero vectors on separate lines. Calculate cosine similarity and print it to two decimal places.
- Perpendicular vectors
- Similar vectors
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…