Natural Language Processing & Language Models

Natural Language Processing (NLP) is the field of getting computers to work with human language — translating it, summarizing it, answering questions about it, or generating new text. Just like computer vision starts from "an image is a matrix of numbers," NLP starts from a similar question: how do you turn words into numbers?

Tokens

Text is first broken into tokens — the basic units a model processes. A token is often a whole word, but modern systems frequently use "subword" tokens, so that an uncommon word like "unbelievable" might be split into pieces like un, believ, able. This lets a model handle new or rare words by recombining familiar pieces rather than needing to have seen every possible word during training.

Embeddings: turning words into vectors

Once text is split into tokens, each token is converted into a vector (Module 2 again) called an embedding. Unlike an arbitrary ID number, an embedding is learned so that words used in similar contexts end up with similar vectors — the embeddings for "king" and "queen" end up close together in this vector space, and famously, vector arithmetic on embeddings can capture real relationships (the classic example: king − man + woman ≈ queen).

This is exactly why the dot product from Module 2 matters here too: measuring how similar two embeddings are (and therefore how related two words or pieces of text are) is typically done with a dot product or a closely related formula.

The core trick: predicting the next token

A large language model (LLM), like the kind that powers modern AI chat assistants, is trained on a deceptively simple task: given some text so far, predict what token comes next. It does this by outputting a probability for every possible next token, and the highest-probability guesses tend to make sense grammatically and contextually purely because the model was trained on enormous amounts of real text where those patterns hold.

Generating a longer response is just this single-token prediction, repeated: predict a token, add it to the text, predict the next one based on the now-longer text, and so on. The toy demo below works the same way, just with a tiny hand-built table of probabilities instead of a model trained on billions of words.

Why this feels like "understanding"

A model that's very good at predicting the next token, at massive scale, ends up implicitly encoding a huge amount about grammar, facts, and reasoning patterns present in its training text — which is why LLM output often looks fluent and even insightful. But it's worth being precise: the model is not looking up facts in a database or "thinking" the way a person does. It's producing the statistically most plausible continuation given everything it learned from its training data — which is also exactly why it can produce fluent, confident-sounding text that is factually wrong (see Module 11).

Modern chat-style language models add another layer on top of next-token prediction — techniques like reinforcement learning from human feedback (mentioned in Module 4) — to make their responses more helpful and better-aligned with what people actually want, but the underlying generation mechanism is still, fundamentally, "predict the next token, repeatedly."