What Is Pre-training? How AI Models Get Their General Education

Before a doctor learns to perform surgery, they spend years in school learning to read, write, do math and understand how the world works. That general education is the foundation for everything else.

AI chatbots like ChatGPT, Claude and Gemini go through something similar. Before they learn to be helpful assistants, they spend months on a giant first stage of learning called pre-training. It's where they pick up language, facts, and a surprising amount of general knowledge, all by playing one simple game over and over.

This article explains pre-training in plain language: what it is, how it works, what a model learns from it, and why it's so expensive.

What is pre-training?

Pre-training is the first and biggest stage of training an AI model. The model reads an enormous amount of general material, like text from books, websites and articles, and learns broad patterns from it. Only after that does it get trained for a specific job.

The "pre" just means before: it's the training that happens before the specialized training. In fact, the "P" in GPT, as in ChatGPT, stands for pre-trained: Generative Pre-trained Transformer.

Here's how the stages line up, compared with a person's education:

1. Pre-training: read a huge library and learn general patterns. This takes months and most of the money.
2. Fine-tuning: practice on smaller, carefully chosen examples of a specific job, like answering questions helpfully.
3. Human feedback: people rate or compare the model's answers, and the model is adjusted to give the kinds of answers people prefer.

The one simple game: guess the next word

So how does a model "read" and learn? It plays a guessing game. Show it some text, hide what comes next, and ask it to guess the next word. Then reveal the answer, and adjust the model slightly so it would guess a bit better next time.

The clever part is that every sentence turns into many practice questions:

Notice something important: nobody had to label anything. The right answer, the next word, is already sitting in the text. That's why it's called self-supervised learning: the data supervises itself. Compare that with the careful human work described in What Is Data Labeling?. Because no labels are needed, models can learn from practically unlimited amounts of text.

(Strictly speaking, models work with tokens, which are words or pieces of words, rather than whole words. But "guess the next word" is a good way to picture it.)

What a model learns from such a simple game

Guessing the next word sounds almost silly. But to guess well across trillions of examples, a model has to pick up a lot more than vocabulary:

To predict "went", it has to learn grammar. To predict "Paris", it has to soak up facts. To predict "cracked", it has to absorb a bit of common sense about how the world works. Read enough computer code, and it starts to pick up programming. Read enough text in different languages, and it picks up translation.

None of these skills is taught directly. They emerge because they help with the one goal: guessing the next word well.

A truly enormous library

Pre-training needs a mind-boggling amount of text. It usually comes from a mix of:

• Web pages, often from huge public archives of the internet.
• Books and encyclopedias, like Wikipedia.
• Computer code from public projects.
• Scientific papers, news, forums and more.

Before training, this text is cleaned up: duplicates are removed, spam and low-quality pages are filtered out, and harmful or private content is reduced as much as possible.

And the amount keeps growing:

In just six years, the amount of text grew more than 4,000 times. How much reading is 15 trillion tokens? Let's compare it with a very dedicated human reader:

Someone reading 8 hours a day, every single day for 80 years, would get through about 3.5 billion words. Llama 3's pre-training text is roughly 3,200 of those lifetimes. Put another way, one person would need about 250,000 years of full-time reading to get through it. (This is a rough comparison, assuming about three-quarters of a word per token.)

Why pre-training costs so much

Reading all that text isn't just a matter of storage. For every single token, the model makes a guess, checks it, and adjusts its billions of internal settings. That takes:

• Thousands of specialized chips (GPUs) working together, often for months.
• Huge amounts of electricity to run and cool them.
• Teams of experts to collect and clean data and keep training running smoothly.

The biggest pre-training runs today are estimated to cost tens or even hundreds of millions of dollars. OpenAI's CEO, Sam Altman, said in 2023 that training GPT-4 cost more than $100 million. That's why only a handful of companies and research labs pre-train the largest models from scratch.

Bigger isn't automatically better, though. In 2022, DeepMind's Chinchilla research found that many big models had been trained on too little text for their size, and that a smaller model trained on more data could beat a larger one. Getting the balance right saves a lot of money.

What a pre-trained model can and can't do

After pre-training, you have what's called a base model (or foundation model). It's very knowledgeable, but it isn't an assistant yet. It just continues text.

Ask a base model "What is the capital of France?" and it might reply with "What is the capital of Germany? What is the capital of Spain?", because on the web, questions like that often appear in lists of quiz questions. It's not being difficult: it's doing exactly what it learned, which is continuing text in a likely way.

That's why fine-tuning and human feedback come next: they teach the model to follow instructions, answer helpfully, and avoid harmful responses. But almost all of the model's knowledge and language ability comes from pre-training.

A few other things to know:

• Knowledge cutoff: a model only knows about the world up to when its pre-training text was collected. Ask it about later events, and it may not know, or may guess.
• It learns the bad with the good: the internet contains mistakes, myths and prejudice, and a model can absorb those too. Later training stages and filtering try to reduce this, but it's an ongoing challenge.
• Not just for chatbots: pre-training is used all over AI. Vision models are often pre-trained on millions of photos and then fine-tuned to spot, say, problems in X-rays. This "learn general things first, then specialize" idea is called transfer learning.

A small glossary

• Pre-training: the first, biggest stage of training, where a model learns general patterns from a huge amount of data.
• Fine-tuning: further training on a smaller, focused set of examples to specialize a model.
• Human feedback: people rating or comparing answers to steer the model toward better ones.
• Base model (or foundation model): a model after pre-training, before it's turned into an assistant.
• Self-supervised learning: learning where the data provides its own answers, like guessing the next word.
• Token: a word or piece of a word, the unit models read and write.
• Knowledge cutoff: the point in time after which a model's training data has no information.
• Transfer learning: reusing what a model learned in general for a new, specific task.
• GPT: Generative Pre-trained Transformer, the type of model behind ChatGPT.

Wrapping up

Pre-training is the general education of an AI model. By playing one simple game, guessing the next word, across a library thousands of human lifetimes long, a model absorbs grammar, facts, common sense, code and more. It's slow, enormously expensive, and only a few organizations can do it at the largest scale, but it's the foundation that everything else is built on.

So when an AI chatbot explains a recipe, translates a sentence or helps fix a bug, remember that the helpful assistant on top was built on months of reading underneath.