Trace pretraining and model adaptation

A chat model is built in stages:

1. Pretraining - next-token prediction over trillions of tokens of web pages, books, code and more. This is where knowledge and language ability come from, and nearly all of the compute goes. The result is a base model: brilliant at continuing text, not at following instructions.
2. Supervised fine-tuning (SFT) - training on curated examples of instructions and good responses, so the model behaves like an assistant.
3. Preference tuning - people (or models) compare pairs of responses; the model is pushed toward preferred ones. RLHF (Ouyang et al., 2022) trains a reward model and optimizes against it with reinforcement learning; DPO (Rafailov et al., 2023) learns directly from the preference pairs, more simply.

Many recent models add reinforcement learning on verifiable tasks - math problems and code with checkable answers - which teaches longer step-by-step “reasoning” before answering.

Training compute is roughly $6ND$ floating-point operations, where $N$ is the parameter count and $D$ the number of training tokens (Kaplan et al., 2020). The Chinchilla study (Hoffmann et al., 2022) found that for a fixed compute budget, the best results come from about 20 tokens per parameter - many earlier models were too big and undertrained. Today many models train far beyond 20× because a smaller model that’s cheaper to run is worth extra training.

Fine-tuning every weight of a big model is expensive. LoRA (Hu et al., 2021) freezes the original weights and learns a small low-rank update instead: for a $d_{in} \times d_{out}$ weight matrix, it trains two thin matrices of rank $r$, adding only $r(d_{in} + d_{out})$ parameters. With $r = 8$ that’s well under 1% of the original - small enough to train on one GPU and to keep many adapters for different tasks.