Model size, memory and the KV cache
Estimate how much memory a model needs, why quantization helps, and what the KV cache costs.
- Estimate weight memory from parameter count and precision
- Explain quantization and its trade-offs
- Compute the size of the KV cache for long contexts
A model’s parameters (weights) are just numbers, and every number takes memory. Rough rule: memory for weights = parameters × bytes per parameter.
| Precision | Bytes per parameter | 8B model |
|---|---|---|
| fp32 | 4 | 32 GB |
| fp16 / bf16 | 2 | 16 GB |
| int8 | 1 | 8 GB |
| int4 | 0.5 | 4 GB |
Quantization stores weights with fewer bits. It lets bigger models run on smaller GPUs (or laptops) and speeds up inference, usually with a small quality loss that grows as bits shrink.
The KV cache
During generation, each new token attends to every earlier token. Recomputing their keys and values every step would be wasteful, so models keep them in a KV cache. Its size grows with context length:
KV bytes = 2 (keys and values) × layers × KV heads × head size × tokens × batch × bytes per value
For long contexts this can rival the weights themselves. That’s why many models use grouped-query attention (several query heads share one KV head), which shrinks the cache - and why long prompts cost more memory and time.
bytes_per_parameter = {"fp32": 4, "bf16": 2, "int8": 1, "int4": 0.5}
for precision, size in bytes_per_parameter.items():
print(precision, 8e9 * size / 1e9, "GB")fp32 32.0 GB bf16 16.0 GB int8 8.0 GB int4 4.0 GB
Key takeaways
Weight memory ≈ parameters × bytes per parameter.
Quantization trades a little quality for much less memory and faster inference.
The KV cache grows with context length; grouped-query attention shrinks it.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Estimate weight memory
Each line is PARAMS_IN_BILLIONS PRECISION (fp32, bf16, fp16, int8 or int4). Print NB PRECISION: X GB with X = billions × bytes per parameter, to 1 decimal.
- Four models
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Size the KV cache
Each line is layers kv_heads head_dim tokens batch bytes. Print the KV cache size in GB (1 GB = 10⁹ bytes) with 2 decimals: KV cache: X GB.
- Grouped-query vs full attention
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…