Serving LLMs fast and cheap
Why generating text is a memory-bandwidth problem, and the tricks - batching, caching, quantization, speculative decoding - that make chatbots affordable.
- Explain the difference between prefill and decode, and the latency numbers that matter
- Estimate decode speed from memory bandwidth, and see how batching multiplies throughput
- Describe quantization, KV-cache paging, prefix caching and speculative decoding
Every chat reply has two phases:
- Prefill: the model reads your whole prompt in one parallel pass, filling the KV cache. Lots of arithmetic, done all at once - it’s compute-bound. It sets the time to first token (TTFT).
- Decode: the model generates one token at a time. Each step does little arithmetic but must read every weight from GPU memory. It’s memory-bandwidth-bound, and it sets the inter-token latency - how fast the words stream.
That gives a handy back-of-the-envelope rule: for a single user, tokens per second ≈ memory bandwidth ÷ bytes of weights.
bandwidth = 1000e9 # bytes per second - a typical modern GPU
for name, params, bytes_per in [("8B bf16", 8e9, 2), ("8B int4", 8e9, 0.5), ("70B bf16", 70e9, 2)]:
print(f"{name}: about {bandwidth / (params * bytes_per):.0f} tokens/s")8B bf16: about 62 tokens/s 8B int4: about 250 tokens/s 70B bf16: about 7 tokens/s
Batching: share the weight reads
Reading the weights once per step is the expensive part - so serve many users per step. With a batch of 32 sequences, one pass through the weights produces 32 tokens. Total throughput rises almost linearly until compute or KV-cache memory runs out.
Continuous batching goes further: instead of waiting for the whole batch to finish, the server swaps finished requests out and new ones in at every step. That’s the core idea of servers like vLLM, TGI and SGLang.
1bandwidth = 1000e9
2step = 8e9 * 2 / bandwidth # seconds to read an 8B bf16 model once
3for batch in [1, 8, 32]:
4 print(f"batch {batch}: {batch / step:.0f} tokens/s total")batch 1: 62 tokens/s total batch 8: 500 tokens/s total batch 32: 2000 tokens/s total
Quantization up close
Fewer bytes per weight means faster decode and smaller GPUs. The simplest scheme is absmax int8: divide a group of weights by max(|w|) / 127, round to integers, and multiply back when computing. The trouble is outliers: one huge weight stretches the scale, and all the small weights round to just a few levels.
1import numpy as np
2
3w = np.array([0.12, -0.5, 0.33, 0.07, 8.0]) # one outlier
4scale = np.abs(w).max() / 127
5q = np.round(w / scale).astype(int)
6print("q:", q.tolist())
7print("restored:", np.round(q * scale, 3).tolist())q: [2, -8, 5, 1, 127] restored: [0.126, -0.504, 0.315, 0.063, 8.0]
The fix used by real formats (GPTQ, AWQ, GGUF’s k-quants) is a separate scale per small group of weights - say 32 or 128 - plus smarter rounding that protects the weights that matter most. You’ll see the difference in the exercise.
Speculative decoding: guess, then check
Decode steps are bandwidth-bound, so a big model can check several tokens in one pass for nearly the price of one. Speculative decoding (Leviathan et al., 2023) uses a small, fast draft model to guess the next tokens; the big target model verifies them all at once, keeps the longest correct prefix, and adds its own token where the draft went wrong. With the right acceptance rule, the output distribution is exactly the target model’s.
If each draft token is accepted with probability , one target pass yields on average tokens:
k = 4
for alpha in [0.6, 0.8, 0.9]:
print(f"acceptance {alpha}: {(1 - alpha ** (k + 1)) / (1 - alpha):.2f} tokens per target pass")acceptance 0.6: 2.31 tokens per target pass acceptance 0.8: 3.36 tokens per target pass acceptance 0.9: 4.10 tokens per target pass
Try it
One round of speculative decoding
Follow a draft model and a target model writing “the cat sat on the mat”. Predict before each reveal.
Key takeaways
Prefill is compute-bound and sets time to first token; decode is bandwidth-bound and sets streaming speed.
Single-user decode speed ≈ bandwidth ÷ weight bytes; batching shares each weight read across many requests.
PagedAttention and prefix caching make the KV cache go further; group-wise quantization shrinks the weights.
Speculative decoding drafts cheaply and verifies in one big pass, without changing the output distribution.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Group-wise int8 quantization
Line 1 is a list of weights; line 2 is the group size. Quantize each group separately with absmax int8: scale = max(|w|) / 127 for the group, and q = round(w / scale).
Print group N: scale S q Q1 Q2 … (scale to 5 decimals), then mean error: E - the average of |w − q × scale| over all weights, to 5 decimals.
- One group with an outlier
- Small groups
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Simulate speculative decoding
Each input line is one round: draft: a b c | target: a b x y. The target list holds the big model’s own prediction at each drafted position, plus one more.
Accept draft tokens while they match the target’s, stopping at the first mismatch. Then add the target’s token at that position (the correction - or, if every draft token matched, the bonus token at the end). Print round R: +N tokens…, then a summary: T tokens in R target passes (X.XX per pass).
- A good draft model
- Every guess wrong
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…