Protect services with rate limiting
Compare limiter algorithms and run one across many servers.
- Compare fixed window, sliding window, and token bucket limiters.
- Explain how burst size and sustained rate map to a token bucket.
- Design a rate limiter shared by many API servers.
A rate limiter caps how many requests a client (by user, API key, or IP) can make in a period. It protects services from abuse and overload, enforces fair use and pricing tiers, and limits the blast radius of a buggy client. Rejected requests get HTTP 429 Too Many Requests, ideally with a Retry-After header.
Limiter algorithms
- Fixed window counter: count requests per clock window (for example per minute). Simple and cheap, but a client can send a full limit at the end of one window and another at the start of the next - about 2× in a short span.
- Sliding window log: store each request’s timestamp and count those within the last window. Accurate, but memory grows with traffic.
- Sliding window counter: blend the current and previous window counts. A cheap, good approximation.
- Token bucket: a bucket holds up to capacity tokens and refills at a steady rate. Each request spends a token. Capacity sets the allowed burst; the refill rate sets the sustained throughput.
1limit = 3
2counts = {}
3for timestamp in [1, 2, 2, 3, 3, 61]:
4 window = timestamp // 60
5 counts[window] = counts.get(window, 0) + 1
6 print(timestamp, "allowed" if counts[window] <= limit else "rejected")1 allowed 2 allowed 2 allowed 3 rejected 3 rejected 61 allowed
With many API servers, each server’s local counter only sees part of the traffic. A distributed limiter keeps counters in a shared, fast store such as Redis, using atomic operations (INCR with an expiry, or a Lua script for a token bucket) so concurrent requests cannot race. Limiters usually live in an API gateway or middleware, and should fail open or closed deliberately if the store is unavailable.
Key takeaways
Token bucket: capacity is the burst, refill rate is the sustained limit.
Fixed windows are cheap but allow boundary bursts.
Distributed limiters need a shared store and atomic updates.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: simulate system design building blocks
Use small Python programs to estimate capacity and simulate caches, load balancers, hash rings, and rate limiters. These exercises run locally in your browser.
Implement a token bucket
Read capacity rate (tokens per second), then a line of non-decreasing request timestamps in whole seconds. The bucket starts full. Before each request, add (t - last) * rate tokens, capped at capacity. If at least one token is available, spend it and print t allowed; otherwise print t rejected.
- Burst then refill
- Refill is capped
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…