Partition data with sharding
Split data across machines and add nodes without reshuffling everything.
- Compare range and hash partitioning.
- Choose a shard key that avoids hotspots.
- Explain how consistent hashing limits data movement.
When data or write traffic outgrows one machine, you shard (partition) it: each node holds a subset of the rows. The shard key decides where each row lives, and it is one of the most consequential choices in a design. A good key has many distinct values, spreads load evenly, and keeps data that is queried together on the same shard.
Partitioning strategies
- Range partitioning: split by key ranges (A-F, G-M, …). Range scans are efficient, but sequential keys such as timestamps send all new writes to one shard.
- Hash partitioning: place rows by
hash(key). Load spreads evenly, but range queries must ask every shard. - Directory-based: a lookup service maps keys to shards. Flexible, but the directory must be fast and highly available.
The naive hash scheme hash(key) % N has a nasty property: when N changes, most keys map to a different node, forcing a massive data migration.
A worked example: take 12 keys spread by key % 3, then go to key % 4. 9 of the 12 keys move, even though you only added one node.
Consistent hashing places both nodes and keys on a ring of hash values. Each key belongs to the first node clockwise from it. Adding a node only takes over the keys between it and its predecessor - about 1/N of the data - and removing one hands its keys to the next node. Each physical node usually owns many virtual nodes on the ring so load stays even. Systems such as Cassandra and DynamoDB are built on this idea.
Practice: choose a shard key
Try it
Up to 15 XPCompare shard keys
You have to name and defend a shard key in the interview. Try each one, and look at writes and reads together. Then turn on the huge customer.
- Shard 10%
- Shard 20%
- Shard 30%
- Shard 4100%
- To read one customer’s orders
- ask 4 shards
- To read one week’s orders
- ask 1 shard
Adding a shard: 4 to 5
- hash % shard count
- 80% of keys move
- Consistent hashing
- 24% of keys move
Practice: add a node to a ring
Try it
Up to 15 XPWhich keys move?
Each key belongs to the first node clockwise from it. Slide the new node around the ring, and watch which keys change owner. Why do so few move?
Notice how uneven the arcs are, too. That’s the problem virtual nodes solve.
Large circles are nodes, and small dots are keys, colored by owner. Outlined keys just moved.
- Keys moved on the ring
- 3 of 17
- Keys moved by hash % N, from 3 to 4
- 11 of 17
Key takeaways
Pick a high-cardinality shard key that matches your main queries.
Range partitioning helps scans but risks hotspots; hashing spreads load but scatters ranges.
Consistent hashing with virtual nodes moves only ~1/N of keys when membership changes.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…