Teaching a Machine to Classify

Classification is about predicting a category rather than a number: is this email spam? Is this tumor benign or malignant? Which digit is handwritten here? k-Nearest Neighbors (k-NN) is one of the simplest classification algorithms, and it's a great way to build intuition because it has almost no "training" step at all.

The idea: judge by your neighbors

To classify a new point, k-NN looks at the $k$ closest points in the training data (by distance — usually straight-line/Euclidean distance, computed the same way you'd compute a vector's length in Module 2) and takes a majority vote of their labels. If most of a new point's nearest neighbors are labeled "cat," k-NN predicts "cat."

There's no equation to fit and no loss to minimize during training — k-NN just remembers all the training data, and does the work at prediction time. This makes it easy to understand and implement, though it can get slow with very large datasets since every prediction compares against every stored example.

Choosing k

The choice of $k$ matters a lot:

• A very small $k$ (like 1) makes the model sensitive to noise — a single mislabeled or unusual neighbor can flip the prediction.
• A very large $k$ smooths things out but can blur real boundaries between classes, especially if it starts pulling in points from the "wrong" region entirely.

Try adjusting $k$ in the demo above and clicking near the boundary between the two clusters — you'll see the prediction can flip depending on how many neighbors "vote."

Decision boundaries

Every classifier, no matter how it works internally, effectively draws a boundary through feature space that separates one predicted class from another. For k-NN with two clusters like the ones above, that boundary tends to roughly bisect the space between the clusters — though unlike linear regression's straight line, k-NN's boundary can bend and curve depending on where the training points sit.