Evaluate a classifier honestly
Go beyond accuracy with precision, recall, F1 and the decision threshold.
- Read a confusion matrix
- Calculate precision, recall and F1
- Choose a decision threshold for the job at hand
A disease affects 1 in 100 people. A “classifier” that says healthy to everyone is 99% accurate - and completely useless. When one class is rare, accuracy hides failure. We need measures that look at the class we actually care about.
The confusion matrix
Every prediction lands in one of four boxes:
| Predicted spam | Predicted ham | |
|---|---|---|
| Really spam | true positive (TP) | false negative (FN) - missed |
| Really ham | false positive (FP) - false alarm | true negative (TN) |
- Precision = TP / (TP + FP): of everything flagged, how much was right?
- Recall = TP / (TP + FN): of everything that should be flagged, how much did we catch?
- F1 = 2 × precision × recall / (precision + recall): one number that is high only when both are.
Try it
Tune a spam filter
A spam filter gives each message a score from 0 to 1. You choose the threshold above which a message goes to the spam folder.
- Find a threshold with 100% recall (no spam gets through). What does it cost you?
- Find one with 100% precision (no real email is lost). What does that cost?
- Which would you choose for a personal inbox, where losing a real email is worse than seeing a spam?
| Flagged spam | Called ham | |
|---|---|---|
| Really spam | 5 true positives | 1 missed (false negatives) |
| Really ham | 2 false alarms (false positives) | 3 true negatives |
- 0.97WIN a free cruise!!! (spam)flagged spam
- 0.91Claim your prize money (spam)flagged spam
- 0.84Cheap meds, no prescription (spam)flagged spam
- 0.72Free pizza in the break room (ham)flagged spam
- 0.66Limited offer for members (spam)flagged spam
- 0.58Your invoice is attached (ham)flagged spam
- 0.52Exclusive deal just for you (spam)flagged spam
- 0.35Reminder: dentist at 9am (ham)ham
- 0.28Hi, are you free this weekend? (spam)ham
- 0.12Notes from today’s meeting (ham)ham
- 0.05Happy birthday from Mom (ham)ham
1true_positives, false_positives, false_negatives = 8, 2, 4
2precision = true_positives / (true_positives + false_positives)
3recall = true_positives / (true_positives + false_negatives)
4f1 = 2 * precision * recall / (precision + recall)
5print(round(precision, 2), round(recall, 2), round(f1, 2))0.8 0.67 0.73
Key takeaways
Accuracy misleads when classes are imbalanced - look at the confusion matrix.
Precision asks “were the flags right?”, recall asks “did we catch them all?”, F1 balances both.
The threshold trades one for the other; choose it by the cost of each kind of mistake.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: apply NLP with Python
Try each text-processing idea in Python, run it against sample inputs, and use the results to see where the method works or falls short.
Calculate precision, recall and F1
The first line holds the true labels and the second line the predicted labels, as space-separated 1s (positive) and 0s (negative). Print precision: P, recall: R and f1: F, each rounded to 2 decimals. Treat any 0/0 as 0.0.
- Some of each
- Cautious classifier
- Nothing flagged
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…