Choose useful evaluation metrics
Read a confusion matrix, compute precision, recall and F1, use ROC AUC, and pick regression metrics.
- Build a confusion matrix and compute precision, recall and F1
- Explain ROC curves and AUC
- Choose metrics from the costs of different errors
Accuracy - the share of correct predictions - can be badly misleading. If 1% of transactions are fraud, a model that always says “legitimate” is 99% accurate and completely useless. The confusion matrix counts each kind of outcome: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). From it:
Precision: of what we flagged, how much was right? Recall: of what was really there, how much did we catch? F1 is their harmonic mean, which stays low if either one is low.
Try it
Up to 15 XPWhich metric matters most?
Pick the metric each team should watch most closely.
“Cancer screening: missing a case is far worse than a follow-up test”
“Spam filter: a lost important email is worse than some spam getting through”
“Estimating delivery times in minutes”
“Finding every customer affected by a data breach”
“Auto-blocking accounts for fraud, where each block angers a real customer”
Threshold-free: ROC and AUC
Precision and recall depend on the threshold. An ROC curve plots the true positive rate (recall) against the false positive rate at every threshold. The area under it, AUC, summarizes ranking quality in one number: the probability that a random positive example gets a higher score than a random negative one. 0.5 is random guessing; 1.0 is perfect ranking.
For regression, the mean absolute error is in the target’s own units (“off by 4 minutes on average”); RMSE penalizes big misses more.
1labels = [0] * 99 + [1]
2always_negative = [0] * 100
3accuracy = sum(a == b for a, b in zip(labels, always_negative)) / 100
4recall = 0 / 1
5print(accuracy, recall)0.99 0.0
Key takeaways
Accuracy hides failures on rare classes; read the confusion matrix.
Precision: flagged and right. Recall: real and caught. F1 balances both.
AUC measures ranking across all thresholds; MAE and RMSE measure regression error.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Confusion matrix and metrics
Line 1 holds true labels, line 2 predicted labels (0/1). Print TP=a FP=b FN=c TN=d, then accuracy, precision, recall and f1, each as NAME: X with 3 decimals (use 0.000 when a denominator is 0).
- A fraud model
- Always negative
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
ROC AUC by counting pairs
Line 1 holds true labels; line 2 scores. Compute AUC as the share of (positive, negative) pairs where the positive has the higher score, counting ties as half. Print AUC: X (3 decimals).
- The fraud scores
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…