Deploy, monitor, and govern ML systems
Detect drift with the population stability index, check performance across subgroups, and plan responses before launch.
- Distinguish data drift from concept drift
- Measure drift with the population stability index
- Compare error rates across subgroups and plan human oversight
A model is trained on the past and used in the future, and the world keeps changing:
- Data drift - the inputs change: a new customer segment, a new device, a pandemic.
- Concept drift - the relationship changes: the same features now mean a different outcome (fraudsters adapt).
Labels often arrive late, so teams watch the inputs first. A common drift score is the population stability index over binned feature shares, comparing the expected (training) share with the actual (recent) share in each bin:
A common rule of thumb: below 0.1 is stable, 0.1-0.25 deserves a look, above 0.25 is a significant shift.
Try it
Up to 15 XPData drift or concept drift?
Sort each production change.
“A marketing campaign brings in many younger customers than the training data had”
“Fraudsters switch tactics, so patterns that used to be safe are now risky”
“A new phone model sends images at a different resolution”
“After a policy change, the same symptoms now lead to a different diagnosis code”
Fairness and oversight
An overall accuracy can hide a model that fails one group far more often than another. Report error rates - especially false negative and false positive rates - per relevant subgroup, investigate gaps, and decide in advance who reviews flagged cases, how people can appeal, and when to roll back. Document the model’s intended use and limits (a “model card”).
import math
expected, actual = 0.20, 0.35
print(round((actual - expected) * math.log(actual / expected), 4))0.0839
Key takeaways
Monitor inputs (data drift) and, when labels arrive, outcomes (concept drift).
PSI compares binned feature shares; above about 0.25 signals a major shift.
Check errors per subgroup, keep humans in the loop for important decisions, and plan rollbacks.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Compute the population stability index
Line 1 holds the training share of each bin; line 2 the recent share. Floor every share at 0.0001 to avoid . Print PSI: X (4 decimals) and status: stable (< 0.1), status: watch (< 0.25) or status: shifted.
- Customer age
- A campaign brings younger customers
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Error rates by subgroup
Read a CSV with group,label,prediction (0/1). For each group (alphabetically), print GROUP: n=N accuracy=A fnr=F - the false negative rate is missed positives ÷ actual positives (2 decimals each; n/a if the group has no positives). Then print largest fnr gap: G (2 decimals) between groups.
- Two regions
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…