Loading
0x90Lesson 10 of 11

Deploy, monitor, and govern ML systems

Detect drift with the population stability index, check performance across subgroups, and plan responses before launch.

22 min 6-question quiz 2 code exercises
By the end of this lesson you can
  • Distinguish data drift from concept drift
  • Measure drift with the population stability index
  • Compare error rates across subgroups and plan human oversight

A model is trained on the past and used in the future, and the world keeps changing:

  • Data drift - the inputs change: a new customer segment, a new device, a pandemic.
  • Concept drift - the relationship changes: the same features now mean a different outcome (fraudsters adapt).

Labels often arrive late, so teams watch the inputs first. A common drift score is the population stability index over binned feature shares, comparing the expected (training) share eie_i with the actual (recent) share aia_i in each bin:

PSI=∑i(ai−ei)ln⁡aiei\text{PSI} = \sum_{i} (a_i - e_i) \ln \frac{a_i}{e_i}

A common rule of thumb: below 0.1 is stable, 0.1-0.25 deserves a look, above 0.25 is a significant shift.

Try it

Up to 15 XP

Data drift or concept drift?

Sort each production change.

0 of 4 sortedScore 0/0
  • “A marketing campaign brings in many younger customers than the training data had”

  • “Fraudsters switch tactics, so patterns that used to be safe are now risky”

  • “A new phone model sends images at a different resolution”

  • “After a policy change, the same symptoms now lead to a different diagnosis code”

Fairness and oversight

An overall accuracy can hide a model that fails one group far more often than another. Report error rates - especially false negative and false positive rates - per relevant subgroup, investigate gaps, and decide in advance who reviews flagged cases, how people can appeal, and when to roll back. Document the model’s intended use and limits (a “model card”).

psi_bin.py
import math
expected, actual = 0.20, 0.35
print(round((actual - expected) * math.log(actual / expected), 4))
Output
0.0839

Key takeaways

  • Monitor inputs (data drift) and, when labels arrive, outcomes (concept drift).

  • PSI compares binned feature shares; above about 0.25 signals a major shift.

  • Check errors per subgroup, keep humans in the loop for important decisions, and plan rollbacks.

Lesson quiz

6 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Compute the population stability index

+25 XP

Line 1 holds the training share of each bin; line 2 the recent share. Floor every share at 0.0001 to avoid ln⁡0\ln 0. Print PSI: X (4 decimals) and status: stable (< 0.1), status: watch (< 0.25) or status: shifted.

  • Customer age
  • A campaign brings younger customers
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Error rates by subgroup

+25 XP

Read a CSV with group,label,prediction (0/1). For each group (alphabetically), print GROUP: n=N accuracy=A fnr=F - the false negative rate is missed positives ÷ actual positives (2 decimals each; n/a if the group has no positives). Then print largest fnr gap: G (2 decimals) between groups.

  • Two regions
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…