Loading
0x40Lesson 5 of 11

Train models and generalize to new data

See overfitting happen, choose model complexity with validation data and cross-validation, and rein it in with regularization.

25 min 6-question quiz 2 code exercises
By the end of this lesson you can
  • Recognize underfitting and overfitting from training and validation error
  • Use k-fold cross-validation for reliable model choices
  • Explain how regularization reduces overfitting

A model is only useful if it generalizes - performs well on new data, not just the examples it trained on. Two ways to fail:

  • Underfitting (high bias): the model is too simple to capture the pattern. Training and validation error are both high.
  • Overfitting (high variance): the model memorizes noise in the training data. Training error is low, but validation error is high.

The cure is to choose complexity by validation error, never training error - and to keep a separate test set for one final, honest estimate.

Try it

Up to 15 XP

Overfitting in action

Ten noisy training points (filled) come from a smooth curve; ten more (hollow) are held out for validation. Raise the polynomial degree from 1 to 9. Training error keeps falling - at degree 9 the curve passes through every training point - but where is validation error lowest?

● training · ○ validation
Error by degree (log scale): solid training, dashed validation

Training error

0.378

Validation error

0.277

Underfitting: too simple to follow the curve.

Cross-validation and regularization

With little data, one validation split is noisy. k-fold cross-validation splits the training data into kk folds, trains kk times - each time validating on a different fold - and averages the scores. Every example gets used for validation exactly once.

Regularization fights overfitting by penalizing large weights. Ridge (L2) regression minimizes

L(w)=1n∑i=1n(y^i−yi)2+λ∑jwj2L(w) = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2 + \lambda \sum_j w_j^2

so the model only uses big weights when they really pay off. A larger λ\lambda means a simpler, smoother model. Other levers: more data, fewer features, and early stopping.

folds.py
1examples = list(range(10))
2k = 5
3for fold in range(k):
4    print(fold, examples[fold::k])
Output
0 [0, 5]
1 [1, 6]
2 [2, 7]
3 [3, 8]
4 [4, 9]

Key takeaways

  • Underfitting: both errors high. Overfitting: low training error, high validation error.

  • Choose complexity with validation data or k-fold cross-validation; keep the test set for the end.

  • Regularization, more data and simpler models reduce overfitting.

Lesson quiz

6 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Diagnose the fit

+25 XP

Each line is model train_error validation_error. Print MODEL: underfitting if both errors are above 0.2, MODEL: overfitting if validation error exceeds training error by more than 0.1 (and not underfitting), otherwise MODEL: good fit. Then print choose: MODEL for the lowest validation error.

  • Four degrees
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

k-fold cross-validation

+25 XP

Line 1 is k; line 2 holds x values; line 3 y values. For fold ff, the validation examples are those at indices ii with i mod k=fi \bmod k = f. Fit a line on the rest with np.polyfit(x, y, 1), then compute the validation mean squared error.

Print fold F: mse M for each fold and mean mse: M (4 decimals).

  • Five folds
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…