Train models and generalize to new data
See overfitting happen, choose model complexity with validation data and cross-validation, and rein it in with regularization.
- Recognize underfitting and overfitting from training and validation error
- Use k-fold cross-validation for reliable model choices
- Explain how regularization reduces overfitting
A model is only useful if it generalizes - performs well on new data, not just the examples it trained on. Two ways to fail:
- Underfitting (high bias): the model is too simple to capture the pattern. Training and validation error are both high.
- Overfitting (high variance): the model memorizes noise in the training data. Training error is low, but validation error is high.
The cure is to choose complexity by validation error, never training error - and to keep a separate test set for one final, honest estimate.
Try it
Up to 15 XPOverfitting in action
Ten noisy training points (filled) come from a smooth curve; ten more (hollow) are held out for validation. Raise the polynomial degree from 1 to 9. Training error keeps falling - at degree 9 the curve passes through every training point - but where is validation error lowest?
Training error
0.378
Validation error
0.277
Underfitting: too simple to follow the curve.
Cross-validation and regularization
With little data, one validation split is noisy. k-fold cross-validation splits the training data into folds, trains times - each time validating on a different fold - and averages the scores. Every example gets used for validation exactly once.
Regularization fights overfitting by penalizing large weights. Ridge (L2) regression minimizes
so the model only uses big weights when they really pay off. A larger means a simpler, smoother model. Other levers: more data, fewer features, and early stopping.
1examples = list(range(10))
2k = 5
3for fold in range(k):
4 print(fold, examples[fold::k])0 [0, 5] 1 [1, 6] 2 [2, 7] 3 [3, 8] 4 [4, 9]
Key takeaways
Underfitting: both errors high. Overfitting: low training error, high validation error.
Choose complexity with validation data or k-fold cross-validation; keep the test set for the end.
Regularization, more data and simpler models reduce overfitting.
Lesson quiz
6 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Diagnose the fit
Each line is model train_error validation_error. Print MODEL: underfitting if both errors are above 0.2, MODEL: overfitting if validation error exceeds training error by more than 0.1 (and not underfitting), otherwise MODEL: good fit. Then print choose: MODEL for the lowest validation error.
- Four degrees
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
k-fold cross-validation
Line 1 is k; line 2 holds x values; line 3 y values. For fold , the validation examples are those at indices with . Fit a line on the rest with np.polyfit(x, y, 1), then compute the validation mean squared error.
Print fold F: mse M for each fold and mean mse: M (4 decimals).
- Five folds
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…