Overfitting is not incompetence. It is a natural consequence of training on finite data with a model complex enough to memorise it.
The validation discipline: always hold out a test set that no preprocessing or hyperparameter decision touches. Track training and validation loss together — when they diverge significantly, you are overfitting.
For the AfriCred credit model, we validated on a six-month holdout after the training data ended. The model that looked best on cross-validation looked worse on temporal validation — the loan market had shifted. That discovery saved us from deploying a model that would have degraded rapidly.
Catch the overfitting. The test set is the last line of defence.
— Dick Bassey | DevDick | 2023