Overfitting Is Not Embarrassing. Deploying It Is.

·1 min read·

Every ML model overfits during training at some point. The engineers who catch it before production know what to look for.

Overfitting is not incompetence. It is a natural consequence of training on finite data with a model complex enough to memorise it.

The validation discipline: always hold out a test set that no preprocessing or hyperparameter decision touches. Track training and validation loss together — when they diverge significantly, you are overfitting.

For the AfriCred credit model, we validated on a six-month holdout after the training data ended. The model that looked best on cross-validation looked worse on temporal validation — the loan market had shifted. That discovery saved us from deploying a model that would have degraded rapidly.

Catch the overfitting. The test set is the last line of defence.

— Dick Bassey | DevDick | 2023

Share this article
← Back to all writing