The term overfitting originates from the way predictive models are built – they are “fitted” to match the historical data. The fit can be poor – called underfitting – in which case the predictions are far away from most of the actual data points. Or it can be too close – called ovefitting – in which case, we are also force-fitting the noise rather than capturing the true underlying structure. As obvious as it may sound, many analysts/forecasters completely ignore this problem and hence develop not predictive models but chaotic models.
Overfitting usually happens in cases when the data is limited and noisy, but the main (de)motivation behind building overfitted models is the urge to build super-predictive models. Combine these ingredients and you have a recipe for predictive chaos.
Here’s a simple enough example to explain overfitting
An overfitted model will score high on various statistical tests and measures, but it scores those extra points by cheating – by fitting the noise rather than the true underlying structure. This might make it easier to sell the model to the client, but has the potential to hurt their business.
At Affine, we perform multiple diagnostic checks, both during model training and testing phases, to ensure that our models are overfit-free. Our in-house multi-tier validation framework leverages bagging (boostrap aggregation), where the models are built as well as validated on multiple boostrap samples (pulled with and without replacement). The no. of bootstrap samples may var from 20-50 depending on the underlying statistical model and samp size amongst other things. Multiple model parameters and performance metrics are validated for consistency across these bagged samples and summarized to create a final validation report that lets our analytic scientists take a call on overfitting as well as take measures to get rid of it.
‘Overfitting Diagnostic Check’ is just one of the many checkpoints through which we build fair, robust and more importantly, business-ready “longer shelf-life” models for our clients. What motivates our analytic scientists to perform these checks, which other may call overheads? Simple – skepticism, curiosity and a mindset which forces them to be critical of their own work to achieve continued excellence.
Watch out this space for more Affine Transformations…