Minimum Description Length and Occam's Razor
Symptom You fit a model. It scores 94% on training data and 71% on held-out data. So you simplify: fewer parameters, more regularization. Training drops to 88%, held-out rises to 84%. You simplify further and both drop. Somewhere in there was an optimum, and you found it by trial and error, with a validation set and patience. ...