Symptom
Two models, same dataset, same 30% error.
Model A is a depth-2 decision tree. It gets 29% error on training and 30% on held-out data. Retrain it on a different sample of the same size and you get an almost identical tree.
Model B is a depth-30 tree. It gets 0% on training and 30% held-out. Retrain it on a different sample and you get a completely different tree that disagrees with the first on a fifth of all inputs.
Same headline number, and the fix for one is the opposite of the fix for the other. More data will help B a great deal and A almost not at all. A richer model class will help A and make B worse. If you only look at held-out error you cannot tell which situation you are in, and you will pick an intervention by guessing.
The decomposition is the diagnostic that separates them.
Statement
Fix a point $x$. Let $y = f(x) + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\mathrm{Var}(\varepsilon) = \sigma^2$. Let $\hat f_S$ be the model fitted to a random training set $S$, and take expectations over both $S$ and the noise.
Bias–variance decomposition (squared loss).
$$\mathbb{E}\big[(y - \hat f_S(x))^2\big] = \underbrace{\big(f(x) - \mathbb{E}_S[\hat f_S(x)]\big)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}_S\big[(\hat f_S(x) - \mathbb{E}_S[\hat f_S(x)])^2\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{irreducible}}$$Bias is how far the average fitted model is from the truth: error from the class being too restricted to represent $f$.
Variance is how much the fitted model moves when the training sample changes: error from the fit depending on accidents of the data.
Irreducible error is the noise in $y$ itself. No model, no amount of data, no algorithm reduces it. It is the floor.
The identity is exact, not approximate, and it holds pointwise in $x$. It is stated for squared loss, and that restriction is where most of the caveats live.
Argument
The derivation is two lines and worth seeing, because every step is just “cross terms vanish”.
Write $\bar f(x) = \mathbb{E}_S[\hat f_S(x)]$. Then
$$\mathbb{E}[(y - \hat f_S)^2] = \mathbb{E}[(f + \varepsilon - \hat f_S)^2] = \mathbb{E}[(f - \hat f_S)^2] + \mathbb{E}[\varepsilon^2] + 2\mathbb{E}[\varepsilon(f - \hat f_S)].$$The last term is zero: $\varepsilon$ is the noise on the test label, and $\hat f_S$ depends only on the training set, so they are independent and $\mathbb{E}[\varepsilon] = 0$. That leaves $\mathbb{E}[(f - \hat f_S)^2] + \sigma^2$. Now add and subtract $\bar f$:
$$\mathbb{E}[(f - \bar f + \bar f - \hat f_S)^2] = (f - \bar f)^2 + \mathbb{E}[(\bar f - \hat f_S)^2] + 2(f - \bar f)\,\mathbb{E}[\bar f - \hat f_S],$$and the cross term vanishes because $\mathbb{E}_S[\hat f_S] = \bar f$ by definition. What remains is bias$^2$ + variance + $\sigma^2$. $\blacksquare$
Now measure it, which is the part people skip. You can estimate both terms with a bootstrap: draw $B$ resamples, fit a model on each, and for each test point compute the mean prediction across the $B$ models and the spread around it. Averaged over test points, the spread is the variance term, and the gap between the mean prediction and the label is bias$^2$ plus noise.
Running this on the two trees above might give:
| bias² | variance | noise | total | |
|---|---|---|---|---|
| depth-2 tree | 0.22 | 0.02 | 0.06 | 0.30 |
| depth-30 tree | 0.02 | 0.22 | 0.06 | 0.30 |
Identical totals, opposite composition, and now the interventions are obvious. For A, deepen the tree or add features. For B, prune, regularise, get more data, or average many models — which is exactly what a random forest is, and why it works: bagging leaves bias roughly untouched and drives the variance term down by averaging, since the mean of $B$ roughly-independent estimators has variance $\approx \mathrm{Var}/B$. Boosting attacks the other term, fitting successive models to the residual, which reduces bias and can raise variance.
The relationship to VC dimension. T096’s $d$ controls a uniform bound over the class; here we decompose the error of one procedure. The two are different views of the same trade. Growing $\mathcal{H}$ can only lower the best achievable error, so bias falls; a larger $d$ loosens the uniform convergence bound, and that looseness manifests as variance. The VC bound is worst-case and one-sided; the decomposition is exact and measurable on your own data. Reach for the second when you need to decide something today.
Where the noise floor bites. $\sigma^2 = 0.06$ in the table above means no model gets below 6% here. Knowing this number changes project decisions: it is the difference between “our model is 24 points from perfect” and “our model is 24 points from perfect and 6 of those are unreachable, so stop staffing this and go improve the labels.” Estimating it usually means having some inputs labelled multiple times and measuring the disagreement, which is cheap and almost never done.
The modern complication, stated honestly. Very large models often show double descent: test error rises as capacity approaches the interpolation threshold — the point where the model exactly fits the training data — and then falls again as capacity grows further. The classic U-shaped curve is real and is the left half of a more complicated picture. The decomposition itself is still an identity and still holds. What fails is the folk claim that variance must increase monotonically with capacity; in the overparameterised regime, implicit regularisation from the optimiser appears to select low-variance solutions from among the many that fit. That mechanism is not fully understood.
Forbids
Driving expected squared error below $\sigma^2$. The noise term does not depend on the model, the algorithm, or the sample size. A reported test error below the noise floor means the floor was mis-estimated or the test set leaked.
Reducing both terms without limit inside a fixed class and sample size. They are two parts of one budget; moving along the capacity axis trades them.
Diagnosing a model from held-out error alone. Two models with identical error can need opposite treatments, as above. The single number is not sufficient information for the decision you are about to make.
Expecting bagging to fix bias. Averaging identically-biased models leaves the bias exactly where it was. If your forest underfits, more trees is not the answer.
Does not forbid
It does not say more capacity always increases test error, which is the misreading that double descent exposed. The trade-off is a trade-off along the capacity axis for classical model families in the underparameterised regime. Past the interpolation threshold, test error can fall again. The identity is still true there — it is an identity — but the shape of the curve is not what the textbook picture shows.
It does not require the model to be wrong on average to have low variance. A model can have both low bias and low variance; that is what a well-matched class with adequate data looks like. The “dilemma” is a statement about the frontier you are on, not a law that every model is broken in one of two ways.
It does not apply unmodified to 0–1 loss. For classification, the clean three-way split fails; Domingos’s unified decomposition gives an analogue where variance can be beneficial — a high-variance vote can land on the right side of the boundary more often than a biased one. Quoting squared-loss intuitions at a classifier is a real and common error.
It does not say regularisation is always right. Regularisation buys variance reduction with bias. That is a good trade only when variance is the larger term, which is why measuring first is the point of this post.
It does not describe the training error. All three terms concern expected test error at a point. A model with zero training error can have any combination of the three.
Boundary
- Squared loss only, in the clean form. Other losses need other decompositions, and they do not all behave the same way.
- Expectation over training sets is not observable. You have one training set. Bootstrap estimates approximate the expectation and inherit the bootstrap’s own biases.
- It is pointwise. Bias and variance vary across the input space, and the aggregate can hide a model that is badly biased in one region and wildly variable in another. Looking at the per-region split is often where the actual bug is.
- $\sigma^2$ is a property of your labelling process, not of nature. Better annotation guidelines change it, so “irreducible” means irreducible by modelling, not irreducible in principle.
- It is descriptive, not prescriptive. It tells you where the error is. It does not tell you which of the many bias-reducing or variance-reducing moves is cheapest for your team.
The habit to keep: when a model disappoints, do not ask whether it is underfitting or overfitting, measure it. Refit on bootstrap samples, look at the spread, and let the two numbers pick the intervention.