Recipe

Grow and prune a decision tree

Use when: you have training data and want a tree that works well on new data, not just one that fits the training data closely

Goal: a pruned tree, with alpha chosen by cross-validation

Taught in Cost-complexity pruning

1. 1

   Run recursive binary splitting until every leaf is small (fewer than 5 samples, say) or pure.

   Example

   On a housing dataset, this might grow a tree with 40+ leaves. That is clearly overfit.

   Why: stopping too early risks missing a useful split hidden one level deeper.

2. 2

   For each alpha value, find the subtree that makes RSS + alpha times leaf count as small as possible.

   Why: this gives one candidate subtree per alpha, not just one arbitrary pruned tree.

3. 3

   Run k-fold cross-validation. Score each alpha's matching subtree on the held-out folds.

   Why: training RSS always favors the unpruned tree. Only scores on held-out data show true overfitting.

4. 4

   Pick the alpha whose subtree has the lowest cross-validated error. If a few values tie closely, pick the larger alpha and the simpler tree.

   Why: the cross-validated error is the honest estimate of how well the tree will generalize. Ties should favor the simpler tree.

5. 5

   Prune the large tree from step 1 back to the subtree that matches the chosen alpha.

   Why: this is the final, tuned model, not the oversized starting tree.