https://piazza.com/class_profile/get_resource/ln18bjs43q41tr/ln18nhgamp78

Decision Trees

Ensembling

Random Forest

Example

• x: 54-d vector, y: vector of class

• we know training will reach 0%, but once test error stops decreasing we have to stop
• 1 tree: 12.6% test error

• Decision tree was just our first weak classifier
• solution? boosting to increase margin by increasing number of trees

• each individual tree is pretty bad, but ensembling them gives us a good majority vote
• we just need a diversity of trees s.t. it doesn’t make sense to optimize just one tree