Overfitting occurs when the model incorporates the random error as well as the relationship. In practice, the model performs well on the training set but poorly on new data. Overfitting often occurs when the model capacity is too high. For example, a polynomial of the same degree as the degrees of freedom will perfectly include each point but will be very poor at prediction. During model fitting, training set accuracy will (almost) always increase. Overfitting can be seen by monitoring the validation set accuracy. Validation set accuracy will follow a U-shape, where it initially decreases and subsequently increases as overfitting occurs. Options for avoiding overfitting include - reduce the model capacity (e.g., number of parameters) - increase the dataset size (via data augmentation or multitask training) - [[regularization]]: add a penalty - weight constraints: constrain the magnitude of the model weights - [[dropout]]: randomly drop units from a neural network - [[early stopping]]: stop training when the validation set accuracy begins to decrease