Regularisation

Use-case

Regularization ensures that the weights of the regression model aren’t too large. This is important because large weights usually cause over-fitting which leads to poor performance on test data. It also prevents oscillation and introduces smoothness (which is however an inductive bias in itself!)

Mathematically

The regularized error function is given by:

$$\boxed{\tilde{E}(\mathbf{w})=\stackrel{\text{Sum of Squared Error}}{\overbrace{\frac{1}{2}\sum _{n=1}^N{\left(y\left(x_n,\mathbf{w}\right)-t_n\right)}^2}}+\underset{\text{regularization}}{\underbrace{\frac{\lambda }{2}\Vert \mathbf{w}{\Vert }^2}}}$$

where,

• $\lambda \to$ balances the regularization term against the error term. It is different for each model.
• $1/2\to$ is the scaling factor that simplifies the derivative during optimization (calculating the gradient).