MAP = Regularized Least Squares

MAP solution

A MAP estimation with a Gaussian prior on the weights and a Gaussian likelihood on the data is equivalent to minimizing the sum of squared errors with L2 regularization.

$$\text{MAP with Gaussian prior}=\text{Minimizing squared error + regularization}$$

Formula:

$$w_{MAP}=\arg \underset{w}{\max }P(w|D)=\arg \underset{w}{\min }\left[\underset{\text{from Likelihood (Squared Error) }}{\underbrace{\frac{1}{2}\sum _n{\left(t_n-w^T\varphi \left(x_n\right)\right)}^2}}+\underset{\text{from Prior (L2 Regularization) }}{\underbrace{\frac{\lambda }{2}\Vert w{\Vert }^2}}\right]$$

Key Insight:

The regularization strength $\lambda$ is inversely proportional to the variance of the Gaussian prior $\left({\sigma }_p^2\right)$ :

$$\lambda =\frac{1}{{\sigma }_p^2}$$

• Strong Regularization (large $\lambda$ ) $\leftrightarrow$ Small Prior Variance (weights are assumed to be near zero).
• Weak Regularization (small $\lambda$ ) $\leftrightarrow$ Large Prior Variance (weights are allowed to vary more).

Key Point: The model is linear in parameters $\mathbf{w}$ but can be non-linear in inputs $x$ through the choice of basis functions $\varphi (x)$.