Finding Optimum Parameters for Linear Learning Model

For a linear model $y(x,w)=w^T\varphi (x)$ with squared error function:

$$E(w)=\frac{1}{2}\sum _{n=1}^N{\left(t_n-w^T\varphi \left(x_n\right)\right)}^2$$

The optimal parameters $w^{\ast }$ are found by:

1. Setting the gradient to zero: $\nabla E(w)=0$
2. Solving the resulting equation

Simplification Write in matrix form (where $\Phi$ is the Design matrix)

$$E(w)=\frac{1}{2}\Vert t-\Phi w{\Vert }^2$$

Setting Gradient to zero

$${\nabla }_{w}E(w)=-{\Phi }^T(t-\Phi w)=0$$

Solve for $w$:

$${\Phi }^T\Phi w={\Phi }^{T}t$$

The optimal parameter vector is:

$$w^{\ast }={\left({\Phi }^T\Phi \right)}^{-1}{\Phi }^{T}t$$

For the regularized case (MAP estimation):

$$w_{MAP}={\left(\lambda I+{\Phi }^T\Phi \right)}^{-1}{\Phi }^{T}t$$

Where $\lambda$ is the regularization parameter.