Generalized Learning Rule

Using the Gradient Descent Method the Generalized Learning Rule can be obtained as

$$\boxed{{\mathbf{w}}_{k+1}={\mathbf{w}}_k-\eta \nabla E(\mathbf{w})}$$

where,

• ${\text{w}}_{k+1}\to$Model Weight (Learned)
• $\eta \to$Learning Rate is step size taken during each iteration of the Gradient descent
• $E(\mathbf{w})\to$Error Function

$$E(\mathbf{w})=\frac{1}{2N}\sum _{n=1}^N{\left(y\left(x_n,\mathbf{w}\right)-t_n\right)}^2+\underset{\text{ (regularization)}}{\underbrace{\lambda ||\text{w}|{|}^2}}$$

• The initial position is random or a good guess is taken.
• Iteration is run until $E(\text{w})\approx 0$