Gradient Descent Algorithm

Use-case

It is used to find the smallest cost function for a regression. So that the regression gives a fairly accurate approximation of the data set.

$$\boxed{{\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J\left({\theta }_0,{\theta }_1\right)\text{ (for }j=0,j=1\text{ )}}$$

Theory

Hypothesis Function

$$h_{\theta }(x)={\theta }_{1}x+{\theta }_0$$

Where,

• $({\theta }_0,{\theta }_1)\to$ Parameters guessed by the Hypothesis function through regression
• $x\to$ Domain

Example

Cost/Loss Function: Mean Squared Error

$$J\left({\theta }_0,{\theta }_1\right)=\frac{1}{m}\sum _{i=1}^m\left[h_{\theta }{\left(x(i)-y(i)\right)}^2\right]$$

Where, $m\to$ Number of data points.

The cost function is computed for all the guess values can be visualized as follows. Visually, the point with the lowest cost function can be identified. However, it is not feasible to map the cost function to every possible guess value of ${\theta }_1,{\theta }_0$. Gradient algorithm is then used to find a path to a minima of the cost function.

$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J\left({\theta }_0,{\theta }_1\right)\text{ (for }j=0,j=1\text{ ) }$$

This gives the smallest value of ${\theta }_j$ for a given $\alpha$ because we know that the gradient points to the direction of the steepest slope. Where $\alpha \to$ step size $\to$ learning rate

$$\begin{array}{rl}& {\theta }_0:={\theta }_0+\nabla {\theta }_0\to -\alpha \frac{\partial }{\partial {\theta }_0}J\left({\theta }_0,{\theta }_1\right)\\ & {\theta }_1:={\theta }_1+\nabla {\theta }_1\to -\alpha \frac{\partial }{\partial {\theta }_1}J\left({\theta }_0,{\theta }_1\right)\end{array}$$

Repeat until convergence