Using Likelihood to Model Probability

Likelihood function

The likelihood function measures how likely it to observe an output for a give set of parameters to a model. Mathematically it is represented through the conditional probability.

$$p(t_0|x_0,w,\beta )$$

where,

• $p\to$ PDF
• $t_0\to$ an output
• $x_0\to$ an input
• $\text{w}\to$ model parameter. Example: In a linear regression model relating $x$ and $t$

$$t\approx P(x)=w_0+w_{1}x$$

Here, $\mathbf{w}=\left[w_0,w_1\right]$ are the parameters of the model. $w_0$ is the intercept, and $w_1$ is the slope.

• $\beta \to$ precision, is the inverse of the variance (${\sigma }^2$).
  Note: High precision means low variance and vice versa.

Assumption

The probability is assumed to be Gaussian distributed with mean on the model and variance ${\beta }^{-1}={\sigma }^2$ (for the 1-dimensional case)¹

$$p(t_0|x_0,w,\beta )=\mathcal{N}(t_0|y(w,x_0),{\beta }^{-1})$$

where,

• $y\to$ is the regression model used.

Measurement to Stochastic Model

For a given parametric model $y(x,\mathbf{w})$ and target $t$. The noise can modelled by the Gaussian Distribution $\mathcal{N}$.

$$t=y(x,\mathbf{w})+\mathcal{N}\left(0,{\beta }^{-1}\right)$$

The Mean is $0$ because we center the Gaussian around the predicted value.

Since we have a distribution (Probabilistic), the equality is removed.

$$t-y(x,\mathbf{w})\sim \mathcal{N}\left(0,{\beta }^{-1}\right)$$

Finally we get

$$\boxed{t\sim \mathcal{N}\left(y(x,\mathbf{w}),{\beta }^{-1}\right)}$$

Applying this to the whole Data-set

We define the Likelihood function (Data-Likelihood) and express it as a joint density function where the stochasity at each target value is expressed through the Gaussian.

$$\boxed{L(\mathbf{w})=P(T\mid X,\mathbf{w},\beta )=\prod _{n=1}^N\mathcal{N}\left(t_n\mid y\left(x_n\mathbf{w}\right),{\beta }^{-1}\right)}$$

Note:

• We assume the data points to be independent. Inductive bias!
• The Joint Density of a Independent values is the product of the densities of individual data.

Parameter Optimisation

Involves maximising the Likelihood for the given parameters i.e. we maximise the probability of observing the measured output ($t$) for a given model parameter ($\text{w}$) and input ($x$)

$${\omega }_{ML}={\omega }^{\ast }=\arg \underset{\omega }{\max }L(\omega )$$

Note: the actual value of $L(\mathbf{w})$ at the maximum is not important!

Generalisation

Use the parameters ${\mathbf{w}}_{ML}$ & ${\beta }_{ML}$ then the optimal output distribution is Gaussian:

$$p(t\mid x,{\mathbf{w}}_{ML},{\beta }_{ML})=\mathcal{N}(t\mid y(x,{\mathbf{w}}_{ML}),{\beta }_{ML}^{-1})$$

Footnotes

1. Pattern Recognition Bishop Pg. 29 ↩