Parameter Estimation for Maximum Likelihood (Multivariate Gausssian)

Maximum Likelihood (ML) Parameter Estimation

The maximum likelihood estimate finds parameters $\theta$ that maximize the likelihood of observing the training data:

$${\hat{\mathbf{\theta }}}^{ML}=\arg \underset{\mathbf{\theta }}{\max }L(\mathbf{\theta })=\arg \underset{\mathbf{\theta }}{\max }p(O|\mathbf{\theta })$$

In practice, we maximize the log-likelihood $\mathrm{LL}(\mathbf{\theta })$

$${\hat{\mathbf{\theta }}}^{(\mathrm{ML})}=\arg \underset{\mathbf{\theta }}{\max }\mathrm{LL}(\mathbf{\theta })=\arg \underset{\theta }{\max }\sum _{\tau =1}^T\log \mathrm{p}\left(\mathbf{x}={\mathbf{o}}_{\tau }\mid \mathbf{\theta }\right)$$

To find the parameters that maximize the likelihood of observing the training data, one must find the maximum of the log-likelihood function, $LL(\theta )$. This is typically done by taking the partial derivative with respect to each parameter, setting it to zero, and solving the resulting system of equations.

The formula is:

$$\frac{\partial }{\partial {\theta }_r}LL(\theta )=\sum _{\tau =1}^T\frac{\partial }{\partial {\theta }_r}\log p(x=o_{\tau }|\theta )\stackrel{!}{=}0,\text{for }r=1,2,...,R$$

• ${\hat{\theta }}^{(ML)}$: The parameter vector that maximizes the likelihood.
• $LL(\theta )$: The log-likelihood function.
• $o_{\tau }$: The training data vectors.
• ${\theta }_r$: The individual parameters to be estimated.
• $T$: Total training samples in the dataset
• $T$: Total parameters in the model (that need to be estimated)

Note: $\stackrel{!}{=}0$ This notation means “set equal to zero”. We set the derivative to zero to find the critical points (maxima, minima, or saddle points) of the log-likelihood function. In this context, we are looking for the maximum.

---

Why Parameter Estimation is Needed

In pattern recognition, we need parameter estimation to train classifiers by determining the unknown parameters of probability distributions from training data. They are:

• Prior probabilities $P(s=i)$
• Likelihood probability density functions $p(\mathbf{x}|s=i)$

Assuming that the Likelihood can by represented by a Gaussian, we find mean ${\mu }_i$ and covariance ${\Sigma }_i$ such that it closely represents the Likelihood. The ML estimates for the parameters are estimated as follows:

• Sample Mean (unbiased): This is the average of all the training vectors.

  $$\hat{\mu }=\frac{1}{T}\sum _{\tau =1}^T{o}_{\tau }$$

• Sample Covariance (unbiased): This is the average of the outer products of the centered data vectors.

  $$\hat{\Sigma }=\frac{1}{T-1}\sum _{\tau =1}^T(o_{\tau }-\hat{\mu })(o_{\tau }-\hat{\mu }{)}^T$$

  Note: Using a denominator of T-1 provides an unbiased estimate.