Parameter Estimation for Maximum Likelihood

Use-case: Why We Need Parameter Estimation

In pattern recognition, we need parameter estimation to train classifiers by determining the unknown parameters of probability distributions from training data. Specifically, we need to estimate:

• Prior probabilities $P(s=i)$
• Likelihood probability density functions $p(\mathbf{x}|s=i)$

Maximum Likelihood (ML) Parameter Estimation

The maximum likelihood estimate finds parameters $w$ that maximize the likelihood of observing the training data:

$${\hat{\mathbf{w}}}_{ML}=\arg \underset{\mathbf{w}}{\max }L(\mathbf{w})=\arg \underset{\mathbf{w}}{\max }\prod _{n=1}^{N}p({\mathbf{x}}_n|\mathbf{w})$$

In practice, we maximize the log-likelihood:

$${\hat{\mathbf{w}}}_{ML}=\arg \underset{\mathbf{w}}{\max }\sum _{n=1}^N\log p({\mathbf{x}}_n|\mathbf{w})$$

For a multivariate Gaussian distribution, the ML estimates are:

• Mean: $\hat{\mathbf{\mu }}=\frac{1}{N}{\sum }_{n=1}^N{\mathbf{x}}_n$ (sample mean)
• Covariance: $\hat{\mathbf{\Sigma }}=\frac{1}{N}{\sum }_{n=1}^N({\mathbf{x}}_n-\hat{\mathbf{\mu }})({\mathbf{x}}_n-\hat{\mathbf{\mu }}{)}^T$ (sample covariance)