Predictive vs Bayesian Predictive Distribution

Bayesian Predictive Distribution

The Bayesian approach¹(left plot) shows wider uncertainty bands because it accounts for parameter uncertainty on top of data noise. The uncertainty also varies spatially - being larger where there’s less data.

$$p(t|x,\mathbf{X},\mathbf{T})=\int p(t|x,w)p(w|\mathbf{X},\mathbf{T})dw=\mathcal{N}(t|m(x),s^2(x))$$

• Data-dependent mean: $m(x)$
• Input-dependent variance: $s^2(x)$

Predictive Distribution (Maximum Likelihood)

The Predictive Distribution uses a single “best” set of parameters found in the data.

$$p(t|x,w_{ML},{\beta }_{ML})=\mathcal{N}(t|y(x,w_{ML}),{\beta }_{ML}^{-1})$$

• Fixed mean: $y(x,w_{ML})$
• Fixed variance: ${\beta }_{ML}^{-1}$ (constant for all inputs)

Key Difference Summary

Predictive Distribution (Maximum Likelihood):

• Uses fixed parameters $w_{ML}$ and ${\beta }_{ML}$ (point estimates)
• Uncertainty comes only from data noise (aleatoric uncertainty)
• Constant variance across all inputs

Bayesian Predictive Distribution:

• Integrates over all possible parameters weighted by their posterior probability
• Uncertainty comes from both data noise AND parameter uncertainty (aleatoric + epistemic)
• Input-dependent variance - uncertainty varies with location

Footnotes

1. Help with the notation: $p(t|x,X,T)$ is the probability distribution of a new, unseen output $t$, conditioned on the new input $x$ and all the evidence from the training data, $X$ and $T$. ↩