Gaussian Mixture Regression (GMR)

Using GMM for Regression

Gaussian Mixture Regression (GMR) uses a GMM to perform Regression. The process is:

1. Model the Joint Space: Combine the input and output data into a single vector, $z=(y,x)$. Train a GMM on this joint data space to learn the distribution $p(z)$.

2. Conditioning: For a new input query $x_q$, calculate the conditional probability distribution $p(y|x_q)$ from the learned joint GMM. This conditional distribution is itself a Gaussian.

3. Predict: The regression output for $x_q$ is the expected value (mean) of this conditional distribution. The analytical solution for the predicted mean $\bar{y}$ is:

   $$\overline{y}=\sum _{e=1}^E{h}_e(x)({\mu }_{e,Y}+{\Sigma }_{e,YX}{\Sigma }_{e,X}^{-1}(x-{\mu }_{e,X}))$$

   where $h_e(x)$ are weights corresponding to the responsibilities.

GMR has the same mathematical form as Locally Weighted Regression (LWR) and is considered a special case of a broader unified model for regression. The unified model is expressed as:

$$t(x)=\sum _{e=1}^E{\varphi }_c(x,{\theta }_e)(w_e^{T}x+b_e)$$

GMR fits into this framework, demonstrating its connection to a wide family of regression algorithms, including those based on mixtures of linear models and weighted sums of basis functions.