The unified model is a mathematical framework showing that many regression algorithms can be expressed in a single, general form.

Mathematical Formulation

Components

  • : Basis functions (typically Gaussian/RBF)
  • : Weight vectors for local linear models
  • : Bias/offset terms
  • : Number of local models
  • : Parameters of basis functions (e.g., mean, variance)

Deconstructing the Model

For any given , the model calculates which expert is most relevant and gives its output more weight in the final sum.

Prediction from Expert This is the simple linear model ( ). Each of the “experts” is a linear model with its own weight vector and bias .

Weight for Expert This is the gating function . It calculates a value based on the input . Typically, these gating functions are designed to be high for some regions of the input space and low for others.

Relating Models to the Unified Model

1. Simple Linear Regression

  • Setting: , (constant everywhere)
  • Result:
  • Interpretation: One global linear model, no local weighting

2. RBF (Radial Basis Function) Networks

  • Setting: (zero weight vectors), keep only
  • Result:
  • Interpretation: Weighted sum of constants, each Gaussian holds one value

3. Weighted Regression (WR)

  • Setting: , but varies with input
  • Result:
  • Interpretation: Single linear model weighted by input-dependent function

4. Locally Weighted Regression (LWR)

  • Setting: Full unified model with all parameters
  • Result:
  • Interpretation: Multiple local linear models, each weighted by Gaussian

Key Relationships:

  • RBF ⊂ LWR: RBF is special case of LWR with
  • Linear Regression ⊂ LWR: Linear regression is LWR with ,
  • LWR = Most General: Can represent all others by parameter choices

Hierarchy of complexity

Linear Regression → RBF → LWR