Locally Weighted Linear Regression (LWR) is a non-parametric
regression technique designed to address the limitations of traditional linear
regression, especially when the data relationship is not well modeled by a
simple global linear function.
- Basic
     Idea: Instead of fitting a single global linear model, LWR
     fits a linear model locally around the query point x. It places
     more weight on training examples close to x, and less weight on examples
     farther away.
- Weighting
     Scheme: Each training example (x(i), y(i))
     is assigned a weight w(i) based on its distance from the query point x,
     typically using a kernel function like the Gaussian:
w(i)=exp(−2τ2∥x(i)−x∥2),
where τ is a bandwidth parameter controlling how
quickly the weight decreases with distance.
- Fitting
     and Prediction: To predict y at x, LWR:
- Solves
     a weighted least squares problem minimizing:
∑iw(i)(y(i)−θTx(i))2,
where each data point's contribution is scaled by its
weight.
- Uses
     the fitted parameters θ to output the prediction:
y^=θTx.
- Non-Parametric
     Nature: Unlike standard linear regression
     that produces a single set of parameters θ, LWR adapts parameters locally
     for each query point. It requires retaining the entire training set for
     prediction, making it a non-parametric method.
Advantages:
- Can
     handle complex, non-linear relationships without explicitly defining a
     global model.
- Makes
     prediction sensitive to the local structure of data.
- Reduces
     the dependency on carefully selecting features.
Considerations:
- Choosing
     the bandwidth τ is critical; too small leads to high variance
     (overfitting), too large leads to high bias (underfitting).
- Computationally
     expensive for large datasets since it fits a model for each query point.
This method smooths between fitting the data globally and
simply using nearest neighbor predictions, providing a flexible approach to
regression when data relationships vary locally
 

Comments
Post a Comment