Locally Weighted Linear Regression (LWR) is a non-parametric
regression technique designed to address the limitations of traditional linear
regression, especially when the data relationship is not well modeled by a
simple global linear function.
- Basic
Idea: Instead of fitting a single global linear model, LWR
fits a linear model locally around the query point x. It places
more weight on training examples close to x, and less weight on examples
farther away.
- Weighting
Scheme: Each training example (x(i), y(i))
is assigned a weight w(i) based on its distance from the query point x,
typically using a kernel function like the Gaussian:
w(i)=exp(−2τ2∥x(i)−x∥2),
where τ is a bandwidth parameter controlling how
quickly the weight decreases with distance.
- Fitting
and Prediction: To predict y at x, LWR:
- Solves
a weighted least squares problem minimizing:
∑iw(i)(y(i)−θTx(i))2,
where each data point's contribution is scaled by its
weight.
- Uses
the fitted parameters θ to output the prediction:
y^=θTx.
- Non-Parametric
Nature: Unlike standard linear regression
that produces a single set of parameters θ, LWR adapts parameters locally
for each query point. It requires retaining the entire training set for
prediction, making it a non-parametric method.
Advantages:
- Can
handle complex, non-linear relationships without explicitly defining a
global model.
- Makes
prediction sensitive to the local structure of data.
- Reduces
the dependency on carefully selecting features.
Considerations:
- Choosing
the bandwidth τ is critical; too small leads to high variance
(overfitting), too large leads to high bias (underfitting).
- Computationally
expensive for large datasets since it fits a model for each query point.
This method smooths between fitting the data globally and
simply using nearest neighbor predictions, providing a flexible approach to
regression when data relationships vary locally
Comments
Post a Comment