Keywords: polynomial regression
Extending Linear Regression with Features
The original linear regression is in the form:
where the input vector and parameter are -dimension vectors whose first components are and bias respectively. This equation is linear for both the input vector and parameter vector. Then an idea come to us, if we set then equation (1) convert to:
where and the function with input , is called feature.
This feature function was used widely, especially in reducing dimensions of original input(such as in image processing) and increasing the flexibility of the predictor(such as in extending linear regression to polynomial regression).
When we set the feature as:
the linear regression converts to:
However, the estimation of the parameter is not changed by the extension of the feature function. Because in the least-squares or other optimization algorithms the variables or random variables are , and we do not care about the change of input space. And when we using the algorithm described in ‘least squares estimation’:
to estimate the parameter, we got:
To the same task in the ‘least squares estimation’, regression of the weights of the newborn baby with days is like:
The linear regression result of a male baby is :
And code of the least square polynomial regression with power is
def fit_polynomial(self, x, y, d):
The entire project can be found The entire project can be found https://github.com/Tony-Tan/ML and please star me.
And the result of the regression is:
The blue regression line looks pretty well comparing to the right line.
Bishop, Christopher M. Pattern recognition and machine learning. springer, 2006. ↩︎