Kernel regression

In

In any nonparametric regression, the conditional expectation of a variable ${\displaystyle Y}$ relative to a variable ${\displaystyle X}$ may be written:

${\displaystyle \operatorname {E} (Y\mid X)=m(X)}$

where ${\displaystyle m}$ is an unknown function.

Nadaraya–Watson kernel regression

Nadaraya and Watson, both in 1964, proposed to estimate ${\displaystyle m}$ as a locally weighted average, using a kernel as a weighting function.^[1][2][3] The Nadaraya–Watson estimator is:

${\displaystyle {\widehat {m}}_{h}(x)={\frac {\sum _{i=1}^{n}K_{h}(x-x_{i})y_{i}}{\sum _{i=1}^{n}K_{h}(x-x_{i})}}}$

where ${\displaystyle K_{h}(t)={\frac {1}{h}}K\left({\frac {t}{h}}\right)}$ is a kernel with a bandwidth ${\displaystyle h}$ such that ${\displaystyle K(\cdot )}$ is of order at least 1, that is ${\displaystyle \int _{-\infty }^{\infty }uK(u)\,du=0}$.

Derivation

Starting with the definition of conditional expectation,

${\displaystyle \operatorname {E} (Y\mid X=x)=\int yf(y\mid x)\,dy=\int y{\frac {f(x,y)}{f(x)}}\,dy}$

we estimate the joint distributions f(x,y) and f(x) using kernel density estimation with a kernel K:

${\displaystyle {\hat {f}}(x,y)={\frac {1}{n}}\sum _{i=1}^{n}K_{h}(x-x_{i})K_{h}(y-y_{i}),}$

${\displaystyle {\hat {f}}(x)={\frac {1}{n}}\sum _{i=1}^{n}K_{h}(x-x_{i}),}$

We get:

${\displaystyle {\begin{aligned}\operatorname {\hat {E}} (Y\mid X=x)&=\int y{\frac {{\hat {f}}(x,y)}{{\hat {f}}(x)}}\,dy,\\[6pt]&=\int y{\frac {\sum _{i=1}^{n}K_{h}(x-x_{i})K_{h}(y-y_{i})}{\sum _{j=1}^{n}K_{h}(x-x_{j})}}\,dy,\\[6pt]&={\frac {\sum _{i=1}^{n}K_{h}(x-x_{i})\int y\,K_{h}(y-y_{i})\,dy}{\sum _{j=1}^{n}K_{h}(x-x_{j})}},\\[6pt]&={\frac {\sum _{i=1}^{n}K_{h}(x-x_{i})y_{i}}{\sum _{j=1}^{n}K_{h}(x-x_{j})}},\end{aligned}}}$

which is the Nadaraya–Watson estimator.

Priestley–Chao kernel estimator

${\displaystyle {\widehat {m}}_{PC}(x)=h^{-1}\sum _{i=2}^{n}(x_{i}-x_{i-1})K\left({\frac {x-x_{i}}{h}}\right)y_{i}}$

where ${\displaystyle h}$ is the bandwidth (or smoothing parameter).

Gasser–Müller kernel estimator

${\displaystyle {\widehat {m}}_{GM}(x)=h^{-1}\sum _{i=1}^{n}\left[\int _{s_{i-1}}^{s_{i}}K\left({\frac {x-u}{h}}\right)\,du\right]y_{i}}$

where ${\displaystyle s_{i}={\frac {x_{i-1}+x_{i}}{2}}.}$^[4]

Example

This example is based upon Canadian cross-section wage data consisting of a random sample taken from the 1971 Canadian Census Public Use Tapes for male individuals having common education (grade 13). There are 205 observations in total.^{[citation needed]}

The figure to the right shows the estimated regression function using a second order Gaussian kernel along with asymptotic variability bounds.

Script for example

The following commands of the

install.packages("np")
library(np) # non parametric library
data(cps71)
attach(cps71)

m <- npreg(logwage~age)

plot(m, plot.errors.method="asymptotic",
     plot.errors.style="band",
     ylim=c(11, 15.2))

points(age, logwage, cex=.25)
detach(cps71)

Related

According to

• GNU Octave mathematical program package
• Julia: KernelEstimator.jl
• MATLAB: A free MATLAB toolbox with implementation of kernel regression, kernel density estimation, kernel estimation of hazard function and many others is available on these pages (this toolbox is a part of the book ^[6]).
• Python: the KernelReg class for mixed data types in the statsmodels.nonparametric sub-package (includes other kernel density related classes), the package kernel_regression as an extension of scikit-learn (inefficient memory-wise, useful only for small datasets)
• R: the function npreg of the np package can perform kernel regression.^[7][8]
• Stata: npregress, kernreg2

See also

• Kernel smoother
• Local regression

References