Regularization and Smoothing
High dimension problems
As we once mentioned in EML(in chapter “Regulations and Generalization”), when the number of parameters grow bigger, we get a mode with smaller bias yet greater variance. However, we’ve never stated this in a clear and well-formulated manner, we are only describing what we observed. Now we want to make a clearer statement about these problems.
Consider the linear regression for simplicity, suggest we want to fit a model where
And we know that the norm solution of this problem is
But if , which means we have more features than samples, then we are in trouble for now the the matrix cannot be invertible, so the expression above isn’t even well-defined! Moreover, the least squares optimization problem does not have a unique solution in this case. Indeed, as you may already know, if is a unique solution to the problem, then then any other vector of the form
is also a solution of the problem, thus we can generate infinitely many others!
Furthermore, we can always take the one least squares solution to be:
Here denotes the generalized inverse (also called the Moore-Penrose pseudoinverse) of a matrix
. The special property of this solutio is that it holds the smallest norm .
This ambiguity of the solution causes us a lot troubles. First, given the uncertainty of the solution, which is the parameters we want, we can’t make any interpretations about these coefficients anymore.
The second trouble is that the barrier(check out EML in chapter “Regulations and Generalization” for more) is generally disastrous for least squares prediction. If and is the least squares prediction at a new predictor value (for the usual least squares coefficients), then under fairly standard conditions for regression theory, the prediction MSE behaves as:
What do we notice? This explodes as approaches . Big trouble!
But also remind of what we’ve mentioned in EML, this isn’t always true as the parameters gets big enough.
Regulations
So to deal with the problems we mentioned in the previous chapter, we want find some ways to regulate our models parameters within a acceptable range, and gladly, we’ve already mentioned all these regulation methods in EML so you can go and check it yourself.
Smoothing
In EML, we once mentioned Laplace Smoothing where we used it to solve the denominator we encountered in Naive Bayes. However, the smoothing we want in time series analysis is quite different from Laplace Smoothing, it’s more alike to convolution where we try to make discrete points continous curves.
Recall the signal plus noise model from our earlier lectures,
where are white noise errors, and denote the unknown means that we want to estimate, assumed to be smoothly varying across the index .
Two broad classes of smoothers of interest are as follow. The first are linear filters, which are (weighted) local averages, of the form
The second are penalized least squares smoothers, which are given by solving
for some penalty function and tuning parameter .
Linear filters
A moving average (MA), either of centered or trailing type, is one of the basic and widely-used examples of a linear filter. As we mentioned before, we have both centered MA smoothers and trailing MA smoothers. However, we can also get smoother-looking estimates by using a smoother weight sequence. This is precisely what kernel smoothing does: it uses ( is the window size) and
for a particular kernel function and bandwidth . Think of the bandwidth as a tuning parameter, just like the regularization level in ridge or lasso. Note that a centered MA is actually kernel smoothing with a “boxcar” kernel: .


Hodrick-Prescott filter
The Hodrick-Prescott filter, or simply HP filter, is a penalized estimator, defined by
We can see that the penalty is squared norm of the terms . Each of these are a second difference of , centered at a particular index.
Like the ridge problem, which is similar in spirit (but aimed at solving a different problem), the HP filter problem has an exact solution. First define the second difference matrix by,
and then rewrite the HP filter optimization problem as
The HP filter solution can be derived by differentiating the criterion in, setting it equal to zero, and solving, which yields
The HP filter has a computational advantage a linear filter with smooth weights, such as the Gaussian kernel smoother (the HP filter can be computed in linear-time because is banded). In terms of their qualitative behavior, the HP filter and a Gaussian kernel smoother are pretty similar. In fact, though it not obvious at all, the HP filter admits an equivalent form as a kernel smoother, for a special implicit kernel.
The HP filter is worth knowing about because it is quite a popular tool in macroeconomics, where it is primarily used for time series decomposition: it is used to estimate a trend component in a time series, and then the residuals from this are used to estimate a cyclic component. However, recently, some econometricians have criticized this practice, on the basis that it can induce spurious correlations in the residuals, along with other concerns.

Trend filter
The trend filter, or simply trend filter, is another penalized estimator, defined by
Like before, this could be rewrite as
Swapping the squared penalty with an penalty on second differences results in a very different behavior in the estimator. Think of the analogy of ridge versus the lasso. Trend filtering yields estimates that are sparse in second differences: if we denote by the solution vector, then many components of will be exactly equal to zero, and a larger choice of will generally result in more zeros.
Given
this means that many components will lie on the line defined by their neighbors.
Or in other words, the trend filter solution will have a piecewise linear structure, with a kink at each point such that . These kink points (also called knot points) are chosen adaptively, based on the data.

To tune any one of the smoothers described above, we can use cross-validation. We define (say) 5 folds using every 5th point in the sequence. That is, we leave out every 5th point, fit the smoother on the remaining points, over a grid of tuning parameter values, and calculate the squared error on the held-out points. Doing this 4 more times, where each time we shift the indices of the held-out points forward by one, we will be able to compute the average error over all points, and use this to select the tuning parameter value.
Note that trend filtering can be extended to model a piecewise polynomial of an arbitrary degree, not just linear; as in the above, the knots in this piecewise polynomial will be chosen adaptively based on the data. So it can generate smoother-looking fits, like the HP filter or kernel smoothing, but the key difference is that it has a property called local adaptivity, which comes from its ability to select knots based on the data.


