Model descriptions¶
This package implements a number of models.
Ridge¶
Let
The equation has a closed-form solution
This model is implemented in
Ridge
(scikit-learn-compatible estimator)solve_ridge_svd()
(function)
KernelRidge¶
By the Woodbury matrix identity,
This model can be extended to arbitrary positive semidefinite kernels
This model is implemented in
KernelRidge
(scikit-learn-compatible estimator)solve_kernel_ridge_eigenvalues()
(function)solve_kernel_ridge_gradient_descent()
(function)solve_kernel_ridge_conjugate_gradient()
(function)
RidgeCV and KernelRidgeCV¶
In practice, because the ridge regression and kernel ridge regression
hyperparameter
These models are implemented in
RidgeCV
(scikit-learn-compatible estimator)solve_ridge_cv_svd()
(function)KernelRidgeCV
(scikit-learn-compatible estimator)solve_kernel_ridge_cv_eigenvalues()
(function)
GroupRidgeCV / BandedRidgeCV¶
In some applications, features are naturally grouped into groups (or feature spaces). To adapt the regularization level to each feature space, ridge regression can be extended to group-regularized ridge regression (also known as banded ridge regression 3). In this model, a separate hyperparameter is optimized for each feature space:
This is equivalent to solving a ridge regression:
where the feature space
This model is implemented in
GroupRidgeCV
(scikit-learn-compatible estimator)solve_group_ridge_random_search()
(function)
See also multiple-kernel ridge regression, which is equivalent to group-regularization ridge regression when using one linear kernel per group of features:
MultipleKernelRidgeCV
(scikit-learn-compatible estimator)solve_multiple_kernel_ridge_random_search()
(function)solve_multiple_kernel_ridge_hyper_gradient()
(function)
Note
“Group ridge regression” is also sometimes called “Banded ridge regression”.
WeightedKernelRidge¶
To extend kernel ridge to group-regularization, we can compute the kernel as a
weighted sum of multiple kernels,
This model is called weighted kernel ridge regresion. The log-kernel-weights
This model is implemented in
WeightedKernelRidge
(scikit-learn-compatible estimator)solve_weighted_kernel_ridge_neumann_series()
(function)
MultipleKernelRidgeCV¶
In weighted kernel ridge regression, when the log-kernel-weights
This model is implemented in
MultipleKernelRidgeCV
(scikit-learn-compatible estimator)solve_multiple_kernel_ridge_hyper_gradient()
(function)solve_multiple_kernel_ridge_random_search()
(function)
Model flowchart¶
The following flowchart can be used as a guide to select the right estimator.
References¶
- 1
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55-67.
- 2
Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables.
- 3
Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019). Voxelwise encoding models with non-spherical multivariate normal priors. Neuroimage, 197, 482-492.