Sparse regression as a sparse Eigenvalue problem

Baback Moghaddam*, Amit Gruber, Yair Weiss, Shai Avidan

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

5 Scopus citations

Abstract

We extend the ℓ0-norm "subspectral". algorithms developed for sparse-LDA [5] and sparse-PCA [6] to more general quadratic costs such as MSE in linear (or kernel) regression. The resulting "Sparse Least Squares" (SLS) problem is also NP-hard, by way of its equivalence to a rank-1 sparse eigenvalue problem. Specifically, for minimizing general quadratic cost functions we use a highly-efficient method for direct eigenvalue computation based on partitioned matrix inverse techniques that leads to × 103 speed-ups over standard eigenvalue decomposition. This increased efficiency mitigates the O(n4) complexity that limited the previous algorithms' utility for high-dimensional problems. Moreover, the new computation prioritizes the role of the less-myopic backward elimination stage which becomes even more efficient than forward selection. Similarly, branch-and-bound search for Exact Sparse Least Squares (ESLS) also benefits from partitioned matrix techniques. Our Greedy Sparse Least Squares (GSLS) algorithm generalizes Natarajan's algorithm [9] also known as Order-Recursive Matching Pursuit (ORMP). Specifically, the forward pass of GSLS is exactly equivalent to ORMP but is more efficient, and by including the backward pass, which only doubles the computation, we can achieve a lower MSE than ORMP. In experimental comparisons with LARS [3], forward-GSLS is shown to be not only more efficient and accurate but more flexible in terms of choice of regularizaron.

Original languageEnglish
Title of host publication2008 Information Theory and Applications Workshop - Conference Proceedings, ITA
Pages121-127
Number of pages7
DOIs
StatePublished - 2008
Event2008 Information Theory and Applications Workshop - ITA - San Diego, CA, United States
Duration: 27 Jan 20081 Feb 2008

Publication series

Name2008 Information Theory and Applications Workshop - Conference Proceedings, ITA

Conference

Conference2008 Information Theory and Applications Workshop - ITA
Country/TerritoryUnited States
CitySan Diego, CA
Period27/01/081/02/08

Fingerprint

Dive into the research topics of 'Sparse regression as a sparse Eigenvalue problem'. Together they form a unique fingerprint.

Cite this