Regularization techniques for learning with matrices

Sham M. Kakade*, Shai Shalev-Shwartz, Ambuj Tewari

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

122 Scopus citations

Abstract

There is growing body of learning problems for which it is natural to organize the parameters into a matrix. As a result, it becomes easy to impose sophisticated prior knowledge by appropriately regularizing the parameters under some matrix norm. This work describes and analyzes a systematic method for constructing such matrix-based regularization techniques. In particular, we focus on how the underlying statistical properties of a given problem can help us decide which regularization function is appropriate. Our methodology is based on a known duality phenomenon: a function is strongly convex with respect to some norm if and only if its conjugate function is strongly smooth with respect to the dual norm. This result has already been found to be a key component in deriving and analyzing several learning algorithms. We demonstrate the potential of this framework by deriving novel generalization and regret bounds for multi-task learning, multi-class learning, and multiple kernel learning.

Original languageAmerican English
Pages (from-to)1865-1890
Number of pages26
JournalJournal of Machine Learning Research
Volume13
StatePublished - Jun 2012

Keywords

  • Generalization bounds
  • Multi-class learning
  • Multi-task learning
  • Multiple kernel learning
  • Regret bounds
  • Regularization
  • Strong convexity

Fingerprint

Dive into the research topics of 'Regularization techniques for learning with matrices'. Together they form a unique fingerprint.

Cite this