Neural networks have recently re-emerged as a powerful hypothesis class, yielding impressive classification accuracy in multiple domains. However, their training is a non-convex optimization problem which poses theoretical and practical challenges. Here we address this difficulty by turning to “improper” learning of neural nets. In other words, we learn a classifier that is not a neural net but is competitive with the best neural net model given a sufficient number of training examples. Our approach relies on a novel kernel construction scheme in which the kernel is a result of integration over the set of all possible instantiation of neural models. It turns out that the corresponding integral can be evaluated in closed-form via a simple recursion. Thus we translate the non-convex learning problem of a neural net to an SVM with an appropriate kernel. We also provide sample complexity results which depend on the stability of the optimal neural net.
|Original language||American English|
|Number of pages||9|
|State||Published - 2016|
|Event||19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016 - Cadiz, Spain|
Duration: 9 May 2016 → 11 May 2016
|Conference||19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016|
|Period||9/05/16 → 11/05/16|
Bibliographical noteFunding Information:
Acknowledgments: This work was supported by the ISF Centers of Excellence grant 1789/11, by the Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI), and by a Google Research Award. Roi Livni is a recipient of the Google EuropeFellowship inLearningTheory, andthisresearchis supported in part by this Fellowship.
Copyright 2016 by the authors.