Optimal architectures in a solvable model of deep networks

Jonathan Kadmon, Haim Sompolinsky

Research output: Contribution to journalConference articlepeer-review

24 Scopus citations

Abstract

Deep neural networks have received a considerable attention due to the success of their training for real world machine learning applications. They are also of great interest to the understanding of sensory processing in cortical sensory hierarchies. The purpose of this work is to advance our theoretical understanding of the computational benefits of these architectures. Using a simple model of clustered noisy inputs and a simple learning rule, we provide analytically derived recursion relations describing the propagation of the signals along the deep network. By analysis of these equations, and defining performance measures, we show that these model networks have optimal depths. We further explore the dependence of the optimal architecture on the system parameters.

Original languageAmerican English
Pages (from-to)4788-4796
Number of pages9
JournalAdvances in Neural Information Processing Systems
StatePublished - 2016
Event30th Annual Conference on Neural Information Processing Systems, NIPS 2016 - Barcelona, Spain
Duration: 5 Dec 201610 Dec 2016

Bibliographical note

Publisher Copyright:
© 2016 NIPS Foundation - All Rights Reserved.

Fingerprint

Dive into the research topics of 'Optimal architectures in a solvable model of deep networks'. Together they form a unique fingerprint.

Cite this