Expression of fractals through neural network functions

Nadav Dym, Barak Sober*, Ingrid Daubechies

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

6 Scopus citations

Abstract

To help understand the underlying mechanisms of neural networks (NNs), several groups have studied the number of linear regions l of piecewise linear (PwL) functions, generated by deep neural networks (DNN). In particular, they showed that l can grow exponentially with the number of network parameters p, a property often used to explain the advantages of deep over shallow NNs. Nonetheless, a dimension argument shows that DNNs cannot generate all PwL functions with l linear regions when l > p. It is thus natural to seek to characterize specific families of functions with l > p linear regions that can be constructed by DNNs. Iterated Function Systems (IFS) recursively construct a sequence of PwL functions Fk with a number of linear regions which is exponential in k. We show that Fk can be generated by a NN using only O(k) parameters. IFS are used extensively to generate natural-looking landscape textures in artificial images as well as for compression of natural images. The surprisingly good performance of this compression suggests that human visual system may lock in on self-similarities. The combination of this phenomenon with the capacity of DNNs to efficiently approximate IFS may contribute to the success of DNNs, particularly striking for image processing tasks.

Original languageAmerican English
Article number2991422
Pages (from-to)57-66
Number of pages10
JournalIEEE Journal on Selected Areas in Information Theory
Volume1
Issue number1
DOIs
StatePublished - May 2020
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2020 IEEE.

Keywords

  • Fractals
  • Iterated function systems
  • Neural network functions
  • Neural networks expressive power

Fingerprint

Dive into the research topics of 'Expression of fractals through neural network functions'. Together they form a unique fingerprint.

Cite this