Tensors for deep learning theory: Analyzing deep learning architectures via tensorization

Deep learning architectures have enabled unprecedented advances in a wide range of artificial intelligence-related applications. The empirical success of these architectures has posed fundamental riddles regarding their operation in the front lines of modern theoretical machine learning research. Related theoretical efforts can be broadly divided into (i) explaining the observed success of deep learning architectures and (ii) harnessing these insights for improving their operation. In this chapter, we outline a tensor analysis-based contribution to understanding and improving the expressivity of prominent deep learning architecture classes. We detail a successful proof methodology which includes analyzing grid tensors of the functions realized by deep learning architecture classes, which was applied for convolutional, recurrent, and self-attention networks. The rank of an architecture’s grid tensor is used for bounding the input dependencies that can be modeled by the architecture and for establishing superiority of one architectural configuration over the other. We demonstrate how the above methodology has promoted the understanding of the architectures’ operations and consequently led to their practical improvements.