Pruning and Slicing Neural Networks using Formal Verification

Ori Lahav, Guy Katz

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

11 Scopus citations

Abstract

Deep neural networks (DNNs) play an increasingly important role in various computer systems. In order to create these networks, engineers typically specify a desired topology, and then use an automated training algorithm to select the network's weights. While training algorithms have been studied extensively and are well understood, the selection of topology remains a form of art, and can often result in networks that are unnecessarily large - and consequently are incompatible with end devices that have limited memory, battery or computational power. Here, we propose to address this challenge by harnessing recent advances in DNN verification. We present a framework and a methodology for discovering redundancies in DNNs - i.e., for finding neurons that are not needed, and can be removed in order to reduce the size of the DNN. By using sound verification techniques, we can formally guarantee that our simplified network is equivalent to the original, either completely, or up to a prescribed tolerance. Further, we show how to combine our technique with slicing, which results in a family of very small DNNs, which are together equivalent to the original. Our approach can produce DNNs that are significantly smaller than the original, rendering them suitable for deployment on additional kinds of systems, and even more amenable to subsequent formal verification. We provide a proof-of-concept implementation of our approach, and use it to evaluate our techniques on several real-world DNNs.

Original languageAmerican English
Title of host publicationProceedings of the 21st Formal Methods in Computer-Aided Design, FMCAD 2021
EditorsRuzica Piskac, Michael W. Whalen, Warren A. Hunt, Georg Weissenbacher
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages183-192
Number of pages10
ISBN (Electronic)9783854480464
DOIs
StatePublished - 2021
Event21st International Conference on Formal Methods in Computer-Aided Design, FMCAD 2021 - Virtual, Online, United States
Duration: 18 Oct 202122 Oct 2021

Publication series

NameProceedings of the 21st Formal Methods in Computer-Aided Design, FMCAD 2021

Conference

Conference21st International Conference on Formal Methods in Computer-Aided Design, FMCAD 2021
Country/TerritoryUnited States
CityVirtual, Online
Period18/10/2122/10/21

Bibliographical note

Publisher Copyright:
© 2021 FMCAD Associ.

Fingerprint

Dive into the research topics of 'Pruning and Slicing Neural Networks using Formal Verification'. Together they form a unique fingerprint.

Cite this