DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks

Divya Gopinath, Guy Katz, Corina S. Păsăreanu*, Clark Barrett

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

53 Scopus citations


Deep neural networks have achieved impressive results in many complex applications, including classification tasks for image and speech recognition, pattern analysis or perception in self-driving vehicles. However, it has been observed that even highly trained networks are very vulnerable to adversarial perturbations. Adding minimal changes to inputs that are correctly classified can lead to wrong predictions, raising serious security and safety concerns. Existing techniques for checking robustness against such perturbations only consider searching locally around a few individual inputs, providing limited guarantees. We propose DeepSafe, a novel approach for automatically assessing the overall robustness of a neural network. DeepSafe applies clustering over known labeled data and leverages off-the-shelf constraint solvers to automatically identify and check safe regions in which the network is robust, i.e. all the inputs in the region are guaranteed to be classified correctly. We also introduce the concept of targeted robustness, which ensures that the neural network is guaranteed not to misclassify inputs within a region to a specific target (adversarial) label. We evaluate DeepSafe on a neural network implementation of a controller for the next-generation Airborne Collision Avoidance System for unmanned aircraft (ACAS Xu) and for the well known MNIST network. For these networks, DeepSafe identified many regions which were safe, and also found adversarial perturbations of interest.

Original languageAmerican English
Title of host publicationAutomated Technology for Verification and Analysis - 16th International Symposium, ATVA 2018, Proceedings
EditorsChao Wang, Shuvendu K. Lahiri
PublisherSpringer Verlag
Number of pages17
ISBN (Print)9783030010898
StatePublished - 2018
Externally publishedYes
Event16th International Symposium on Automated Technology for Verification and Analysis, ATVA 2018 - Los Angeles, United States
Duration: 7 Oct 201810 Oct 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11138 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference16th International Symposium on Automated Technology for Verification and Analysis, ATVA 2018
Country/TerritoryUnited States
CityLos Angeles

Bibliographical note

Publisher Copyright:
© 2018, Springer Nature Switzerland AG.


Dive into the research topics of 'DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks'. Together they form a unique fingerprint.

Cite this