Towards Scalable Verification of Deep Reinforcement Learning

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

14 Scopus citations

Abstract

Deep neural networks (DNNs) have gained significant popularity in recent years, becoming the state of the art in a variety of domains. In particular, deep reinforcement learning (DRL) has recently been employed to train DNNs that realize control policies for various types of real-world systems. In this work, we present the whiRL 2.0 tool, which implements a new approach for verifying complex properties of interest for DRL systems. To demonstrate the benefits of whiRL 2.0, we apply it to case studies from the communication networks domain that have recently been used to motivate formal verification of DRL systems, and which exhibit characteristics that are conducive for scalable verification. We propose techniques for performing k-induction and semi-automated invariant inference on such systems, and leverage these techniques for proving safety and liveness properties that were previously impossible to verify due to the scalability barriers of prior approaches. Furthermore, we show how our proposed techniques provide insights into the inner workings and the generalizability of DRL systems. whiRL 2.0 is publicly available online.

Original languageAmerican English
Title of host publicationProceedings of the 21st Formal Methods in Computer-Aided Design, FMCAD 2021
EditorsRuzica Piskac, Michael W. Whalen, Warren A. Hunt, Georg Weissenbacher
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages193-203
Number of pages11
ISBN (Electronic)9783854480464
DOIs
StatePublished - 2021
Event21st International Conference on Formal Methods in Computer-Aided Design, FMCAD 2021 - Virtual, Online, United States
Duration: 18 Oct 202122 Oct 2021

Publication series

NameProceedings of the 21st Formal Methods in Computer-Aided Design, FMCAD 2021

Conference

Conference21st International Conference on Formal Methods in Computer-Aided Design, FMCAD 2021
Country/TerritoryUnited States
CityVirtual, Online
Period18/10/2122/10/21

Bibliographical note

Funding Information:
Acknowledgements. We thank Nathan Jay, Tomer Eliyahu and the anonymous reviewers for their contributions to this project. The project was partially supported by the Israel Science Foundation (grant number 683/18), the Binational Science Foundation (grant numbers 2017662 and 2019798), and the Center for Interdisciplinary Data Science Research at The Hebrew University of Jerusalem.

Publisher Copyright:
© 2021 FMCAD Associ.

Fingerprint

Dive into the research topics of 'Towards Scalable Verification of Deep Reinforcement Learning'. Together they form a unique fingerprint.

Cite this