Verifying equivalence of spark programs

Shelly Grossman; Sara Cohen; Shachar Itzhaky; Noam Rinetzky; Mooly Sagiv

doi:10.1007/978-3-319-63390-9_15

Verifying equivalence of spark programs

Shelly Grossman^*, Sara Cohen, Shachar Itzhaky, Noam Rinetzky, Mooly Sagiv

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

Apache Spark is a popular framework for writing large scale data processing applications. Our long term goal is to develop automatic tools for reasoning about Spark programs. This is challenging because Spark programs combine database-like relational algebraic operations and aggregate operations, corresponding to (nested) loops, with User Defined Functions (UDF s). In this paper, we present a novel SMT-based technique for verifying the equivalence of Spark programs. We model Spark as a programming language whose semantics imitates Relational Algebra queries (with aggregations) over bags (multisets) and allows for UDFs expressible in Presburger Arithmetics. We prove that the problem of checking equivalence is undecidable even for programs which use a single aggregation operator. Thus, we present sound techniques for verifying the equivalence of interesting classes of Spark programs, and show that it is complete under certain restrictions. We implemented our technique, and applied it to a few small, but intricate, test cases.

Original language	English
Title of host publication	Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings
Editors	Viktor Kuncak, Rupak Majumdar
Publisher	Springer Verlag
Pages	282-300
Number of pages	19
ISBN (Print)	9783319633893
DOIs	https://doi.org/10.1007/978-3-319-63390-9_15
State	Published - 2017
Event	29th International Conference on Computer Aided Verification, CAV 2017 - Heidelberg, Germany Duration: 24 Jul 2017 → 28 Jul 2017

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10427 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	29th International Conference on Computer Aided Verification, CAV 2017
Country/Territory	Germany
City	Heidelberg
Period	24/07/17 → 28/07/17

Bibliographical note

Publisher Copyright:
© Springer International Publishing AG 2017

Access to Document

10.1007/978-3-319-63390-9_15

Cite this

Grossman, S., Cohen, S., Itzhaky, S., Rinetzky, N., & Sagiv, M. (2017). Verifying equivalence of spark programs. In V. Kuncak, & R. Majumdar (Eds.), Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings (pp. 282-300). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10427 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-63390-9_15

Grossman, Shelly ; Cohen, Sara ; Itzhaky, Shachar et al. / Verifying equivalence of spark programs. Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings. editor / Viktor Kuncak ; Rupak Majumdar. Springer Verlag, 2017. pp. 282-300 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{dd48dfb8f1f84961b8057324ffa0d1d2,

title = "Verifying equivalence of spark programs",

abstract = "Apache Spark is a popular framework for writing large scale data processing applications. Our long term goal is to develop automatic tools for reasoning about Spark programs. This is challenging because Spark programs combine database-like relational algebraic operations and aggregate operations, corresponding to (nested) loops, with User Defined Functions (UDF s). In this paper, we present a novel SMT-based technique for verifying the equivalence of Spark programs. We model Spark as a programming language whose semantics imitates Relational Algebra queries (with aggregations) over bags (multisets) and allows for UDFs expressible in Presburger Arithmetics. We prove that the problem of checking equivalence is undecidable even for programs which use a single aggregation operator. Thus, we present sound techniques for verifying the equivalence of interesting classes of Spark programs, and show that it is complete under certain restrictions. We implemented our technique, and applied it to a few small, but intricate, test cases.",

author = "Shelly Grossman and Sara Cohen and Shachar Itzhaky and Noam Rinetzky and Mooly Sagiv",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2017; 29th International Conference on Computer Aided Verification, CAV 2017 ; Conference date: 24-07-2017 Through 28-07-2017",

year = "2017",

doi = "10.1007/978-3-319-63390-9_15",

language = "אנגלית",

isbn = "9783319633893",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "282--300",

editor = "Viktor Kuncak and Rupak Majumdar",

booktitle = "Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings",

address = "גרמניה",

}

Grossman, S, Cohen, S, Itzhaky, S, Rinetzky, N & Sagiv, M 2017, Verifying equivalence of spark programs. in V Kuncak & R Majumdar (eds), Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10427 LNCS, Springer Verlag, pp. 282-300, 29th International Conference on Computer Aided Verification, CAV 2017, Heidelberg, Germany, 24/07/17. https://doi.org/10.1007/978-3-319-63390-9_15

Verifying equivalence of spark programs. / Grossman, Shelly; Cohen, Sara; Itzhaky, Shachar et al.
Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings. ed. / Viktor Kuncak; Rupak Majumdar. Springer Verlag, 2017. p. 282-300 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10427 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Verifying equivalence of spark programs

AU - Grossman, Shelly

AU - Cohen, Sara

AU - Itzhaky, Shachar

AU - Rinetzky, Noam

AU - Sagiv, Mooly

PY - 2017

Y1 - 2017

N2 - Apache Spark is a popular framework for writing large scale data processing applications. Our long term goal is to develop automatic tools for reasoning about Spark programs. This is challenging because Spark programs combine database-like relational algebraic operations and aggregate operations, corresponding to (nested) loops, with User Defined Functions (UDF s). In this paper, we present a novel SMT-based technique for verifying the equivalence of Spark programs. We model Spark as a programming language whose semantics imitates Relational Algebra queries (with aggregations) over bags (multisets) and allows for UDFs expressible in Presburger Arithmetics. We prove that the problem of checking equivalence is undecidable even for programs which use a single aggregation operator. Thus, we present sound techniques for verifying the equivalence of interesting classes of Spark programs, and show that it is complete under certain restrictions. We implemented our technique, and applied it to a few small, but intricate, test cases.

AB - Apache Spark is a popular framework for writing large scale data processing applications. Our long term goal is to develop automatic tools for reasoning about Spark programs. This is challenging because Spark programs combine database-like relational algebraic operations and aggregate operations, corresponding to (nested) loops, with User Defined Functions (UDF s). In this paper, we present a novel SMT-based technique for verifying the equivalence of Spark programs. We model Spark as a programming language whose semantics imitates Relational Algebra queries (with aggregations) over bags (multisets) and allows for UDFs expressible in Presburger Arithmetics. We prove that the problem of checking equivalence is undecidable even for programs which use a single aggregation operator. Thus, we present sound techniques for verifying the equivalence of interesting classes of Spark programs, and show that it is complete under certain restrictions. We implemented our technique, and applied it to a few small, but intricate, test cases.

UR - http://www.scopus.com/inward/record.url?scp=85026736663&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-63390-9_15

DO - 10.1007/978-3-319-63390-9_15

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85026736663

SN - 9783319633893

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 282

EP - 300

BT - Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings

A2 - Kuncak, Viktor

A2 - Majumdar, Rupak

PB - Springer Verlag

T2 - 29th International Conference on Computer Aided Verification, CAV 2017

Y2 - 24 July 2017 through 28 July 2017

ER -

Grossman S, Cohen S, Itzhaky S, Rinetzky N, Sagiv M. Verifying equivalence of spark programs. In Kuncak V, Majumdar R, editors, Computer Aided Verification - 29th International Conference, CAV 2017, Proceedings. Springer Verlag. 2017. p. 282-300. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-63390-9_15

Verifying equivalence of spark programs

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this