Considerations and Pitfalls in Controlled Experiments on Code Comprehension

Dror G. Feitelson

doi:10.1109/ICPC52881.2021.00019

Considerations and Pitfalls in Controlled Experiments on Code Comprehension

Dror G. Feitelson^*

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.

Original language	American English
Title of host publication	Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021
Publisher	IEEE Computer Society
Pages	106-117
Number of pages	12
ISBN (Electronic)	9781665414036
DOIs	https://doi.org/10.1109/ICPC52881.2021.00019
State	Published - May 2021
Event	29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021 - Virtual, Online Duration: 20 May 2021 → 21 May 2021

Publication series

Name	2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021)
ISSN (Print)	1092-8138

Conference

Conference	29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021
City	Virtual, Online
Period	20/05/21 → 21/05/21

Bibliographical note

Funding Information:
Dror Feitelson holds the Berthold Badler chair in Computer Science. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant no. 832/18).

Publisher Copyright:
© 2021 IEEE.

Keywords

Code comprehension
experimental methodology

Access to Document

10.1109/ICPC52881.2021.00019

Cite this

Feitelson, D. G. (2021). Considerations and Pitfalls in Controlled Experiments on Code Comprehension. In Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021 (pp. 106-117). Article 9463032 (2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021)). IEEE Computer Society. https://doi.org/10.1109/ICPC52881.2021.00019

@inproceedings{7cb0cdd06d5c4b408828173a5db36ed6,

title = "Considerations and Pitfalls in Controlled Experiments on Code Comprehension",

abstract = "Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.",

keywords = "Code comprehension, experimental methodology",

author = "Feitelson, {Dror G.}",

note = "Funding Information: Dror Feitelson holds the Berthold Badler chair in Computer Science. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant no. 832/18). Publisher Copyright: {\textcopyright} 2021 IEEE.; 29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021 ; Conference date: 20-05-2021 Through 21-05-2021",

year = "2021",

month = may,

doi = "10.1109/ICPC52881.2021.00019",

language = "American English",

series = "2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021)",

publisher = "IEEE Computer Society",

pages = "106--117",

booktitle = "Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021",

address = "United States",

}

Feitelson, DG 2021, Considerations and Pitfalls in Controlled Experiments on Code Comprehension. in Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021., 9463032, 2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021), IEEE Computer Society, pp. 106-117, 29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021, Virtual, Online, 20/05/21. https://doi.org/10.1109/ICPC52881.2021.00019

Considerations and Pitfalls in Controlled Experiments on Code Comprehension. / Feitelson, Dror G.
Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021. IEEE Computer Society, 2021. p. 106-117 9463032 (2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021)).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Considerations and Pitfalls in Controlled Experiments on Code Comprehension

AU - Feitelson, Dror G.

N1 - Funding Information: Dror Feitelson holds the Berthold Badler chair in Computer Science. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant no. 832/18). Publisher Copyright: © 2021 IEEE.

PY - 2021/5

Y1 - 2021/5

N2 - Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.

AB - Understanding program code is a complicated endeavor. As such, myriad different factors can influence the outcome. Investigations of program comprehension, and in particular those using controlled experiments, have to take these factors into account. In order to promote the development and use of sound experimental methodology, we discuss potential problems with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance.

KW - Code comprehension

KW - experimental methodology

UR - http://www.scopus.com/inward/record.url?scp=85113240105&partnerID=8YFLogxK

U2 - 10.1109/ICPC52881.2021.00019

DO - 10.1109/ICPC52881.2021.00019

M3 - Conference contribution

AN - SCOPUS:85113240105

T3 - 2021 IEEE/ACM 29TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION (ICPC 2021)

SP - 106

EP - 117

BT - Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021

PB - IEEE Computer Society

T2 - 29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021

Y2 - 20 May 2021 through 21 May 2021

ER -

Considerations and Pitfalls in Controlled Experiments on Code Comprehension

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this