Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension

Dror G. Feitelson*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


Understanding program code is a complicated endeavor. As a result, studying code comprehension is also hard. The prevailing approach for such studies is to use controlled experiments, where the difference between treatments sheds light on factors which affect comprehension. But it is hard to conduct controlled experiments with human developers, and we also need to find a way to operationalize what “comprehension” actually means. In addition, myriad different factors can influence the outcome, and seemingly small nuances may be detrimental to the study’s validity. In order to promote the development and use of sound experimental methodology, we discuss both considerations which need to be applied and potential problems that might occur, with regard to the experimental subjects, the code they work on, the tasks they are asked to perform, and the metrics for their performance. A common thread is that decisions that were taken in an effort to avoid one threat to validity may pose a larger threat than the one they removed.

Original languageAmerican English
Article number123
Number of pages42
JournalEmpirical Software Engineering
Issue number6
StatePublished - Nov 2022

Bibliographical note

Funding Information:
Dror Feitelson holds the Berthold Badler chair in Computer Science. This research was supported by the ISRAEL SCIENCE FOUNDATION (grant no. 832/18). This paper is an extended version of an “honorable mention” paper from ICPC 2021.

Publisher Copyright:
© 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.


  • Code comprehension
  • Controlled experiment
  • Experimental methodology
  • Threats to validity


Dive into the research topics of 'Considerations and Pitfalls for Reducing Threats to the Validity of Controlled Experiments on Code Comprehension'. Together they form a unique fingerprint.

Cite this