Rate it again: Using the wisdom of many to improve performance evaluations

Meir Barneron*, Avi Allalouf, Ilan Yaniv

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


Much research shows that judgmental estimation could be improved by combining estimates from independent judges as well as within judges. These results have been obtained mostly with judgments about matters of fact, that is, for which there are objective truth criteria. In the present research, we extend these findings to performance evaluations. In a controlled field study, expert judges provided evaluations of a large number of essays written by college applicants taking college entrance tests. The judges were asked to evaluate each essay twice—on two occasions, a week apart. This design allowed us to assess the benefits of two methods of combining evaluations: within rater and across raters. Accuracy gains were obtained with both methods. Although the within-rater combinations yielded fewer gains than the across-rater ones, they were still appreciable in comparison with the across rater ones. Our findings extend the class of judgments to which the “wisdom of many” could be applied. These findings are potentially applicable to performance evaluations in social, educational, and employment settings.

Original languageAmerican English
Pages (from-to)485-492
Number of pages8
JournalJournal of Behavioral Decision Making
Issue number4
StatePublished - 1 Oct 2019

Bibliographical note

Publisher Copyright:
© 2019 John Wiley & Sons, Ltd.


  • combining opinions
  • judgment
  • performance evaluation
  • wisdom of many in one mind


Dive into the research topics of 'Rate it again: Using the wisdom of many to improve performance evaluations'. Together they form a unique fingerprint.

Cite this