Inferring accuracy for judges and items: Choice of unit of analysis reverses the conclusions

Yaacov Schul*, Ilan Yaniv

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

This study explores the relationship between the precision and the accuracy of forecasts using either judge or item as the unit of analysis. Participants in five experiments answered general-knowledge questions by indicating intervals that were likely to include the correct answer. Results indicate that the precision of an interval estimate is not a straightforward cue to the likelihood that such an interval includes the truth (hit rate). Whereas judges who state more precise estimates (i.e. who provide narrower interval estimates) have lower hit rates, questions for which the average judgment is more precise have higher hit rates. Thus, the relation between precision and accuracy depends on whether one 'slices' the data by judge or by question. We offer an explanation for this seemingly paradoxical effect and implement it as a computer simulation to demonstrate its validity.

Original languageAmerican English
Pages (from-to)343-354
Number of pages12
JournalJournal of Behavioral Decision Making
Volume10
Issue number4
DOIs
StatePublished - 1997

Keywords

  • Accuracy
  • Aggregation
  • Confidence
  • Error
  • Judgmental estimation
  • Precision

Fingerprint

Dive into the research topics of 'Inferring accuracy for judges and items: Choice of unit of analysis reverses the conclusions'. Together they form a unique fingerprint.

Cite this