Inferring accuracy for judges and items: Choice of unit of analysis reverses the conclusions

This study explores the relationship between the precision and the accuracy of forecasts using either judge or item as the unit of analysis. Participants in five experiments answered general-knowledge questions by indicating intervals that were likely to include the correct answer. Results indicate that the precision of an interval estimate is not a straightforward cue to the likelihood that such an interval includes the truth (hit rate). Whereas judges who state more precise estimates (i.e. who provide narrower interval estimates) have lower hit rates, questions for which the average judgment is more precise have higher hit rates. Thus, the relation between precision and accuracy depends on whether one 'slices' the data by judge or by question. We offer an explanation for this seemingly paradoxical effect and implement it as a computer simulation to demonstrate its validity.