Measurement errors in probability judgments

This paper investigates the psychometric properties of three measures of subjective uncertainty - a zero-to-hundred subjective probability scale and two seven point rating scales. Individual level analysis applied to data obtained from two separate studies suggests that the scales produce fairly similar results: The inter-response mode correlations were high, and individual plots comparing various methods were quite similar. Covariance structure models based on multitrait-multimethod matrices are utilized to assess the reliability and method variance of the scales. The cumulative evidence suggests that rating scales are consistently just as reliable as the subjective probability scale. The probability scale contained significant method error. In fact, the two raring scales were found to have lower systematic method variance and lower random error variance than the subjective probability scale. The paper concludes with a discussion regarding possible explanations of these results and directions for future research.