The role of sample size in sample evaluation

D. Kahneman and A. Tversky (Cognitive Psychology, 1972, 3, 430-454) claimed that "the notion that sampling variance decreases in proportion to sample size is apparently not part of man's repertoire of intuitions." This study presents a series of experiments showing that it is possible to elicit judgments indicating that perceived sample accuracy increases with sample size. However, these judgments seem to reflect sensitivity to sample-to-population ratio rather than absolute sample size. In fact, people may trade sample size for sample-to-population ratio, even when this actually decreases expected sample accuracy. The widely held belief that the accuracy of a sample is connected with its relative size to the universe is mistaken. A sample smaller than 1%, taken from one universe, can be much more reliable than one comprising 10% of another. To determine with equal accuracy the average age of the population of New York City and of Peoria, Illinois, will require samples of equal size (variances of population being equal). (Zeisel, 1960).