Differences unrelated to clinical competence in the results of repeated multiple-station tests of clinical skills

Purpose. To determine the extent to which systematic differences not related to gruop differences in clinical competence could be observed in the results of six administrations of the same multiple-station test of clinical skills. Method. The same 16 stations were administered as part of the Ontario International Medical Graduates Program clinical skills screening examination on three different occasions, and on each occasion in two sessions. The interval between the first (fall 1986) and second (March 1990) administrations was four years, and that between the second and third (July 1990) administrations, four months. International medical graduates were the candidates in the first two administrations; the third administration was to a combined sample of fourth-year clinical clerks and interns. A different item functioning (DIF) approach with sessions within administration as the grouping variable was used to describe the extent of differences unrelated to clinical competence in the results of the different test administrations. For the generalizability and DIF calculations the designs were balanced by sampling down to 33 cases in each session. Results. Station DIF values varied considerably from station to station. DIF effects between sessions with administrations were less than those between administrations, and were less between the two administrations separated by four months than between the two adminstrations separated by four years. Conclusion. Hypotheses concerning the relative magnitudes of station DIF and total-test aggregated values of DIF, as a function of the time intervals between test occasions, were substantiated, demonstrating that the greater the time interval between test administrations, the greater the magnitude of DIF.