The structured oral examination as a method for assessing surgical residents

The purpose of this study was to determine the overall reliability, inter-rater reliability, and criterion validity of the structured oral examination (SOE) for assessing surgical residents. An SOE consisting of four predetermined clinically oriented scenarios was administered to 23 second postgraduate year surgical residents. Each scenario had five to six questions, each with a specific marking scheme. Candidates were assessed by two examiners and scores were derived independently. Overall reliability (Cronbach's alpha) was 0.75. Inter-rater reliability was significant for each pair of examiners and each question (r = 0.78 to 0.91; p < 0.0001). Criterion validity was measured by correlating SOE scores with multiple-choice examination (MCQ) and objective structured clinical examination (OSCE) scores. Correlations between the SOE and MCQ and OSCE were significant and fell into the moderate range (0.48 to 0.51). The results of this study show that the SOE is useful in the assessment of clinical knowledge and problem-solving abilities of the surgical resident. Overall and inter-rater reliabilities achieved exceed those of traditional oral examination formats.