Clinical decision support system recommendations: how often do radiologists and clinicians accept them?

Objective: To assess the acceptance and reliability of clinical decision support system (CDSS) imaging referral scores (ESR iGuide). Methods: A pilot study was conducted in a tertiary hospital. Four different experts were invited to rate 40 simulated clinical cases on a 5-level scale, for the level of agreement with the ESR iGuide’s recommended procedures. In cases of disagreement, physicians were asked to indicate the reason. Descriptive measures were calculated for the level of agreement. We also explored the degree of agreement between four different specialists, and examined the cases in which clinicians disagreed with ESR iGuide best practice recommendations. Results: The mean rating of the four experts for the 40 clinical simulated cases was 4.17 ± 0.65, median 4.25 (on a scale of 1–5). All four raters totally agreed with the system recommendation in 75% of cases. No significant relationship was found between the degree of agreement and the number of indications and the patient’s age or gender. In an optimistic scenario, using a binary agree/disagree variable, the Overall Percentage Agreement for the rating of the 40 simulated cases between the four experts was 77.28%. There were a total of 20 disagreements out of 160 cases with the ESR iGuide, of which 7 were among the two radiologists. Conclusions: CDSS can be an effective tool for guiding the selection of appropriate imaging examinations, thus cutting costs due to unnecessary imaging scans. Since this is a pilot study, further research on a larger scale, preferably at national level, is required. Key Points: • The average of the mean rating of the four experts was 4.17 ± 0.65, median 4.25, on a scale of 1–5 where 5 represents total agreement with the CDSS tool. • In an optimistic scenario, using a binary agree/disagree variable, the Overall Percentage Agreement between the four experts was 77.28%. • Radiologists had fewer disagreements with the recommendations of the CDSS tool than other physicians, indicating a better fit of the support system to radiology experts’ perspective.