The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net

Michal Chorev; Yoel Shoshan; Adam Spiro; Shaked Naor; Alon Hazan; Vesna Barros; Iuliana Weinstein; Esma Herzel; Varda Shalev; Michal Guindy; Michal Rosen-Zvi

doi:10.1007/978-3-030-59725-2_22

The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net

Michal Chorev^*
, Yoel Shoshan
, Adam Spiro
, Shaked Naor
, Alon Hazan
, Vesna Barros
, Iuliana Weinstein
, Esma Herzel
, Varda Shalev
, Michal Guindy
, Michal Rosen-Zvi

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7% of the radiologists’ missed cancers. Thus, significantly increasing radiologist’s sensitivity to 80.3%, while only slightly decreasing their specificity to 95.3%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7% of the radiologists’ missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.

Original language	English
Title of host publication	Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings
Editors	Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, Leo Joskowicz
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	220-229
Number of pages	10
ISBN (Print)	9783030597245
DOIs	https://doi.org/10.1007/978-3-030-59725-2_22
State	Published - 2020
Externally published	Yes
Event	23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020 - Lima, Peru Duration: 4 Oct 2020 → 8 Oct 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12266 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020
Country/Territory	Peru
City	Lima
Period	4/10/20 → 8/10/20

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Keywords

Breast imaging
Computer-aided diagnosis
Deep learning

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-59725-2_22

Cite this

Chorev, M., Shoshan, Y., Spiro, A., Naor, S., Hazan, A., Barros, V., Weinstein, I., Herzel, E., Shalev, V., Guindy, M., & Rosen-Zvi, M. (2020). The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net. In A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, & L. Joskowicz (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings (pp. 220-229). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12266 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-59725-2_22

Chorev, Michal ; Shoshan, Yoel ; Spiro, Adam et al. / The Case of Missed Cancers : Applying AI as a Radiologist’s Safety Net. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings. editor / Anne L. Martel ; Purang Abolmaesumi ; Danail Stoyanov ; Diana Mateus ; Maria A. Zuluaga ; S. Kevin Zhou ; Daniel Racoceanu ; Leo Joskowicz. Springer Science and Business Media Deutschland GmbH, 2020. pp. 220-229 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a665cebc44914322825811e159241ef0,

title = "The Case of Missed Cancers: Applying AI as a Radiologist{\textquoteright}s Safety Net",

abstract = "We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7\% of the radiologists{\textquoteright} missed cancers. Thus, significantly increasing radiologist{\textquoteright}s sensitivity to 80.3\%, while only slightly decreasing their specificity to 95.3\%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7\% of the radiologists{\textquoteright} missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.",

keywords = "Breast imaging, Computer-aided diagnosis, Deep learning",

author = "Michal Chorev and Yoel Shoshan and Adam Spiro and Shaked Naor and Alon Hazan and Vesna Barros and Iuliana Weinstein and Esma Herzel and Varda Shalev and Michal Guindy and Michal Rosen-Zvi",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020 ; Conference date: 04-10-2020 Through 08-10-2020",

year = "2020",

doi = "10.1007/978-3-030-59725-2\_22",

language = "אנגלית",

isbn = "9783030597245",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "220--229",

editor = "Martel, \{Anne L.\} and Purang Abolmaesumi and Danail Stoyanov and Diana Mateus and Zuluaga, \{Maria A.\} and Zhou, \{S. Kevin\} and Daniel Racoceanu and Leo Joskowicz",

booktitle = "Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings",

address = "גרמניה",

}

Chorev, M, Shoshan, Y, Spiro, A, Naor, S, Hazan, A, Barros, V, Weinstein, I, Herzel, E, Shalev, V, Guindy, M & Rosen-Zvi, M 2020, The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net. in AL Martel, P Abolmaesumi, D Stoyanov, D Mateus, MA Zuluaga, SK Zhou, D Racoceanu & L Joskowicz (eds), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12266 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 220-229, 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, Lima, Peru, 4/10/20. https://doi.org/10.1007/978-3-030-59725-2_22

The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net. / Chorev, Michal; Shoshan, Yoel; Spiro, Adam et al.
Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings. ed. / Anne L. Martel; Purang Abolmaesumi; Danail Stoyanov; Diana Mateus; Maria A. Zuluaga; S. Kevin Zhou; Daniel Racoceanu; Leo Joskowicz. Springer Science and Business Media Deutschland GmbH, 2020. p. 220-229 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12266 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The Case of Missed Cancers

T2 - 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020

AU - Chorev, Michal

AU - Shoshan, Yoel

AU - Spiro, Adam

AU - Naor, Shaked

AU - Hazan, Alon

AU - Barros, Vesna

AU - Weinstein, Iuliana

AU - Herzel, Esma

AU - Shalev, Varda

AU - Guindy, Michal

AU - Rosen-Zvi, Michal

PY - 2020

Y1 - 2020

N2 - We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7% of the radiologists’ missed cancers. Thus, significantly increasing radiologist’s sensitivity to 80.3%, while only slightly decreasing their specificity to 95.3%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7% of the radiologists’ missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.

AB - We investigate the potential contribution of an AI system as a safety net application for radiologists in breast cancer screening. As a safety net, the AI alerts on cases suspected to be malignant which the radiologist did not recommend for a recall. We analyzed held-out data of 2,638 exams enriched with 90 missed cancers. In screening mammography settings, we show that a system alerting on 11 out of every 1,000 cases, could detect up to 10.7% of the radiologists’ missed cancers. Thus, significantly increasing radiologist’s sensitivity to 80.3%, while only slightly decreasing their specificity to 95.3%. Importantly, the safety net demonstrated a significant contribution to their performance even when radiologists utilized both mammography and ultrasound images. In those settings, it would have alerted 8.5 times per 1,000 cases, and detected 11.7% of the radiologists’ missed cancers. In an analysis of the missed cancers by an expert, we found that most of the cancers detected by the AI were visible post-hoc. Finally, we performed a reader study with five radiologists over 120 exams, 10 of which were originally missed cancers. The AI safety net was able to assist 3 out of the 5 radiologists in detecting missed cancers without raising any false alerts.

KW - Breast imaging

KW - Computer-aided diagnosis

KW - Deep learning

UR - https://www.scopus.com/pages/publications/85092794141

U2 - 10.1007/978-3-030-59725-2_22

DO - 10.1007/978-3-030-59725-2_22

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85092794141

SN - 9783030597245

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 220

EP - 229

BT - Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings

A2 - Martel, Anne L.

A2 - Abolmaesumi, Purang

A2 - Stoyanov, Danail

A2 - Mateus, Diana

A2 - Zuluaga, Maria A.

A2 - Zhou, S. Kevin

A2 - Racoceanu, Daniel

A2 - Joskowicz, Leo

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 4 October 2020 through 8 October 2020

ER -

Chorev M, Shoshan Y, Spiro A, Naor S, Hazan A, Barros V et al. The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net. In Martel AL, Abolmaesumi P, Stoyanov D, Mateus D, Zuluaga MA, Zhou SK, Racoceanu D, Joskowicz L, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 - 23rd International Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 220-229. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-59725-2_22

The Case of Missed Cancers: Applying AI as a Radiologist’s Safety Net

Abstract

Publication series

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this