Please wait a minute...
Shopping Cart Email List Chinese
Advanced Search Fig/Tab
Frontiers of Medicine

ISSN 2095-0217

ISSN 2095-0225(Online)

CN 11-5983/R

Postal Subscription Code 80-967

2018 Impact Factor: 1.847

Featured Articles  |  Most Downloaded  |  Most Reading
Online Submission Subscribe to Our RSS Content Feed
Front. Med.    2020, Vol. 14 Issue (4) : 369-381    https://doi.org/10.1007/s11684-020-0784-7
REVIEW
Current applications of artificial intelligence for intraoperative decision support in surgery
Allison J. Navarrete-Welton1, Daniel A. Hashimoto1,2()
1. Surgical Artificial Intelligence and Innovation Laboratory, Massachusetts General Hospital, Boston, MA 02114, USA
2. Harvard Medical School, Boston, MA 02114, USA
 Download: PDF(241 KB)   HTML
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Research into medical artificial intelligence (AI) has made significant advances in recent years, including surgical applications. This scoping review investigated AI-based decision support systems targeted at the intraoperative phase of surgery and found a wide range of technological approaches applied across several surgical specialties. Within the twenty-one (n=21) included papers, three main categories of motivations were identified for developing such technologies: (1) augmenting the information available to surgeons, (2) accelerating intraoperative pathology, and (3) recommending surgical steps. While many of the proposals hold promise for improving patient outcomes, important methodological shortcomings were observed in most of the reviewed papers that made it difficult to assess the clinical significance of the reported performance statistics. Despite limitations, the current state of this field suggests that a number of opportunities exist for future researchers and clinicians to work on AI for surgical decision support with exciting implications for improving surgical care.
Keywords artificial intelligence      decision support      clinical decision support systems      intraoperative      deep learning      computer vision      machine learning      surgery     
Corresponding Author(s): Daniel A. Hashimoto   
Just Accepted Date: 11 May 2020   Online First Date: 06 July 2020    Issue Date: 26 August 2020
 Cite this article:   
Allison J. Navarrete-Welton,Daniel A. Hashimoto. Current applications of artificial intelligence for intraoperative decision support in surgery[J]. Front. Med., 2020, 14(4): 369-381.
 URL:  
https://academic.hep.com.cn/fmd/EN/10.1007/s11684-020-0784-7
https://academic.hep.com.cn/fmd/EN/Y2020/V14/I4/369
Included Excluded
Artificial intelligence applied to decision support for the surgeon during the intraoperative phase of surgery Decision support during the preoperative and postoperative surgical phases
Anesthesia, surgical training, and surgeon skill evaluation when unrelated to clinical decision support
Studies with at least 10 human patients Studies with fewer than 10 patients
Published prior to May 25, 2019 Animal studies
Peer-reviewed published literature and conference proceedings papers Narrative review papers, editorials, letters to the editor, abstracts
All geographical areas but written in English Languages other than English
Tab.1  Inclusion and exclusion criteria for this scoping review
Fig.1  Modified PRISMA diagram for this scoping review.
Fig.2  Motivations cited by reviewed papers for developing artificial intelligence-based intraoperative decision support systems.
Paper Data set Data set processing Data set distribution Validation Performancea
André et al. 2009 [17] 54 patients; colonic polyps 1036 images; videos discarded if histology and pCLE diagnoses did not match Roughly equal (2 classes) Leave-n-out cross validation (patient held-out) 80.1% weighted k-NN classification accuracy (benign vs. pathological)
André et al. 2010 [15] 68 patients; colonic polyps 121 single-polyp videos Roughly equal (2 classes) Leave-one-patient-out cross-validation 94.2% weighted k-NN classification accuracy (benign vs. pathological)
André et al. 2012 [16] 66 patients; colonic polyps 118 single-polyp videos Unspecified (8 binary classes) 30 × 3-fold cross-validation (patient segregated) 96.7% AUC for the highest-performing semantic concept (“elongated crypt”)
49.4% Kendall τ rank correlation coefficient (Likert similarity scale)
Kohandani Tafresh et al. 2014 [18] 66 patients; colonic polyps 118 videos (mostly single-tissue) Unbalanced (35 benign, 83 neoplastic) Leave-one-patient-out cross-validation 89.9% k-NN classification accuracy
48.8% Spearman ? correlation coefficient (Likert similarity scale)
Gu et al. 2016 [19] 50 patients; breast tissue Unspecified Unspecified (3 coarse classes, 8 sub-classes) 10-fold cross validation (not patient-segregated) 96.6% SVM classification accuracy (coarse class)
Gu et al. 2017 [20] 45 patients; breast tissue 700 pCLE mosaics, 144 matched with histology images Unspecified (3 coarse classes, 8 sub-classes) 10-fold cross validation (not patient-segregated) 89.2% top-1 retrieval accuracy (coarse class)
96.2% top-5 retrieval accuracy (coarse class)
Tab.2  Data set characteristics and algorithm performance on pCLE case retrieval
Paper Imaging modality Aim Method
Tumor margin mapping (n = 3)
Ritschel et al. (2015) [22] CEUS Localize glioblastoma tumor residuals Latent Dirichlet analysis, support vector machine
Ilunga-Mbuyamba et al. (2017) [23] CEUS Localize glioblastoma tumor residuals Data fusion
Fabelo et al. (2019) [31] HSI Map locations of glioblastoma tissue, normal tissue, hypervascularized tissue, and background material Support vector machine, convolutional neural network/deep neural network joint architecture
Tumor classification (n = 3)
Wan et al. (2015) [25] pCLE Distinguish between glioblastoma and meningioma Feature descriptors, bag-of-visual-words dimensionality reduction, support vector machine
Kamen et al. (2016) [26] pCLE Distinguish between glioblastoma and meningioma Feature descriptors, sparse coding with locality constraint to reduce dimensionality, support vector machine
Li et al. (2018) [27] pCLE Distinguish between glioblastoma and meningioma Convolutional neural network, long short-term memory neural network
Tissue identification (n = 4)
Couceiro et al. (2012) [28] pCLE Classify low or high probability of inflammatory bowel disease, based on intestinal crypts Feature descriptors, support vector machine
Halicek et al. (2017) [29] HSI Classify normal or cancerous thyroid and aerodigestive tract tissues Convolutional neural network
Halicek et al. (2018) [30] HSI Distinguish thyroid carcinoma from normal tissue Convolutional neural network
Hou et al. (2019) [32] OCT Distinguish metastatic lymph nodes from normal lymph nodes Artificial neural network
Tab.3  Summary of papers with the aim of accelerating intraoperative pathology
1 F Spencer. Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Am Coll Surg 1978; 63: 9–12
2 JW Suliburk, QM Buck, CJ Pirko, NN Massarweh, NR Barshes, H Singh, TK Rosengart. Analysis of human performance deficiencies associated with surgical adverse events. JAMA Netw Open 2019; 2(7): e198067
https://doi.org/10.1001/jamanetworkopen.2019.8067 pmid: 31365107
3 CM Pugh, S Santacaterina, DA DaRosa, RE Clark. Intra-operative decision making: more than meets the eye. J Biomed Inform 2011; 44(3): 486–496
https://doi.org/10.1016/j.jbi.2010.01.001 pmid: 20096376
4 DA Hashimoto, CG Axelsson, CB Jones, R Phitayakorn, E Petrusa, SK McKinley, D Gee, C Pugh. Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy. Am J Surg 2019; 217(2): 356–361
https://doi.org/10.1016/j.amjsurg.2018.11.011 pmid: 30470551
5 CM Pugh, DA DaRosa. Use of cognitive task analysis to guide the development of performance-based assessments for intraoperative decision making. Mil Med 2013; 178(10 Suppl): 22–27
https://doi.org/10.7205/MILMED-D-13-00207 pmid: 24084302
6 R Flin, G Youngson, S Yule. How do surgeons make intraoperative decisions? Qual Saf Health Care 2007; 16(3): 235–239
https://doi.org/10.1136/qshc.2006.020743 pmid: 17545353
7 DA Hashimoto, G Rosman, ER Witkowski, C Stafford, AJ Navarette-Welton, DW Rattner, KD Lillemoe, DL Rus, OR Meireles. Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 2019; 270(3): 414–421
https://doi.org/10.1097/SLA.0000000000003460 pmid: 31274652
8 DA Hashimoto, G Rosman, D Rus, OR Meireles. Artificial intelligence in surgery: promises and perils. Ann Surg 2018; 268(1): 70–76
https://doi.org/10.1097/SLA.0000000000002693 pmid: 29389679
9 A Hosny, C Parmar, J Quackenbush, LH Schwartz, HJWL Aerts. Artificial intelligence in radiology. Nat Rev Cancer 2018; 18(8): 500–510
https://doi.org/10.1038/s41568-018-0016-5 pmid: 29777175
10 K Bera, KA Schalper, DL Rimm, V Velcheti, A Madabhushi. Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019; 16(11): 703–715
https://doi.org/10.1038/s41571-019-0252-y pmid: 31399699
11 DT Hogarty, JC Su, K Phan, M Attia, M Hossny, S Nahavandi, P Lenane, FJ Moloney, A Yazdabadi. Artificial intelligence in dermatology—where we are and the way to the future: a review. Am J Clin Dermatol 2020; 21(1): 41–47
pmid: 31278649
12 L Maier-Hein, SS Vedula, S Speidel, N Navab, R Kikinis, A Park, M Eisenmann, H Feussner, G Forestier, S Giannarou, M Hashizume, D Katic, H Kenngott, M Kranzfelder, A Malpani, K März, T Neumuth, N Padoy, C Pugh, N Schoch, D Stoyanov, R Taylor, M Wagner, GD Hager, P Jannin. Surgical data science for next-generation interventions. Nat Biomed Eng 2017; 1(9): 691–696
https://doi.org/10.1038/s41551-017-0132-7 pmid: 31015666
13 R Udelsman, P Donovan, C Shaw. Cure predictability during parathyroidectomy. World J Surg 2014; 38(3): 525–533
https://doi.org/10.1007/s00268-013-2327-8 pmid: 24240672
14 B Harangi, A Hajdu, R Lampe, P Torok. Recognizing ureter and uterine artery in endoscopic images using a convolutional neural network. In: 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS). 2017. 726–727. doi: 10.1109/CBMS.2017.137
15 B André, T Vercauteren, AM Buchner, MB Wallace, N Ayache. Endomicroscopic video retrieval using mosaicing and visualwords. In: 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2010. doi: 10.1109/isbi.2010.5490265
16 B André, T Vercauteren, AM Buchner, MB Wallace, N Ayache. Learning semantic and visual similarity for endomicroscopy video retrieval. IEEE Trans Med Imaging 2012; 31(6): 1276–1288
https://doi.org/10.1109/TMI.2012.2188301 pmid: 22353403
17 B André, T Vercauteren, A Perchant, A Buchner, M Wallace, N Ayache. Endomicroscopic image retrieval and classification using invariant visual features. In: 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2009. doi: 10.1109/isbi.2009.5193055
18 M Kohandani Tafresh, N Linard, B André, N Ayache, T Vercauteren. Semi-automated query construction for content-based endomicroscopy video retrieval. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2014. Springer International Publishing, 2014. 89–96. doi: 10.1007/978-3-319-10404-1_12
19 Y Gu, J Yang, GZ Yang. Multi-view multi-modal feature embedding for endomicroscopy mosaic classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2016. 11–19
20 Y Gu, K Vyas, J Yang, GZ Yang. Unsupervised feature learning for endomicroscopy image retrieval. In: Medical Image Computing and Computer Assisted Intervention — MICCAI 2017. Springer International Publishing, 2017. 64–71 doi: 10.1007/978-3-319-66179-7_8
21 G Quellec, M Lamard, G Cazuguel, Z Droueche, C Roux, B Cochener. Real-time retrieval of similar videos with application to computer-aided retinal surgery. Conf Proc IEEE Eng Med Biol Soc 2011; 2011: 4465–4468
https://doi.org/10.1109/IEMBS.2011.6091107 pmid: 22255330
22 K Ritschel, I Pechlivanis, S Winter. Brain tumor classification on intraoperative contrast-enhanced ultrasound. Int J CARS 2015; 10(5): 531–540
https://doi.org/10.1007/s11548-014-1089-6 pmid: 24956998
23 E Ilunga-Mbuyamba, D Lindner, J Avina-Cervantes, F Arlt, H Rostro-Gonzalez, I Cruz-Aceves, C Chalopin. Fusion of intraoperative 3D B-mode and contrast-enhanced ultrasound data for automatic identification of residual brain tumors. Appl Sci (Basel) 2017; 7(4): 415
https://doi.org/10.3390/app7040415
24 P Dollar, Z Tu, P Perona, S Belongie. Integral channel features. In: Procedings of the British Machine Vision Conference. 2009. doi: 10.5244/c.23.91
25 S Wan, S Sun, S Bhattacharya, S Kluckner, A Gigler, E Simon, M Fleischer, P Charalampaki, T Chen, A Kamen. Towards an efficient computational framework for guiding surgical resection through intra-operative endo-microscopic pathology. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI 2015. Springer International Publishing, 2015. 421–429. doi: 10.1007/978-3-319-24553-9_52
26 A Kamen, S Sun, S Wan, S Kluckner, T Chen, AM Gigler, E Simon, M Fleischer, M Javed, S Daali, A Igressa, P Charalampaki. Automatic tissue differentiation based on confocal endomicroscopic images for intraoperative guidance in neurosurgery. BioMed Res Int 2016; 2016: 6183218
https://doi.org/10.1155/2016/6183218 pmid: 27127791
27 Y Li, P Charalampaki, Y Liu, GZ Yang, S Giannarou. Context aware decision support in neurosurgical oncology based on an efficient classification of endomicroscopic data. Int J CARS 2018; 13(8): 1187–1199
https://doi.org/10.1007/s11548-018-1806-7 pmid: 29948845
28 S Couceiro, JP Barreto, P Freire, P Figueiredo. Description and classification of confocal endomicroscopic images for the automatic diagnosis of inflammatory bowel disease. In: Machine Learning in Medical Imaging. Springer Berlin Heidelberg, 2012. 144–151. doi: 10.1007/978-3-642-35428-1_18
29 M Halicek, G Lu, JV Little, X Wang, M Patel, CC Griffith, MW El-Deiry, AY Chen, B Fei. Deep convolutional neural networks for classifying head and neck cancer using hyperspectral imaging. J Biomed Opt 2017; 22(6): 60503
https://doi.org/10.1117/1.JBO.22.6.060503 pmid: 28655055
30 M Halicek, JV Little, X Wang, M Patel, CC Griffith, MW El-Deiry, AY Chen, B Fei. Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks. Proc SPIE Int Soc Opt Eng 2018; 104690X doi: 10.1117/12.2289023
https://doi.org/10.1117/12.2289023 pmid: 30197462
31 H Fabelo, M Halicek, S Ortega, M Shahedi, A Szolna, JF Piñeiro, C Sosa, AJ O’Shanahan, S Bisshopp, C Espino, M Márquez, M Hernández, D Carrera, J Morera, GM Callico, R Sarmiento, B Fei. Deep learning-based framework for in vivo identification of glioblastoma tumor using hyperspectral images of human brain. Sensors (Basel) 2019; 19(4): 920
https://doi.org/10.3390/s19040920 pmid: 30813245
32 F Hou, Y Liang, Z Yang, W Gu, Y Yu. Automatic identification of metastatic lymph nodes in OCT images. Proceedings Volume 10867, Optical Coherence Tomography and Coherence Domain Optical Methods in Biomedicine XXIII; 108673G. 2019. doi: 10.1117/12.2511588
https://doi.org/10.1117/12.2511588
33 S Tian, XC Yin, ZB Wang, F Zhou, HW Hao. A VidEo-Based Intelligent Recognition and Decision System for the phacoemulsification cataract surgery. Comput Math Methods Med 2015; 2015: 202934
https://doi.org/10.1155/2015/202934 pmid: 26693249
34 B Fan, HX Li, Y Hu. An intelligent decision system for intraoperative somatosensory evoked potential monitoring. IEEE Trans Neural Syst Rehabil Eng 2016; 24(2): 300–307
https://doi.org/10.1109/TNSRE.2015.2477557 pmid: 26415181
35 L Gordon, T Grantcharov, F Rudzicz. Explainable artificial intelligence for safe intraoperative decision support. JAMA Surg 2019; 154(11): 1064
https://doi.org/10.1001/jamasurg.2019.2821 pmid: 31509185
36 F Lalys, P Jannin. Surgical process modelling: a review. Int J CARS 2014; 9(3): 495–511
https://doi.org/10.1007/s11548-013-0940-5 pmid: 24014322
37 J Krause, V Gulshan, E Rahimy, P Karth, K Widner, GS Corrado, L Peng, DR Webster. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 2018; 125(8): 1264–1272
https://doi.org/10.1016/j.ophtha.2018.01.034 pmid: 29548646
38 H Lee, S Yune, M Mansouri, M Kim, SH Tajmir, CE Guerrier, SA Ebert, SR Pomerantz, JM Romero, S Kamalian, RG Gonzalez, MH Lev, S Do. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat Biomed Eng 2019; 3(3): 173–182
https://doi.org/10.1038/s41551-018-0324-9 pmid: 30948806
39 A Esteva, B Kuprel, RA Novoa, J Ko, SM Swetter, HM Blau, S Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542(7639): 115–118
https://doi.org/10.1038/nature21056 pmid: 28117445
40 NM Safdar, JD Banja, CC Meltzer. Ethical considerations in artificial intelligence. Eur J Radiol 2020; 122: 108768
https://doi.org/10.1016/j.ejrad.2019.108768 pmid: 31786504
[1] FMD-20035-OF-HD_suppl_1 Download
[1] Peng Wang, Zehao Zhao, Linghao Bu, Nijiati Kudulaiti, Qiao Shan, Yuyao Zhou, N. U. Farrukh Hameed, Yangming Zhu, Lei Jin, Jie Zhang, Junfeng Lu, Jinsong Wu. Clinical applications of neurolinguistics in neurosurgery[J]. Front. Med., 2021, 15(4): 562-574.
[2] Hao Qu, Yu Zhao. Advances in tissue state recognition in spinal surgery: a review[J]. Front. Med., 2021, 15(4): 575-584.
[3] Baiwan Zhou, Dongmei An, Fenglai Xiao, Running Niu, Wenbin Li, Wei Li, Xin Tong, Graham J Kemp, Dong Zhou, Qiyong Gong, Du Lei. Machine learning for detecting mesial temporal lobe epilepsy by structural and functional neuroimaging[J]. Front. Med., 2020, 14(5): 630-641.
[4] Jiechao Ma, Yang Song, Xi Tian, Yiting Hua, Rongguo Zhang, Jianlin Wu. Survey on deep learning for pulmonary medical imaging[J]. Front. Med., 2020, 14(4): 450-469.
[5] Joseph JY Sung, Nicholas CH Poon. Artificial intelligence in gastroenterology: where are we heading?[J]. Front. Med., 2020, 14(4): 511-517.
[6] Yang Jiao, Zhan Zhang, Ting Zhang, Wen Shi, Yan Zhu, Jie Hu, Qin Zhang. Development of an artificial intelligence diagnostic model based on dynamic uncertain causality graph for the differential diagnosis of dyspnea[J]. Front. Med., 2020, 14(4): 488-497.
[7] Xiao-Yun Zhou, Yao Guo, Mali Shen, Guang-Zhong Yang. Application of artificial intelligence in surgery[J]. Front. Med., 2020, 14(4): 417-430.
[8] Wei Tian, Bo Liu, Da He, Yajun Liu, Xiaoguang Han, Jingwei Zhao, Mingxing Fan, International Society for Computer Assisted Orthopaedic Surgery. Guidelines for navigation-assisted spine surgery[J]. Front. Med., 2020, 14(4): 518-527.
[9] Yuyang Chen, Shu’an Zhang, Zhonghao Wu, Bo Yang, Qingquan Luo, Kai Xu. Review of surgical robotic systems for keyhole and endoscopic procedures: state of the art and perspectives[J]. Front. Med., 2020, 14(4): 382-403.
[10] Ke Sheng. Artificial intelligence in radiotherapy: a technological review[J]. Front. Med., 2020, 14(4): 431-449.
[11] Shujian Deng, Xin Zhang, Wen Yan, Eric I-Chao Chang, Yubo Fan, Maode Lai, Yan Xu. Deep learning in digital pathology image analysis: a survey[J]. Front. Med., 2020, 14(4): 470-487.
[12] Bowen Jiang, Tej D. Azad, Ethan Cottrill, Corinna C. Zygourakis, Alex M. Zhu, Neil Crawford, Nicholas Theodore. New spinal robotic technologies[J]. Front. Med., 2019, 13(6): 723-729.
[13] Cynthia Rajani, Wei Jia. Bile acids and their effects on diabetes[J]. Front. Med., 2018, 12(6): 608-623.
[14] Bin Yang, Yan Yu, Jing Chen, Yan Zhang, Ye Yin, Nan Yu, Ge Chen, Shifei Zhu, Haiyan Huang, Yongqun Yuan, Jihui Ai, Xinyu Wang, Kezhen Li. Possibility of women treated with fertility-sparing surgery for non-epithelial ovarian tumors to safely and successfully become pregnant---a Chinese retrospective cohort study among 148 cases[J]. Front. Med., 2018, 12(5): 509-517.
[15] Qing Pang, Hao Jin, Zhongran Man, Yong Wang, Song Yang, Zongkuang Li, Yimin Lu, Huichun Liu, Lei Zhou. Radical versus conservative surgical treatment of liver hydatid cysts: a meta-analysis[J]. Front. Med., 2018, 12(3): 350-359.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed