Research Progresses in Laparoscopic Augmented Reality Navigation
WANG Tianmiao1, ZHANG Xiaohui1, ZHANG Xuebin2, WANG Junchen1
1. School of Mechanical Engineering and Automation, Beihang University, Beijing 100191, China;
2. Department of Urology, Peking Union Medical College Hospital, Beijing 100730, China
Abstract：This paper reviews various LARN (Laparoscopic augmented reality navigation) systems currently used in different clinical fields or reported in literatures, and the LARN systems are classified and described from three different perspectives:the source of the navigation data, the registration method, and the display mode. According to the data sources, from preoperative or intraoperative phases, the LARN systems are divided into LARN based on preoperative data and LARN based on intraoperative data respectively. Focusing on the introduction of registration technology, the research status and characteristics of each typical LARN system are described. Finally, the technical difficulties of LARN in the current clinical application are summarized and the development of the LARN is prospected.
[1] Kenngott H G, Wagner M, Nickel F, et al. Computer-assisted abdominal surgery:New technologies[J]. Langenbeck's Archives of Surgery, 2015, 400(3):273-281.
[2] Pomel C, Rouzier R, Pocard M, et al. Laparoscopic total pelvic exenteration for cervical cancer relapse[J]. Gynecologic Oncology, 2003, 91(3):616-618.
[3] Chen M H M, Murphy E A, Halpern V, et al. Laparoscopicassisted abdominal aortic aneurysm repair[J]. Surgical Endoscopy, 1995, 9(8):905-907.
[4] Zinser M J, Sailer H F, Ritter L, et al. A paradigm shift in orthognathic surgery? A comparison of navigation, computer-aided designed/computer-aided manufactured splints, and "classic" intermaxillary splints to surgical transfer of virtual orthognathic planning[J]. Journal of Oral and Maxillofacial Surgery, 2013, 71(12):2151.e1-2151.e21.
[5] Winne C, Khan M, Stopp F. Overlay visualization in endoscopic ENT surgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2011, 6(3):401-406.
[6] Dubach P, Bell B, Weber S, et al. Image-guided otorhinolaryngology[M]//Intraoperative Imaging and Image-Guided Therapy. New York, USA:Springer, 2014:845-856.
[7] Liu W P, Richmon J D, Sorger J M, et al. Augmented reality and cone beam CT guidance for transoral robotic surgery[J]. Journal of Robotic Surgery, 2015, 9(3):223-233.
[8] Mirota D J, Uneri A, Schafer S, et al. High-accuracy 3D imagebased registration of endoscopic video to C-arm cone-beam CT for image-guided skull base surgery[M]//Proceedings of SPIE, Vol.7964. Bellingham, USA:SPIE, 2011:No. UNSP 79640J.
[9] Wengert C, Cattin P C, Duff J M, et al. Markerless endoscopic registration and referencing[M]//Lecture Notes in Computer Science, Vol.4190. Berlin, Germany:Springer, 2006:816-823.
[10] Teber D, Guven S, Simpfendörfer T, et al. Augmented reality:A new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results[J]. European Urology, 2009, 56(2):332-338.
[11] Onda S, Okamoto T, Kanehira M, et al. Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality-based navigation system[J]. Journal of Hepato-Biliary-Pancreatic Sciences, 2014, 21(4):281-287.
[12] Kenngott H G, Neuhaus J, Mueller-Stich B P, et al. Development of a navigation system for minimally invasive esophagectomy[J]. Surgical Endoscopy and Other Interventional Techniques, 2008, 22(8):1858-1865.
[13] Okamoto T, Onda S, Matsumoto M, et al. Utility of augmented reality system in hepatobiliary surgery[J]. Journal of HepatoBiliary-Pancreatic Sciences, 2013, 20(2):249-253.
[14] Ukimura O, Nakamoto M, Sato Y, et al. Augmented reality for image-guided surgery in urology[M]//New Technologies in Urology. London, UK:Springer, 2010:215-222.
[15] Feichtinger M, Aigner R M, Santler G, et al. Case report:Fusion of positron emission tomography (PET) and computed tomography (CT) images for image-guided endoscopic navigation in maxillofacial surgery:Clinical application of a new technique[J]. Journal of Cranio-Maxillofacial Surgery, 2007, 35(6-7):322-328.
[16] Feichtinger M, Pau M, Zemann W, et al. Intraoperative control of resection margins in advanced head and neck cancer using a 3D-navigation system based on PET/CT image fusion[J]. Journal of Cranio-Maxillo-Facial Surgery, 2010, 38(8):589-594.
[17] Sherwinter D A. Transanal near-infrared imaging of colorectal anastomotic perfusion[J]. Surgical Laparoscopy Endoscopy & Percutaneous Techniques, 2012, 22(5):433-436.
[18] van der Vorst J R, Hutteman M, Mieog J S D, et al. Near-infrared fluorescence imaging of liver metastases in rats using indocyanine green[J]. Journal of Surgical Research, 2012, 174(2):266-271.
[19] Morita Y, Sakaguchi T, Unno N, et al. Detection of hepatocellular carcinomas with near-infrared fluorescence imaging using indocyanine green:Its usefulness and limitation[J]. International Journal of Clinical Oncology, 2013, 18(2):232-241.
[20] Ris F, Hompes R, Cunningham C, et al. Near-infrared (NIR) perfusion angiography in minimally invasive colorectal surgery[J]. Surgical Endoscopy and Other Interventional Techniques, 2014, 28(7):2221-2226.
[21] Golab M R, Breedon P J, Vloeberghs M. A wearable headset for monitoring electromyography responses within spinal surgery[J]. European Spine Journal, 2016, 25(10):3214-3219.
[22] Gao S, Mondal S, Zhu N, et al. A compact NIR fluorescence imaging system with goggle display for intraoperative guidance[C]//IEEE International Symposium on Circuits and Systems. Piscataway, USA:IEEE, 2015:1622-1625.
[23] 丛海波,张恩忠,余志平,等.增强现实技术联合3D打印技术行半椎体置换治疗椎体肿瘤一例[J].中国修复重建外科杂志, 2017, 31(11):1407-1408. Cong H B, Zhang E Z, Yu Z P, et al. A case of hemivertebrae displacement to treat spinal neoplasms based on augmented reality and 3D printing techniques[J]. Chinese Journal of Reparative and Reconstructive Surgery, 2017, 31(11):1407-1408.
[24] Wang S, Parsons M, StoneMcLean J, et al. Augmented reality as a telemedicine platform for remote procedural training[J]. Sensors, 2017, 17(10):No.2294.
[25] Kuhlemann I, Kleemann M, Jauer P, et al. Towards X-ray free endovascular interventions-Using HoloLens for on-line holographic visualization[J]. Healthcare Technology Letters, 2017, 4(5):184-187.
[26] Weiss C R, Marker D R, Fischer G S, et al. Augmented reality visualization using image-overlay for MR-guided interventions:System description, feasibility, and initial evaluation in a spine phantom[J]. American Journal of Roentgenology, 2011, 196(3):W305-W307.
[27] Zhang X R, Chen G W, Liao H E. High-quality see-through surgical guidance system using enhanced 3-D autostereoscopic augmented reality[J]. IEEE Transactions on Biomedical Engineering, 2017, 64(8):1815-1825.
[28] Zhang X R, Chen G W, Liao H E. A high-accuracy surgical augmented reality system using enhanced integral videography image overlay[C]//37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway, USA:IEEE, 2015:4210-4213.
[29] Wang J C, Suenaga H, Hoshi K, et al. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery[J]. IEEE Transactions on Biomedical Engineering, 2014, 61(4):1295-1304.
[30] Wu J R, Wang M L, Liu K C, et al. Real-time advanced spinal surgery via visible patient model and augmented reality system[J]. Computer Methods and Programs in Biomedicine, 2014, 113(3):869-881.
[31] Volonté F, Pugin F, Bucher P, et al. Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery:Not only a matter of fashion[J]. Journal of HepatoBiliary-Pancreatic Sciences, 2011, 18(4):506-509.
[32] Osorio A, Galan J A, Nauroy J, et al. Real time planning, guidance and validation of surgical acts using 3D segmentations, augmented reality projections and surgical tools video tracking[M]//Proceedings of SPIE, Vol.7625. Bellingham, USA:SPIE, 2010:No. UNSP 762529.
[33] Sugimoto M, Yasuda H, Koda K, et al. Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery[J]. Journal of Hepato-BiliaryPancreatic Sciences, 2010, 17(5):629-636.
[34] Marescaux J, Rubino F, Arenas M, et al. Augmented-realityassisted laparoscopic adrenalectomy[J]. Journal of the American Medical Association, 2004, 292(18):2214-2215.
[35] Nakamura K, Naya Y, Zenbutsu S, et al. Surgical navigation using three-dimensional computed tomography images fused intraoperatively with live video[J]. Journal of Endourology, 2010, 24(4):521-524.
[36] Marzano E, Piardi T, Soler L, et al. Augmented reality-guided artery-first pancreatico-duodenectomy[J]. Journal of Gastrointestinal Surgery, 2013, 17(11):1980-1983.
[37] Pessaux P, Diana M, Soler L, et al. Towards cybernetic surgery:Robotic and augmented reality-assisted liver segmentectomy[J]. Langenbeck's Archives of Surgery, 2015, 400(3):381-385.
[38] Suzuki N, Hattori A, Hashizume M. Benefits of augmented reality function for laparoscopic and endoscopic surgical robot systems[EB/OL].[2017-11-01]. http://pdfs.semanticscholar.org/db10/7e388ef2a825df21c3860b96a8b77cebfb68.pdf.
[39] Souzaki R, Ieiri S, Uemura M, et al. An augmented reality navigation system for pediatric oncologic surgery based on preoperative CT and MRI images[J]. Journal of Pediatric Surgery, 2013, 48(12):2479-2483.
[40] Hayashi Y, Misawa K, Oda M, et al. Clinical application of a surgical navigation system based on virtual laparoscopy in laparoscopic gastrectomy for gastric cancer[J]. International Journal of Computer Assisted Radiology and Surgery, 2016, 11(5):827-837.
[41] Hayashi Y, Misawa K, Hawkes D J, et al. Progressive internal landmark registration for surgical navigation in laparoscopic gastrectomy for gastric cancer[J]. International Journal of Computer Assisted Radiology and Surgery, 2016, 11(5):837-845.
[42] Su L M, Vagvolgyi B P, Agarwal R, et al. Augmented reality during robot-assisted laparoscopic partial nephrectomy:Toward real-time 3D-CT to stereoscopic video registration[J]. Urology, 2009, 73(4):896-900.
[43] Puerto-Souza G A, Mariottini G L. Toward long-term and accurate augmented-reality display for minimally-invasive surgery[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA:IEEE, 2013:5384-5389.
[44] Thompson S, Totz J, Song Y, et al. Accuracy validation of an image guided laparoscopy system for liver resection[M]//Proceedings of SPIE, Vol.9415. Bellingham, USA:SPIE, 2015:No.941509.
[45] Cash D M, Miga M I, Sinha T K, et al. Compensating for intraoperative soft-tissue deformations using incomplete surface data and finite elements[J]. IEEE Transactions on Medical Imaging, 2005, 24(11):1479-1491.
[46] Haouchine N, Dequidt J, Peterlik I, et al. Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[C]//12th IEEE and ACM International Symposium on Mixed and Augmented Reality. Piscataway, USA:IEEE, 2013:199-208.
[47] Plantèfeve R, Haouchine N, Radoux J P, et al. Automatic alignment of pre and intraoperative data using anatomical landmarks for augmented laparoscopic liver surgery[M]//Lecture Notes in Computer Science, Vol.8789. Berlin, Germany:Springer, 2014:58-66.
[48] Collins T, Pizarro D, Bartoli A, et al. Computer-assisted laparoscopic myomectomy by augmenting the uterus with preoperative MRI data[C]//IEEE International Symposium on Mixed and Augmented Reality. Piscataway, USA:IEEE, 2014:243-248.
[49] Song Y, Totz J, Thompson S, et al. Locally rigid, vessel-based registration for laparoscopic liver surgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2015, 10(12):1951-1961.
[50] Dagon B, Baur C, Bettschart V. Real-time update of 3D deformable models for computer aided liver surgery[C]//International Conference on Pattern Recognition. Piscataway, USA:IEEE, 2008:2177-2180.
[51] Nam W H, Kang D G, Lee D, et al. Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching[J]. Physics in Medicine and Biology, 2012, 57(1):69-91.
[52] Feuerstein M, Mussack T, Heining S M, et al. Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection[J]. IEEE Transactions on Medical Imaging, 2008, 27(3):355-369.
[53] Bernhardt S, Nicolau S A, Agnus V, et al. Automatic detection of endoscope in intraoperative CT image:Application to AR guidance in laparoscopic surgery[C]//IEEE International Symposium on Biomedical Imaging. Piscataway, USA:IEEE, 2014:563-567.
[54] Bernhardt S, Nicolau S A, Agnus V, et al. Automatic localization of endoscope in intraoperative CT image:A simple approach to augmented reality guidance in laparoscopic surgery[J]. Medical Image Analysis, 2016, 30:130-143.
[55] Mountney P, Fallert J, Nicolau S, et al. An augmented reality framework for soft tissue surgery[M]//Lecture Notes in Computer Science, Vol.8673. Berlin, Germany:Springer, 2014:423-431.
[56] Conrad C, Fusaglia M, Peterhans M, et al. Augmented reality navigation surgery facilitates laparoscopic rescue of failed portal vein embolization[J]. Journal of the American College of Surgeons, 2016, 223(4):e31-e34.
[57] Ieiri S, Uemura M, Konishi K, et al. Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device[J]. Pediatric Surgery International, 2012, 28(4):341-346.
[58] Marques B, Roy F, Haouchine N, et al. Framework for augmented reality in minimally invasive laparoscopic surgery[C]//17th International Conference on E-health Networking, Application & Services. Piscataway, USA:IEEE, 2015:22-27.
[59] Bernhardt S, Nicolau S A, Soler L, et al. The status of augmented reality in laparoscopic surgery as of 2016[J]. Medical Image Analysis, 2017, 37:66-90.
[60] Pizarro D, Bartoli A. Feature-based deformable surface detection with self-occlusion reasoning[J]. International Journal of Computer Vision, 2012, 97(1):54-70.
[61] Takeshita T, Nakajima Y, Kim M K, et al. 3D shape reconstruction endoscope using shape from focus[C]//4th International Conference on Computer Vision Theory and Applications. Setubal, Portugal:Institute for Systems and Technologies of Information, Control and Communication, 2009:411-416.
[62] Cohen D, Mayer E, Chen D, et al. Augmented reality image guidance in minimally invasive prostatectomy[M]//Prostate Cancer Imaging. Computer-Aided Diagnosis, Prognosis, and Intervention. Berlin, Germany:Springer, 2010:101-110.
[63] Zhang X, Wang T, Kobayashi E, et al. Kidney deformation evaluation for augmented reality surgical navigation of laparoscopic partial nephrectomy:An animal experiment[J]. International Journal for Computer Assisted Radiology and Surgery, 2017, 12(S1):S111-S113.
[64] Amir-Khalili A, Nosrati M S, Peyrat J M, et al. Uncertaintyencoded augmented reality for robot-assisted partial nephrectomy:A phantom study[M]//Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions. Berlin, Germany:Springer, 2013:182-191.
[65] Vagvolgyi B, Su L M, Taylor R, et al. Video to CT registration for image overlay on solid organs[C]//Proceedings of Augmented Reality in Medical Imaging and Augmented Reality in Computer-Aided Surgery. 2008:78-86.
[66] Myronenko A, Song X. Point set registration:Coherent point drift[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(12):2262-2275.
[67] Chen E C S, McLeod A J, Baxter J S H, et al. Registration of 3D shapes under anisotropic scaling[J]. International Journal of Computer Assisted Radiology and Surgery, 2015, 10(6):867-878.
[68] Sotiras A, Davatzikos C, Paragios N. Deformable medical image registration:A survey[J]. IEEE Transactions on Medical Imaging, 2013, 32(7):1153-1190.
[69] Collins T, Pizarro D, Bartoli A, et al. Realtime wide-baseline registration of the uterus in laparoscopic videos using multiple texture maps[M]//Lecture Notes in Computer Science, Vol.8090. Berlin, Germany:Springer, 2013:162-171.
[70] Haouchine N, Roy F, Untereiner L, et al. Using contours as boundary conditions for elastic registration during minimally invasive hepatic surgery[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA:IEEE, 2016:495-500.
[71] Benincasa A B, Clements L W, Herrell S D, et al. Feasibility study for image-guided kidney surgery:Assessment of required intraoperative surface for accurate physical to image space registrations[J]. Medical Physics, 2008, 35(9):4251-4261.
[72] Simpson A L, Dumpuri P, Jarnagin W R, et al. Model-assisted image-guided liver surgery using sparse intraoperative data[M]//Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. Berlin, Germany:Springer, 2012:7-40.
[73] San Jose Estepar R, Westin C F, Vosburgh K G. Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures[J]. International Journal of Computer Assisted Radiology and Surgery, 2009, 4(6):549-560.
[74] Ukimura O, Gill I S. Imaging-assisted endoscopic surgery:Cleveland clinic experience[J]. Journal of Endourology, 2008, 22(4):803-809.
[75] Lange T, Papenberg N, Heldmann S, et al. 3D ultrasound-CT registration of the liver using combined landmark-intensity information[J]. International Journal of Computer Assisted Radiology and Surgery, 2009, 4(1):79-88.
[76] Schwabenland I, Sunderbrink D, Nollert G, et al. Flat-panel CT and the future of OR imaging and navigation[J]. Imaging and Visualization in the Modern Operating Room. New York, USA:Springer, 2015:89-106.
[77] Tsutsumi N, Tomikawa M, Uemura M, et al. Image-guided laparoscopic surgery in an open MRI operating theater[J]. Surgical Endoscopy and Other Interventional Techniques, 2013, 27(6):2178-2184.
[78] Baumhauer M, Simpfendoerfer T, Mueller-Stich B P, et al. Soft tissue navigation for laparoscopic partial nephrectomy[J]. International Journal of Computer Assisted Radiology and Surgery, 2008, 3(3-4):307-314.
[79] Wild E, Teber D, Schmid D, et al. Robust augmented reality guidance with fluorescent markers in laparoscopic surgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2016, 11(6):899-907.
[80] Feuerstein M, Reichl T, Vogel J, et al. Magneto-optical tracking of flexible laparoscopic ultrasound:Model-based online detection and correction of magnetic tracking errors[J]. IEEE Transactions on Medical Imaging, 2009, 28(6):951-967.
[81] Oktay O, Zhang L, Mansi T, et al. Biomechanically driven registration of pre-to intra-operative 3D images for laparoscopic surgery[M]//Lecture Notes in Computer Science, Vol.8150. Berlin, Germany:Springer, 2013:1-9.
[82] Bano J, Hostettler A, Nicolau S A, et al. Simulation of pneumoperitoneum for laparoscopic surgery planning[M]//Lecture Notes in Computer Science, Vol.7510. Berlin, Germany:Springer, 2012:91-98.
[83] Choi S W, Kim H C, Kang H S, et al. A haptic augmented reality surgeon console for a laparoscopic surgery robot system[C]//13th International Conference on Control, Automation and Systems. Piscataway, USA:IEEE, 2013:355-357.
[84] Haouchine N, Dequidt J, Peterlik I, et al. Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA:IEEE, 2014:4121-4126.
[85] Baumhauer M, Feuerstein M, Meinzer H P, et al. Navigation in endoscopic soft tissue surgery:Perspectives and limitations[J]. Journal of Endourology, 2008, 22(4):751-766.
[86] Markelj P, Tomazvic D, Likar B, et al. A review of 3D/2D registration methods for image-guided interventions[J]. Medical Image Analysis, 2012, 16(3):642-661.
[87] Stoyanov D, Yang G Z. Soft tissue deformation tracking for robotic assisted minimally invasive surgery[C]//Annual International Conference of the IEEE-Engineering-in-Medicine-andBiology-Society. Piscataway, USA:IEEE, 2009:254-257.
[88] Mountney P, Yang G Z. Motion compensated SLAM for image guided surgery[M]//Lecture Notes in Computer Science, Vol.6362. Berlin, Germany:Springer, 2010:496-504.
[89] Maier-Hein L, Mountney P, Bartoli A, et al. Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery[J]. Medical Image Analysis, 2013, 17(8):974-996.
[90] Dixon B J, Daly M J, Chan H, et al. Surgeons blinded by enhanced navigation:The effect of augmented reality on attention[J]. Surgical Endoscopy and Other Interventional Techniques, 2013, 27(2):454-461.
