Development of Holoeyes Holographic Image-Guided Surgery and Telemedicine System: Clinical Benefits of Extended Reality (Virtual Reality, Augmented Reality, Mixed Reality), The Metaverse, and Artificial Intelligence in Surgery with a Systematic Review

Article Sidebar

PDF Download
Published Jul 6, 2023
DOI: https://doi.org/10.18103/mra.v11i7.1.4045

Downloads

Download data is not yet available.

Submit your own article

Register as an author to reserve your spot in the next issue of the Medical Research Archives.

Author Registration

Join the Society

The European Society of Medicine is more than a professional association. We are a community. Our members work in countries across the globe, yet are united by a common goal: to promote health and health equity, around the world.

Join Europe’s leading medical society and discover the many advantages of membership, including free article publication.

Membership

Main Article Content

Maki Sugimoto
Innovation Lab, Teikyo University Okinaga Research Institute, Tokyo, Japan; Department of Surgery, Teikyo University School of Medicine, Tokyo, Japan
Takuya Sueyoshi
Innovation Lab, Teikyo University Okinaga Research Institute, Tokyo, Japan

Abstract

Image-guided surgery using XR (extended reality: VR/AR/MR) technology has the potential to revolutionize the field of surgery by improving surgical accuracy, reducing procedure time, and enhancing communication and collaboration among the surgical team. We have developed a web-based system, Holoeyes, integrating XR, AI, and metaverse technology to facilitate holographic image-guided surgery. Holoeyes extracts organ shape data from CT or MRI scans and renders them with positional information to obtain X, Y, and Z coordinates. These coordinates are then converted into polygonal information for use in XR technology. The medical device, Holoeyes MD, was developed to create XR applications for surgical planning and navigation. It provides an immersive experience for the surgical team, improving both accuracy and efficiency. The integration of the metaverse in surgery allows for spatial conferencing and review of training, and the avatars replicate the hand and eye movements of the actual surgical procedure. Our Holoeyes system has already been utilized in numerous institutions for pre-and post-operative conferences, surgical planning, and surgical records, with multiple people wearing the headset and sharing information about the pathology, extent of resection, and layers of dissection from all directions. We conducted a systematic review of the literature to investigate the effectiveness of Holoeyes, focusing on the use of XR and the metaverse in surgery. We believe that Holoeyes has the potential to become an indispensable tool in the field of surgery, and we encourage further research and development in this field.

Keywords: Extended Reality (XR), Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Artificial Intelligence (AI), metaverse, Holoeyes, OsiriX

Article Details

How to Cite
SUGIMOTO, Maki; SUEYOSHI, Takuya. Development of Holoeyes Holographic Image-Guided Surgery and Telemedicine System: Clinical Benefits of Extended Reality (Virtual Reality, Augmented Reality, Mixed Reality), The Metaverse, and Artificial Intelligence in Surgery with a Systematic Review. Medical Research Archives, [S.l.], v. 11, n. 7.1, july 2023. ISSN 2375-1924. Available at: <https://esmed.org/MRA/mra/article/view/4045>. Date accessed: 15 may 2024. doi: https://doi.org/10.18103/mra.v11i7.1.4045.
ABNT APA BibTeX CBE EndNote - EndNote format (Macintosh & Windows) MLA ProCite - RIS format (Macintosh & Windows) RefWorks Reference Manager - RIS format (Windows only) Turabian
Issue
Vol 11 No 7.1 (2023): July Issue, Vol.11, Issue 7.1
Section
Review Articles

The Medical Research Archives grants authors the right to publish and reproduce the unrevised contribution in whole or in part at any time and in any form for any scholarly non-commercial purpose with the condition that all publications of the contribution include a full citation to the journal as published by the Medical Research Archives.

 

References

1. Sugimoto M. Cloud XR (Extended Reality: Virtual Reality・Augmented Reality・Mixed Reality) and 5G networks for holographic medical image-guided surgery and telemedicine. Multidisciplinary Computational Anatomy - Principles and Clinical Application of MCA-based Medicine. Editor: Hashizume M. Springer, Singapore. 2021; 381-338.
2. Sugimoto M. Extended Reality (XR:VR/AR/MR), 3D Printing, Holography, A.I., Radiomics, and Online VR Tele-Medicine for Precision Surgery. Surgery and Operating Room Innovation. Editor: Takenoshita S, Yasuhara H. Springer, Singapore. 2020; 65-70.
3. Sugimoto M. Augmented Tangibility Surgical Navigation Using Spatial Interactive 3-D Hologram zSpace with OsiriX and Bio-Texture 3-D Organ Modeling. IEEE 2015 International Conference on Computer Application Technologies (CCATS 2015). 2015; 189-194.
4. Aoki T, Koizumi T, Sugimoto M, et al. Holography-guided percutaneous puncture technique for selective near-infrared fluorescence-guided laparoscopic liver resection using mixed-reality wearable spatial computer. Surgical Oncology. 2020; 35: 476-477.
5. Saito Y, Sugimoto M, Imura S, et al. Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery. Ann Surg. 2020; 271(1): e4-e7.
6. Saito Y, Shimada M, Sugimoto M, et al. Current topics of simulation and navigation in hepatectomy. Ann Gastroenterol Surg. 2021; 6(2): 190-196.
7. Saito Y, Sugimoto M, Morine Y, et al. Intraoperative support with three-dimensional holographic cholangiography in hepatobiliary surgery. Langenbecks Arch Surg. 2022; 407(3): 1285-1289.
8. Kitagawa M, Sugimoto M, Umezawa A, et al. Clinical Benefit of Mixed Reality Holographic Cholangiography for Image-Guided Laparoscopic Cholecystectomy. Surgery and Operating Room Innovation. (Takenoshita S, Yasuhara H ed.) Springer Tokyo Japan. 2020; 107-112.
9. Kitagawa M, Sugimoto M, Haruta H, et al. Intraoperative holography navigation using a mixed-reality wearable computer during laparoscopic cholecystectomy. Surgery. 2022; 171(4): 1006-1013.
10. Morimoto T, Hirata H, Ueno M, et al. Digital Transformation Will Change Medical Education and Rehabilitation in Spine Surgery. Medicina (Kaunas). 2022; 58(4): 508.
11. Morimoto T, Kobayashi T, Hirata H, et al. XR Technology in Spine Medicine: Status Quo and Quo Vadis. J. Clin. Med. 2022; 11: 470.
12. Aoyama R, Anazawa U, Hotta H, et al. A Novel Technique of Mixed Reality Systems in the Treatment of Spinal Cord Tumors. Cureus. 2022; 14(3): e23096.
13. Aoyama R, Anazawa U, Hotta H, et al. Augmented Reality Device for Preoperative Marking of Spine Surgery Can Improve the Accuracy of Level Identification. Spine Surg Relat Res. 2021; 6(3): 303-309.
14. Aoyama R, Anazawa U, Hotta H, et al. The Utility of Augmented Reality in Spinal Decompression Surgery Using CT/MRI Fusion Image. Cureus. 2021; 13(9): e18187.
15. Sakai D, Joyce K, Sugimoto M, et al. Augmented, virtual and mixed reality in spinal surgery: A real-world experience. J Orthop Surg. 2020; 28(3): 2309499020952698.
16. Koyachi M, Sugahara K, Tachizawa K, et al. Mixed-reality and computer-aided design/computer-aided manufacturing technology for mandibular reconstruction: a case description. Quant Imaging Med Surg. 2023. doi: 10.21037/qims-22-1118
17. Sugahara K, Koyachi M, Tachizawa K, et al. Using Mixed Reality and CAD/CAM Technology for Treatment of Maxillary Non-Union after Le Fort I Osteotomy: A Case description. Quant Imaging Med Surg. 2023; 13(2): 1190-1199.
18. Koyachi M, Sugahara K, Odaka K, et al. Accuracy of Le Fort I osteotomy with combined computer-aided design/computer-aided manufacturing technology and mixed reality. Int J Oral Maxillofac Surg. 2021; 50(6): 782-790.
19. Sugahara K, Koyachi M, Koyama Y, et al. Mixed reality and three-dimensional printed models for resection of maxillary tumor: a case report. Quant Imaging Med Surg. 2020; 11(5): 2187-2194.
20. Sato Y, Sugimoto M, Tanaka Y, et al. Holographic image-guided thoracoscopic surgery: possibility of usefulness for esophageal cancer patients with abnormal artery. Esophagus. 2020; 17(4): 508-511.
21. Tokunaga T, Sugimoto M, Saito Y, et al. Intraoperative holographic image-guided surgery in a transanal approach for rectal cancer. Langenbecks Arch Surg. 2022; 407(6): 2579-2584.
22. Ryu S, Kitagawa T, Goto K, et al. Intraoperative Holographic Guidance Using Virtual Reality and Mixed Reality Technology During Laparoscopic Colorectal Cancer Surgery. Anticancer Res. 2022; 42(10): 4849-4856.
23. Hosoya K, Komachi T, Sugimoto M, et al. Using Virtual Reality to Teach Sinus Anatomy. Int J Pract Otolaryngol. 2022; 5(1): e45–e50.
24. Yamazaki A, Ito T, Sugimoto M, et al. Patient-specific virtual and mixed reality for immersive, experiential anatomy education and for surgical planning in temporal bone surgery. Auris Nasus Larynx. 2021; 48(6): 1081-1091.
25. Ito K, Sugimoto M, Tsunoyama T, et al. A trauma patient care simulation using extended reality technology in the hybrid emergency room system. J Trauma Acute Care Surg. 2021; 90(5): e108-e112.
26. Tachibana K, Kikuchi K, Sugimoto M, et al. Virtual Reality Simulation for Minimally Invasive Coronary Artery Bypass Grafting With Aortic No-Touch Total Arterial Grafting Technique. Innovations (Phila). 2022; 17(5): 430-437.
27. Yoshida S, Sugimoto M, Fukuda S, et al. Mixed reality computed tomography‐based surgical planning for partial nephrectomy using a head‐mounted holographic computer. Int J Urol. 2019; 26(6): 681-682.
28. Rosset A, Spadola L, Ratib O. OsiriX: An Open-Source Software for Navigating in Multidimensional DICOM Images. J Digit Imaging. 2004; 17(3): 205.
29. The website of Holoeyes (May 1st, 2023). https://holoeyes.jp/en/