Reconstruction of a 3D Virtual Colon Structure and Camera Motion for Screening Colonoscopy
Main Article Content
Optical colonoscopy is the gold standard for detection and prevention of Colorectal cancer. However, any lesion outside the field of view of the camera during colonoscopy is missed, and can develop from a polyp into a cancer or from a small cancer into a large, metastatic cancer. Hence, viewing the colon mucosa as much as possible is desirable. The ultimate goal is to use the reconstruction results to mark the area likely unobserved during colonoscopy where polyps may be hidden.
Materials and Methods:
This paper presents a novel method for reconstructing (i) a 3D virtual colon structure from a sequence of 2D colonoscopic images and (ii) the endoscope camera navigation path. Unlike existing work that focus on reconstruction of accurate colon surface for a computer-aided surgery, this work focuses on estimating the alignment of the colon haustral folds, thickness and protrusion of the folds, and the camera path for computer-aided screening colonoscopy to inspect as much as possible of the colon mucosa.
On endoscopy video of a synthetic colon model, we achieved at least 73% and 90% accuracy in estimating the directions of camera translation and rotation motions, respectively. The average percentage of the depth error is about 10% (8.5/85.8) of the average depth of all the folds seen in all the image sequences. The average percentage of the circumference error is about 10% (17/178.2) of the average circumference of all the folds seen in all the image sequences. The results are promising and give further insight to address this challenging problem.
We present the work on reconstruction of a colon structure and endoscope motion from a colonoscopic image sequence for screening colonoscopy. We overcame several challenges such as the scale difference of consecutive reconstructed models, estimating closed fold contours from tracked fold edges, and registering the models. As future work, we will investigate a multi-frame registration approach to further reduce the depth and circumference estimation errors by registering nearby frames that are more similar in fold shape together.
The Medical Research Archives grants authors the right to publish and reproduce the unrevised contribution in whole or in part at any time and in any form for any scholarly non-commercial purpose with the condition that all publications of the contribution include a full citation to the journal as published by the Medical Research Archives.
2. American Cancer Society Colorectal Cancer Facts and Figures. 2016.
3. Mura, C., et al., Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Computer and Graphics, 2014. 44: p. 20-32.
7. Hajder, L. and D. Chetverikov, Weak-perspective structure from motion for strongly contaminated data. Pattern Recognition Letters, 2006. 27(14): p. 1581-1589.
8. Hajder, L., D. Chetverikov, and I. Vajk, Robust structure from motion under weak perspective. 2nd International Symposium on 3d Data Processing, Visualization, and Transmission, Proceedings, 2004: p. 828-835.
9. Hajder, L., A. Pernek, and C. Kazo, Weak-perspective structure from motion by fast alternation. Visual Computer, 2011. 27(5): p. 387-399.
10. Kazo, C. and L. Hajder, Rapid Weak-perspective Structure from Motion with Missing Data. 2011 Ieee International Conference on Computer Vision Workshops (Iccv Workshops), 2011.
11. Taylor, C.J. and D.J. Kriegman, Structure and Motion from Line Segments in Multiple Images. Ieee Transactions on Pattern Analysis and Machine Intelligence, 1995. 17(11): p. 1021-1032.
12. Tomasi, C. and T. Kanade, Shape and Motion from Image Streams - a Factorization Method. Proceedings of the National Academy of Sciences of the United States of America, 1993. 90(21): p. 9795-9802.
13. Zhang, L., et al., Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo. Ninth Ieee International Conference on Computer Vision, Vols I and Ii, Proceedings, 2003: p. 618-625.
14. Chandraker, M. What Camera Motion Reveals about Shape with Unknown BRDF. in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. 2014.
15. Cremers, D. and K. Kolev, Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2011. 33(6): p. 1161-74.
16. Furukawa, Y. and J. Ponce, Accurate, Dense, and Robust Multiview Stereopsis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2010. 32(8): p. 1362-1376.
17. Ahmed, A.H. and A.A. Farag. Shape from shading under various imaging conditions. in IEEE Conf. on Computer Vision and Pattern Recognition. 2007. MN.
18. Ahmed, A.H. and A.A. Farag. Shape from shading for hybrid surfaces. in IEEE Conf. on Image Processing (ICIP 2007). 2007. Atlanta, GA.
19. Christou, C.G. and J.J. Koenderink, Light source dependence in shape from shading. Vision Research, 1997. 37(11): p. 1441-1449.
20. Horn, B.K.P., Shape from Shading: A Method for Obtaining the Shape of a Smooth Opaque Object from One View. 1970, MIT.
21. Horn, B.K.P. and M.J. Brooks, The Variational Approach to Shape from Shading. Computer Vision Graphics and Image Processing, 1986. 33(2): p. 174-208.
22. Hougen, D.R. and N. Ahuja, Estimation of the Light-Source Distribution and Its Use in Integrated Shape Recovery from Stereo and Shading. Fourth International Conference on Computer Vision : Proceedings, 1993: p. 148-155.
23. Ikeda, O., Shape-from-shading algorithm for oblique light source. Advances in Visual Computing, Proceedings, Pt 2, 2007. 4842: p. 357-366.
24. Ikeuchi, K. and B.K.P. Horn, Numerical Shape from Shading and Occluding Boundaries. Artificial Intelligence, 1981. 17(1-3): p. 141-184.
25. Kondo, S., et al., Shape and Source from Shading Using Zero Crossings. International Conference on Pattern Recognition, 1992. I: p. 534-537.
26. Lee, C.H. and A. Rosenfeld, Improved Methods of Estimating Shape from Shading Using the Light-Source Coordinate System. Artificial Intelligence, 1985. 26(2): p. 125-143.
27. Ming, X., R.C. Zhao, and P. Maria, Solving self-shadow problem of shape from shading in light source projected system. Proceedings of the 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004: p. 334-337.
28. Oliensis, J. and P. Dupuis, A Global Algorithm for Shape from Shading. Fourth International Conference on Computer Vision : Proceedings, 1993: p. 692-701.
29. Prados, E. and O. Faugeras, Shape from shading: a well-posed problem? 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol 2, Proceedings, 2005: p. 870-877.
30. Rashid, H.U. and P. Burger, Differential Algorithm for the Determination of Shape from Shading Using a Point Light-Source. Image and Vision Computing, 1992. 10(2): p. 119-127.
31. Sato, T. and K. Hosokawa, Visual rather than proprioceptive information contribute more to shape-from-shading when the light-source was actively moved. Perception, 2009. 38: p. 33-33.
32. Visentini-Scarzanella, M.S., D.; Guang-Zhong Yang. Metric depth recovery from monocular images using Shape-from-Shading and specularities. in Image Processing (ICIP). 2012.
33. Wada, T., H. Ukida, and T. Matsuyama, Shape from shading with interreflections under a proximal light source: Distortion-free copying of an unfolded book. International Journal of Computer Vision, 1997. 24(2): p. 125-135.
34. Wu, C., S.G. Narasimhan, and B. Jaramaz, A multi-image shape-from-shading framwork for near-lighting perspective endoscopes. International Journal of Computer Vision, 2010. 86(2-3): p. 211-228.
35. Xu, M., R.C. Zhao, and M. Petrou, Solving self-shadow problem of shape from shading in light source projected system. 2004 7th International Conference on Signal Processing Proceedings, Vols 1-3, 2004: p. 1235-1238.
36. Yang, L. and J.-Q. Han, A Perspective Shape-from-Shading Method using Fast Sweeping Numerical Scheme. Optica Applicata, 2008. 38(2): p. 387-398.
37. Zhang, L., et al., A Unified Framework for Document Restoration using In-painting and Shape-from-Shading. Pattern Recognition, 2009. 42(11): p. 2961-2978.
38. Zhang, R., et al., Shape from Shading: A Survey. IEEE Trans on Pattern Analysis and Machine Intelligence, 1999. 21(8): p. 690-706.
39. Zheng, Q. and R. Chellappa. Estimation of Illuminant Direction, Albedo, and Shape from Shading. in IEEE Computer Vision and Pattern Recognition. 1991.
40. Deguchi, K. Shape Reconstruction from Endoscope Image by its Shadings. in IEEE/SICE/RSJ Int’l Conf. on Multisensor Fusion and Integration for Intelligent Systems. 1996.
41. Koppel, D., et al. Toward Automated Model Building from Video in Computer-Assisted Diagnoses in Colonoscopy. in Proc. of SPIE Medical Imaging Conference. 2007. San Diego, CA, USA.
42. Kaufman, A. and J.N. Wang, 3D surface reconstruction from endoscopic videos. Visualization in Medicine and Life Sciences, 2008: p. 61-+.
43. Sun, D., et al., Surface Reconstruction from Tracked Endoscopic Video Using the Structure from Motion Approach, in Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions: 6th International Workshop, MIAR 2013 and 8th International Workshop, AE-CAI 2013, Held in Conjunction with MICCAI 2013, Nagoya, Japan, September 22, 2013. Proceedings, H. Liao, et al., Editors. 2013, Springer Berlin Heidelberg: Berlin, Heidelberg. p. 127-135.
44. Qingyu Zhao, True Price, Stephen Pizer, Marc Niethammer, Ron Alterovitz, and Julian Rosenman. "The Endoscopogram: a 3D Model Reconstructed from Endoscopic Video Frames." International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016.
45. Saad Nadeem and Arie Kaufman. “Depth Reconstruction and Computer-Aided Polyp Detection in Optical Colonoscopy Video Frames,” arXiv preprint arXiv:1609.01329, 2016.
46. Umeyama, S., Least-Squares Estimation of Transformation Parameters between 2 Point Patterns. Ieee Transactions on Pattern Analysis and Machine Intelligence, 1991. 13(4): p. 376-380.
47. Marino, J., F. Qiu, and A. Kaufman. Registration of Virtual and Optical Colonoscopy Views. in Proc. of MICCAI 2008 Workshop: Computational and Visualization Challenges in the New Era of Virtual Colonoscopy. 2008.
48. Marino, J., F. Qiu, and A. Kaufman, Virtually Assisted Optical Colonoscopy, in SPIE Medical Imaging 2008: Physiology, Function, and Structure from Medical Images. 2008: San Diego, CA, USA. p. 69160J-1.
49. Liu, J., K.R. Subramanian, and T.S. Yoo, A robust method to track colonoscopy videos with non-informative images. International Journal of Computer Assisted Radiology and Surgery, 2013. 8(4): p. 575-592.
50. Zhou, J., et al. Circular generalized cylinder fitting for 3D reconstruction in endoscopic imaging based on MRF. in Proc. of Computer Vision and Pattern Recognition Workshops. 2008. Anchorage, Alaska, USA.
51. Sederberg, T. BYU Bézier curves.
52. Lucas, B.D. and T. Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. in IJCAI'81 Proceedings of the 7th international joint conference on Artificial intelligence. 1981. San Francisco, CA, USA.
53. Tektonidis, M. and K. Rohr, Diffeomorphic Multi-Frame Non-Rigid Registration of Cell Nuclei in 2D and 3D Live Cell Images. IEEE Transactions on Image Processing, 2017. PP(99): p. 1-13.
54. Yang, J., et al., Sparse non-rigid registration of 3D shapes. Proceedings of the Eurographics Symposium on Geometry Processing, 2015: p. 89 - 99.
55. Ge, S. and G. Fan, Sequential non-rigid point registration for 3D human pose tracking. IEEE International Conference on Image Processing (ICIP), 2015. 1105 - 1109.
56. Aktar, N., J. Alam, and M. Pickering, A Non-Rigid 3D Multi-Modal Registration Algorithm Using Partial Volume Interpolation and the Sum of Conditional Variance. 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA), 2014: p. 1 - 7.
57. Matinfar, B. and L. Zagrochev, Non-rigid Registration of 3D Ultrasound Images Using Model-Based Segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014: p. 323 - 328.
58. Serradell, E., et al., Robust non-rigid registration of 2D and 3D graphs. IEEE Conference on Computer Vision and Pattern Recognition, 2012: p. 996 - 1003.
59. Cheng, Z.-Q., et al., Non-rigid Registration in 3D Implicit Vector Space. 2010 Shape Modeling International Conference, 2010: p. 37 - 46.
60. Rui Wang, True Price, Qingyu Zhao, Jan-Michael Frahm, Julian Rosenman, and Stephen Pizer. "Improving 3D Surface Reconstruction from Endoscopic Video via Fusion and Refined Reflectance Modeling." SPIE Medical Imaging, pp. 101330B-101330B, 2017.