EasyVis: A Real-time 3D Rendering System for Laparoscopic Surgery Training Box

Introduction: Traditional laparoscopic surgery training utilizes a single camera, resulting in a limited Field of View (FoV) and no depth information. While box trainers remain the standard, virtual-reality (VR) trainers are emerging as an alternative for training due to their increased realism, ability to provide a sense of depth and the ability to record training data. Inspired by this, we have developed EasyVis - a real-time 3D visualization system for laparoscopic surgery training. Our goal is to improve the surgical training experience by providing surgeons/trainees with a larger FoV and a 3D reconstruction of the surgical scene to create a sense of depth. This is achieved by using an array of five cameras to view and reconstruct the surgical target in 3D and manipulate the viewing angle. This work results in an interactive and real-time 3D rendering system for laparoscopic surgery training. Our 3D rendering system focuses on generating views from any angle that can be chosen by the viewer, based on the five-camera 3D reconstruction.

Materials and

Methods: Figure 1 demonstrates the EasyVis training box system, comprising of an altered hood with uniform LED illumination, 5 cameras and 5 Raspberry Pi’s for video capture/streaming. The bean drop test is often used to assess technical skills, requiring the trainee to move a bean from a saucer, to a cup of different height with a hole, using a grasper. We used this task as the background test for the system. Since the training box background is stationary, while the surgical target (grasper) moves swiftly, we separated the 3D rendered scene into a static background and dynamic foreground. Consequently, the software framework was divided into offline (background) and online processing (foreground), shown in Figure 2. We first performed the background reconstruction using Structured Light (SL), Structure from Motion (SfM) and Multi-View Stereo (MVS). Color markers were used to locate the grasper keypoints during the online processing. Each new camera frame was processed separately to estimate the object’s 2D pose, 2D grasper direction and position, finger position, and the manipulated target 2D position. The correspondence between key points on different camera frames was use to establish the 3D states viz. the grasper position and orientation, the angles between the grasper fingers and the bean position. These 3D states, together with existing 3D surface model of the foreground objects, provides a 3D surface model of the current frame. Combined with the 3D background surface model developed during offline processing, a rendered stereo scene can be produced for any viewing angle using graphic rendering.

Results, Conclusions, and Discussions: Results and Discussions
We evaluated our system using visualization and performance metrics. Figure 3 compares the rendered result against the ground truth camera capture frame. We used the 5 camera views to reconstruct the virtual scene in 3D space, then back-projected the 3D model to the ground truth view. The visual similarity in the rendered image to the five ground truths images from different angles indicates that our system reconstructed the 3D virtual scene correctly. Figure 4 shows the 5 camera images used for the 3D reconstruction and the rendered novel view, demonstrating that our system can truly provide an additional sense of depth by changing the viewing angle of the reconstructed 3D space. We used the Back Projection Error (BPE) metric which back-projects the reconstructed 3D points to 2D coordinates, calculates the pixel distance between them and then compares it to pixel distance of the original detected corresponding 2D points in the camera capture image. This allows us to quantify the 3D reconstruction accuracy. Our system achieved a BPE of 3.193, which is a small proportion of the 640 by 480 pixel image (0.499% of the image width, 0.665% of the image height). Typically, 3D scene reconstruction requires dozens of camera views to achieve high accuracy. This BPE result indicates excellent accuracy with our 5-camera array system. In addition, we tested the processing time, from frame capture to rendering the 2D virtual view, achieving an average time of 16.7 milliseconds, which is less than 33 milliseconds and can be considered as achieving real time. By using parallel computing, the processing time was reduced significantly.

Conclusion
We have introduced a real-time 3D rendering framework for laparoscopic surgery training that gives surgeons an improved FoV and an additional sense of depth. The system takes five camera views as input and outputs a virtual view whose angle can be tailored by the user. Within 16.7 milliseconds, the system reconstructs the 3D virtual scene correctly and achieves an accurate reconstruction result. This framework lays the groundwork for 3D rendering of more complicated laparoscopic training scenarios and the real laparoscopic surgery.

Acknowledgements (Optional): This work was supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the U.S. National Institutes of Health (NIH) under award number R01EB019460.