Poster AA10 - Adapting Neural Networks to Enable Preoperative Virtual Reality Visualization of Calcaneus Fractures

Introduction: Virtual reality (VR) offers tremendous potential for surgical education. However, image processing hurdles have prevented its practical adoption as a clinically relevant tool. Current VR simulators relying on manually crafted models do not allow users to view patient-specific anatomy for preoperative planning. This barrier is especially relevant when considering calcaneal fractures, given the tremendous amount of case-to-case variability that cannot be comprehensively captured by a preconceived set of standardized models. To this end, we propose adapting a neural network-based machine learning method to automate the accurate segmentation of calcaneal CT scans for use in a VR simulator for patient-specific preoperative planning.

Materials and

Methods: We manually segmented 200 axial images parsed from the CT scans of 3 patients with calcaneus fractures using Amira. We adapted the U-Net, an existing neural network framework, to perform automatic segmentation of our CT scans using 70 training images. We then validated the neural network’s accuracy for the remaining images by comparing them to manually segmented ground truth data and calculating the dice similarity coefficient (DSC) and intersection over union (IOU). Finally, we reconstructed the segmented scans and exported them to a Unity interface that we designed to explore fractures in the 3-dimensional VR domain.

Results, Conclusions, and Discussions: Our U-Net based segmentation of the patients’ non-overlapping CT scans yielded a DSC and IOU of 0.83 ± 0.06 and 0.71 ± 0.08 respectively, suggesting accurate and consistent results. Qualitative examination of the segmented scans reveals clear demarcation of cortical bone but increased noise in the trabecular regions along with mild narrowing of fracture spaces, correlating with intensity differences in the raw CT scans. The segmented scans were successfully reconstructed into a VR model with clearly identifiable fracture lines, as shown below (Fig 1).

Adapting a machine learning algorithm for the segmentation of calcaneus CT scans allows for high throughput processing that makes VR accessible for clinical use. The qualitative and quantitative validation results suggest that neural networks enable us to perform accurate automatic segmentation of these scans. With continuing improvements in machine learning-based image processing and VR technology, our future VR simulators may allow us to manipulate bone fragments and rehearse an entire surgery on a particular patient’s anatomy.

Acknowledgements (Optional):