Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Device Technologies and Biomedical Robotics

Device Technologies and Biomedical Robotics - Poster Session C

Poster I19 - Medical Augmented Reality Toolkit for Visualization and Interaction

Friday, October 25, 2024

10:00 AM - 11:00 AM EST

Location: Exhibit Hall E, F & G

Presenting Author(s)

Johnathan Mo

Student
NIH, Columbia University, University of Michigan Ann Arbor
Palo Alto, United States

Primary Investigator(s)

RF

Raisa Z. Freidlin, DSc

Computer Engineer
National Institutes of Health, NIBIB
Bethesda, Maryland, United States

Introduction: Recently, Extended Reality (XR) technologies have experienced significant growth in popularity. Augmented Reality (AR), a sub field of XR, overlays computer-generated content onto a user's view of their physical environment. While AR technologies are well-established in various industries, they are now emerging as promising tools in healthcare (Zollmann et al). The unique features of AR, such as the visualization and intuitive interaction with three-dimensional medical data, are highly relevant in this field. Consequently, AR technologies can assist during surgery, preoperative planning, training, and patient education.  

Advancements in Artificial Intelligence (AI) speech recognition have demonstrated significant improvements in accuracy and natural language understanding, making voice interactions more reliable and accessible. AI speech recognition enhances AR by enabling intuitive, hands-free interactions. It allows users to navigate the User Interface (UI) and interact with the AR environment using natural language voice commands, making the experience seamless and user-friendly. 

Furthermore, allowing multiple users to interact with the same virtual content simultaneously, whether in the same room or across the world, enhances understanding and collaboration in real-time. This shared experience helps users interpret the computer-generated context more effectively, facilitating communication and teamwork in various applications such as surgical planning, training, and education.
In our work, we investigated the feasibility of combining multi-user capability with voice commands in the context of healthcare.

Materials and

Methods: HoloLens 2 (Figure 1) is the second generation of Microsoft’s see-through head-mounted display. Our AR toolkit was developed using the multiplatform Unity graphics engine, version 2022.3.32f1 (Unity Technologies), and the Integrated Development Environment Visual Studio 2022 (Microsoft). This setup supports code development with object-oriented programming language C#, application building, and deployment to the HoloLens 2. The Mixed Reality Toolkit 3 framework (Microsoft) provided User Experience components such as gaze, hand, and head tracking, along with UI controls.

Hands-free interaction was achieved through voice commands powered by natural language processing. Speech recognition was performed using Microsoft Azure Speech AI. The resulting transcription was integrated with a custom set of instructions, which consisted of natural language text describing the callable methods and the objects in the scene. The combined prompt was passed to OpenAI’s Completions API and a response was received in JSON format. Using the Newtonsoft JSON package, the response was parsed, and the extracted methods were called accordingly.

For multi-user capability support, we utilized Photon Fusion (Photon), a high-end state transfer net code SDK for online multiplayer experiences. To our knowledge, this is the first time Photon Fusion has been used for a HoloLens 2 application. We utilized shared mode for the Photon server to synchronize object positions across clients. Through remote procedure calls, we designed a user-friendly interface that visually indicated which player was interacting with each object depending on the player’s unique assigned color.

Results, Conclusions, and Discussions: Figure 1 presents various views of the HoloLens 2, a see-through head-mounted AR display.

The voice command process, illustrated in Figure 2, shows an image before the command is called, the natural language command, and the resulting manipulation. Three examples demonstrate the system’s flexibility, including manipulating multiple objects, distinguishing between objects using voice, and handling nuanced commands.

As shown in Figure 3, multi-user capabilities allow multiple users in different physical environments to observe the same content while “standing” across each other. Avatars provide visual clues about position and view direction of other users (a). A spectator’s perspective of the virtual scene illustrates the system setup (b), alongside an external photo of the user (c).

Figure 4 shows two users in different physical environments looking at the same hologram while “standing” next to each other
In this work, we demonstrate that hands-free voice commands are a powerful tool for enabling users to manipulate objects while performing other tasks with their hands, executing complex interactions with multiple objects, and performing extremely precise manipulations that are not possible solely using hand manipulation. Additionally, multi-user capabilities enhance real-time collaboration and data interpretation. By also integrating previously isolated features like hand, head, and gaze interaction, we aim to create user-friendly, multimodal, and collaborative Medical AR tools

Acknowledgements (Optional):