Associate Professor University of Pittsburgh, United States
Introduction: Integrating Virtual Reality (VR) with brain imaging provides opportunities to collect data of cognitive tasks that more accurately represent the ‘real world’ than what can traditionally be collected in the lab. While neuroimaging studies testing simple cognitive tasks provide important data in advancing the field and benefit from a controlled environment, they are limited in their scope of providing information about cognition in real-world events. Integrating VR into neuroimaging studies allows researchers to collect data from subjects immersed in environments that mimic real world cognitive decision-making whilst still being able to provide reliable data from a controlled environment. VR has had an increasing role in clinical and cognitive research. Expanding research done with VR into neuroimaging allows researchers to better understand brain activity during tasks proven to have clinical significance. Functional Near-Infra Red Spectroscopy (fNIRS) is non-invasive method of collecting brain activity based on metabolism of glucose and relative concentrations of oxygenated hemoglobin (oxy-Hb) and deoxygenated hemoglobin (deoxy-Hb). Oxy-Hb and deoxy-Hb scatter and absorb NIR light at different wavelengths. fNIRS uses light sources and detectors to measure these optical changes and convert them to changes in brain activity (hemodynamic response) via modified Beer-Lambert law. FNIRS is portable and not as sensitive to movement as other methods of brain imaging which makes it a desirable method for integrating with VR.
Materials and
Methods: FNIRS: To collect brain activity during the task, we used NIRX NIRSPORT2. 42 channels were distributed across bilateral frontal and sensorimotor brain regions. Fig. 1 shows the sensitivity of the probes overlying Brodmann areas. In this experiment, we used the wearable fNIRS system. The participant wore the Oculus Meta Quest 2 to run the VR program. The headset was designed to integrate with the Oculus headset to limit mechanical and optical interference.
Box Task: We built a program in Unreal Engine to perform a spatial working memory box task where the user was placed in a room with 16 boxes. The user initially started in a survey period where they clicked on every box to find the 3 reward boxes. When pressed, a box would either turn green to indicate it was a reward or turn red to indicate it was not a reward. The user would then go through a 30 second baseline period where the boxes became invisible followed by a retrieval period where they are spawned at a random location in the room, all the box's colors are reset to the default, and they are given 30 seconds to find the 3 reward boxes. In a level, the participant goes through a survey period and then alternating rest and retrieval periods for five trials. We recorded the participant’s brain activity for two levels.
Results, Conclusions, and Discussions:
Results: We recorded fNIRS data during the VR task with n=4 to find preliminary results. Brain activity changes were estimated between the Survey and the Retrieval periods compared to the baseline. The fNIRS results are shown in Fig. 1. This figure shows the location of the fNIRS probe on the head. Each line indicates a measurement of the brain activity in the region below the sensors. The color of the lines indicates the statistical test for the change in brain activity during the task compared to the resting baseline period.
Discussion :Combining fNIRS and VR involved balancing a number of design challenges to collect and analyze the fNIRS data. We encountered issues with motion sickness and discomfort from the fNIRS head cap, leading us to reduce VR character speed and motion controls. This adjustment limited the number of trials participants could comfortably perform. We found that fewer, longer trials were better tolerated than frequent, shorter ones. Analyzing the data revealed challenges in accurately capturing brain activity and establishing a clear baseline. Participants' familiarity with the VR environment or random spawning near reward locations sometimes led to incomplete task performance data. Additionally, the baseline period, which made the task invisible, needed a clearer contrast from the task phases. Adjusting the visual or auditory cues in the VR environment might help differentiate these periods more effectively.
Conclusion: While integrating VR with neuroimaging presents challenges with experimental design, we have found ways to overcome such challenges. Integrating fNIRS and Virtual Reality shows promise for advancing the field of neuroimaging.
