Skip to main content
2024 BMES Annual Meeting Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.

Neural Engineering

Neural Engineering - Poster Session E

Poster K15 - Towards a Composite Metric of iBCI Performance

Saturday, October 26, 2024
10:00 AM - 11:00 AM EST
Location: Exhibit Hall E, F & G

Introduction:  A brain-computer interface (BCI) is a device that translates neural signals into a command to control an external device (e.g., mouse curser, robotic limb, etc.) (Pandarinath & Bensmaia, 2022). Intracortical brain-computer interfaces (iBCIs), which record neural activity using implanted microelectrodes, have enabled independent computer access using only neural signals for control (Hochberg et al., 2006; Collinger et al., 2013). Existing performance metrics such as success rate, acquisition rate, and path length describe different aspects of iBCI control performance, but there remains no consensus on how to quantify performance in a single metric that correlates with the user’s subjective experience and satisfaction (though see Shenoy et al., 2021 for further discussion). To address this, we developed a pipeline for creating a composite, summary performance metric that reflects the iBCI user’s self-reported performance assessment. We found a significant exponential correlation between the composite metric and neural separability (i.e., the amount of direction-related information) in two out of three participants.

Materials and

Methods:  Three participants (P2, P3, and P4), each of whom had two Utah Arrays (Blackrock Neurotech, Salt Lake City, Utah) implanted into their motor cortices, provided informed consent to participate in this study which was completed as part of an ongoing clinical trial conducted under an FDA Investigational Device Exemption and approved by the University of Pittsburgh Institutional Review Board. iBCI performance metrics (percent correct, normalized path length, acquisition rate, angular error, click time, and cursor-target distance) were calculated during an 8-target center-out and click cursor control task (number sessions: P2 = 32, P3 = 16, P4 = 38). Participants rated their performance and workload each day using the NASA-TLX scale, a validated measure of subjective workload comprised of six individual workload metrics combined into a weighted average (Hart & Staveland, 1988). To generate a composite iBCI performance metric, we used bootstrapped ridge regression to test for a relationship between the individual performance metrics and the composite NASA-TLX score. These model weights were then used to compute a weighted average of the iBCI performance metrics, which then was used as the composite performance metric. To test whether our composite score was related to the neural control signal, it was compared against a measure of the task-relevant neural signal strength (i.e., neural separability on the decoder plane) for each day. Neural separability quantifies the amount of between-target neural variance relative to the within-target variance.

Results, Conclusions, and Discussions: The participants had a range of performance with P2 averaging 81.47 ± 20.36% successful trials, P3 averaging 99.64 ± 0.87%, and P4 averaging 98.55 ± 3.20%. Across all metrics, P4, who had the most recent implant had the higher performance (Figure 1). Pairwise correlations between the individual iBCI performance metrics and NASA-TLX scores showed similar trends across participants, in that higher objective performance metrics were correlated with lower subjective scores (lower being better), but the patterns of correlations were subject-specific (Figure 2). For example, in P3 success rate was not well correlated with the other performance metrics, while for P4, metrics of his ability to click did not correlate strongly with other measures. To capture the relationship between objective performance metrics and each participant’s subjective performance rating, we built regression models for each participant using iBCI performance to predict NASA-TLX score. Though this model was only statistically significant in P2, who had the most variable performance across days, we explored whether a composite BCI performance measure computed with these models correlated with neural separability, a measure of task-relevant neural information. For all 3 participants, better BCI performance as measured with the composite score was significantly linearly related to neural separability (Figure 3; P2 R2 = 0.59, p = 2.39e-07; P3 R2 = 0.31, p = 2.59e-02; P4 R2 = 0.26, p = 1.06e-03).
Individual objective performance metrics did not always correlate with each other or the participant’s experience (TLX scores). For the participant with the most variable performance (P2), we created a composite metric that significantly correlated with the NASA-TLX and neural separability. For P3 and P4, although the composite metric models were not statistically significant, possibly due to limited data and performance saturation, we found preliminary evidence that the composite performance metric is related to neural signal separability. By using one performance metric, iBCI performance analyses are simplified, with potential to investigate how neural, health, or environmental factors influence performance. A performance metric based on subjective workload captures participants’ perspectives and preferences, which likely influences whether they would adopt this technology into their daily routines (Phillips & Zhao, 1993).

Acknowledgements (Optional):