Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Orthopedic and Rehabilitation Engineering

Orthopedic and Rehabilitation Engineering - Poster Session C

Poster K11 - Infant Sentiment Analysis with Pre-trained Vision Transformers

Friday, October 25, 2024

10:00 AM - 11:00 AM EST

Location: Exhibit Hall E, F & G

Presenting Author(s)

Ioana Gidiuta (she/her/hers)

Undergraduate Student in Computer Science, Accelerated Master's Candidate in Bioengineering
University of Pennsylvania
Philadelphia, Pennsylvania, United States

Last Author(s)

Melanie Segado, PhD

Postdoctoral Researcher
University of Pennsylvania, United States

Primary Investigator(s)

Michelle Johnson, PhD (she/her/hers)

Associate Professor
University of Pennsylvania
philadelphia, Pennsylvania, United States

Co-Author(s)

FS

Francis Sowande

Doctoral Student and Researcher
University of Pennsylvania, United States
LP

Laura Prosser

Physical Therapist, PhD.
Children Hospital of Philadelphia, United States
JS

Julie Skorup, DPT

Physical Therapist
Children Hospital of Philadelphia, United States
KK

Konrad Kording, PhD

Nathan Mossell University Professor (PIK), bioengineering & neuroscience, UPenn, CIFAR LMB
University of Pennsylvania, United States

Introduction: Early detection of Cerebral Palsy (CP) in infants is vital for timely intervention. Computer-vision driven advances in movement-based automated prediction of CP risk show promise; however, an infant’s emotional state significantly affects their movements. Therefore, detecting and accounting for emotional state is necessary to improve movement-based automated risk prediction. Here, we present a pipeline to detect emotional state from videos of infants taken in a sensorized gym (the PANDA Gym) and quantify its effect on movement. We find that pre-trained vision transformers can detect infant facial expressions without the need for fine-tuning, paving the way for large-scale studies of infant sentiment analysis. Additionally, we demonstrate that emotion labels can be computed from a small set of simple, explainable features, decreasing the likelihood of overfitting to a specific dataset thereby increasing the likelihood it will generalize to new, unseen data. Using our pipeline, we show that infant movements significantly differ between emotional states, laying the groundwork for improved clinical prediction.

Materials and

Methods: We collected 180 infant recordings (81 distinct subjects) comprising 1,285,000 frames using the PANDA Gym. The yolox-s algorithm was used for keypoint detection, and RTMPose-m was employed for facial keypoint estimation. For body keypoint detection, RTMDet-m and ViTPose-Huge were used. All algorithms were implemented in MMPose and applied using the MMPose inferencer. Frame-to-frame Euclidean displacements of each body keypoint were calculated, normalized from 0 to 1 by the maximum keypoint displacement per infant, summed per frame, and smoothed using a mean filter. The kinematic data was merged with emotion labels to give paired observations for each frame.

Blocks of 10 continuous frames with high average landmark confidence (≥0.75) were identified, and the keypoint coordinates were smoothed using mean and median filtering. All keypoints were rotated with respect to the center of the nose and the coordinates of each keypoint were normalized from -1 to 1. Frame-to-frame curvature and distance features were calculated (Fig. 1). Boolean features, based on the facial emotion characteristics described by Sullivan & Lewis (2003) (Table 1), were computed and used to label emotions (‘anger’, ‘interest’, ‘neutral’, ‘playful’, ‘sadness’, ‘sleepy’, ‘surprise’).

A Decision Forest algorithm was trained and tuned on a fraction of the clinical dataset to recognize emotions based on the original facial features and algorithmically computed labels. The trained model was then used to compute emotion labels for the rest of the clinical dataset. Blocks of 20 consecutive frames were analyzed to calculate the mean sum of displacements per frame and the dominant emotion.

Results, Conclusions, and Discussions:

Results: After computing the emotion labels and balancing the classes, a Grid Search was conducted to determine the best parameters for a Random Forest Classifier (average accuracy 94%). Fig. 2 represents emotion classification results for the 6 infant emotions.

A 1-way ANOVA (emotion by movement) indicated a significant difference between groups (F-value=35.36, p-value < 0.001). As Fig. 1 suggests, there are significant differences between emotion labels such as ‘anger’-‘neutral’ (p < 0.001), and ‘surprise’-‘neutral’ (p < 0.001). Angry, sad, and playful babies showed the most inter-frame keypoint displacement, while neutral, surprised, and attentive babies moved notably less. No significant differences were observed between 'anger' and 'playful', 'interest' and 'neutral', 'interest' and 'playful', 'neutral' and 'playful', and 'playful' and 'surprise', while all other emotion pairs proved to be statistically different. Note that ‘anger’ was the emotion label that stood out the most, likely because this group contains crying infants, whose movements are more aggravated.

Discussion: Additional analysis suggests that negative emotions result in significantly different movement intensities than neutral or positive emotions (p < 0.001), while positive and neutral groups show no statistical difference (Fig. 3.2). Moreover, as shown in Fig 3.3, significant differences were found between ‘intense’ and ‘mild’, ‘intense’ and ‘neutral’, and ‘mild’ and ‘neutral’ emotions. As expected, more intense emotions are correlated with more intense movements. Note that sleepy babies were considered a different category and thus were not included in this analysis, as clinical assessment guidelines are developed around the movements of awake infants. Movement Intensity (Smoothed Distance) showed significant variation within the same emotion group, likely because the data from multiple subjects is considered.

Conclusions: Overall, we found the amount of movement differs significantly between emotion labels. We further found that negative emotions are significantly different from neutral and positive emotions. We therefore propose that the emotional state of the infant may be an important factor to consider in movement-based automated risk assessment. Future work should aim to capture the effect of emotion on a wider range of kinematic movement features, and directly test whether accounting for emotional state can improve clinical predictions.

Acknowledgements (Optional): The work was supported by Penn Engineering’s Rachleff Scholars Program as part of the 2024 summer research experience. Moreover, we gratefully acknowledge the support of the National Institutes of Health (NIH) for their financial support (grants R21-HD084327-01 and R01-HD097686).