ViTAL Lab Computational Behavior and Health Analytics

Research Projects

Human Activity Recognition
Ambient AI in Clinic
Wearable AI at Home
Mobile Computer Vision
Multimodal AI for Mental Health Interview

Generative AI and Cross-modality Techniques for Human Activity Recognition

figure image

In addressing the challenges of building a human activity recognition model with limited wearable sensor data, we proposed a novel method utilizing vast video and textual data resources to generate a large-scale training dataset. This cross-modality transfer learning technique harnesses state-of-the-art computer vision algorithms to interpret human motions from videos or Large Language Models to create text descriptions of human activities, eventually generating virtual inertial measurement signals that can be used for model training. The resulting model, trained on this rich dataset, can tackle real-world applications. Additionally, we introduced an innovative training scheme to account for the inherent uncertainties in activity annotations. This is complemented by a novel sensor feature representation that effectively captures the structural and distributional nuances of sensor signals, capturing human activity, enhancing the model’s robustness.

Behavior Sensing in Clinics using Ambient AI with Video and Wearable Sensors

figure image

Our work explores how smart, privacy-preserving camera and sensor systems can monitor cognitive and social behaviors in real-world clinical settings. Focusing on mild cognitive impairment (MCI), we used edge-computing camera networks and Bluetooth beacons in a large therapeutic facility to detect differences in movement and social interactions between high- and low-functioning individuals. A novel method was also developed to identify when and where group activities occur in indoor spaces using lightweight pose detection, with high accuracy. The system was scalable, low-cost, and effective in localizing and tracking people while preserving privacy. Further, the 3D kinematics motion analysis technologies were adapted to objectively measure freezing of gait in Parkinson’s patients using explainable deep learning in a motion analysis laboratory in academic clinics. A video-based group activity analysis system could identify problematic behaviors in students with autism in real-world special education classroom. Together, these studies demonstrate the promise of edge AI systems for passive, real-time monitoring of cognitive and behavioral health across diverse populations and environments.

Behavior Sensing with Wearables at Home

figure image

We are exploring how wearable sensors and thoughtful home design can help detect and support cognitive and behavioral challenges in people with conditions like Parkinson’s disease (PD), autism spectrum disorder (ASD), and mild cognitive impairment (MCI). In PD, we found that existing activity recognition models used to detect “freezing of gait” (FOG) can be biased by age, sex, and disease stage, but fairness and accuracy improved using transfer learning from diverse datasets. In children with ASD, wearable devices tracking movement and physiological signals like skin temperature were able to detect behaviors like self-injury in everyday settings, offering a real-world way to support care. For older adults with MCI, wearables helped identify functional differences in kitchen tasks—like weaker upper body movements and delayed eye movements—highlighting how behavior reflects cognitive decline. we also found that open kitchen shelving slightly improved motivation and task efficiency, though personal aesthetic preferences still mattered. Overall, these studies show how wearable technology and personalized environments can offer practical, real-world support for people facing cognitive and behavioral health challenges.

Mobile Computer Vision and Closed-loop Intervention Systems for Motion Assessments and Rehabilitation

figure image

We developed mobile AI systems to improve motor assessment and rehabilitation. A mobile phone-based, privacy-preserving system classifies various gait impairment patterns, demonstrating the potential for accessible gait analysis in clinical and tele-rehabilitation settings. We also developed a closed-loop neurostimulation system that integrates real-time video-based movement classification on webcam with wireless transcutaneous vagus nerve stimulation (tVNS), automatically triggering stimulation during rehabilitation exercises, showcasing a promising step toward patient-driven, home-based motor rehabilitation.

Faical Expression, Speech, Language, cardiovascular signal Analysis in Remote Mental Health Interviews

figure image

We are developing novel machine learning tools that use audio, video, language, and heart rate data from remote interviews to help detect mental health conditions and cognitive decline in older adults. These tools can assess risks like dementia, depression, anxiety, social isolation, and personality traits like neuroticism, using features like facial expressions, speech patterns, and cardiovascular signals. Studies also showed that different data types—like speech for cognitive issues and heart rate for mental health—play unique roles. However, fairness remains a concern, as detection accuracy can vary across demographic groups like race, age, and education. While methods like post-training calibration can reduce these biases, they come with the cost of reduced performance. Overall, these tools show promise for affordable, remote mental health and cognitive screening, but continued attention to fairness and clinical integration is critical.