Multimodal Dialog Based Remote Patient Monitoring of Motor Function in Parkinson’s Disease and Other Movement Disorder


The need for remote monitoring to support Parkinson’s Disease (PD) patients, caregivers, and healthcare professionals in their collaborative efforts for better care has never been greater. We have earlier presented Tina, a virtual agent that is powered by a cloud-based multimodal dialog system designed to conduct automated screening interviews that elicit evidence for detection and progress monitoring of neurological disorders like PD. This work presents our innovations to this system to also allow for assessment of patient motor function via finger-tapping exercises. Most existing contactless RPM systems assess dexterity of finger movements with the help of smartphone apps in which a variant of the finger tapping task is used by asking the user to alternately tap buttons on the screen. In contrast, Tina engages participants in a conversation consisting of structured conversational exercises designed to elicit specific speech, facial, and now limb motor behaviors, thereby, combining the benefits of cost-effective, frequent remote monitoring with the ability for clinicians to review the performance. For the finger tapping exercises, participants are instructed to hold either their left or right hand up to the camera and perform a tapping motion for five seconds, while anatomical landmarks of the participant’s hand are derived from the recorded image frames. The finger tapping assessment comprises three tasks that differ based on the instructed goal, i.e. the movement should be as (1) wide, (2) fast, or (3) both wide and fast as possible. Key points of interest during the test are the duration and amplitude of a cycle. If either the duration of the cycle increases or the amplitude decreases, the system will identify this as a positive test result, which should be assessed further. This work demonstrates the feasibility of novel finger-tapping exercises administered through a multimodal dialog agent for assessment of motor function in PD and other movement disorders.

Motor Speech Conference