Collecting Remote Voice and Movement Data from People with Parkinson’s Disease (PD) using Multimodal Conversational AI: Lessons Learned from a National Study


Telehealth is increasingly gaining recognition in the fields of neurology and speech-pathology (SLP) as a means of addressing accessibility issues experienced by people with PD. We present one potential solution – a conversational artificial intelligence (AI) agent, Tina, with whom people with PD can interact from the comfort of their homes while she guides them through a customizable assessment. The purpose of this presentation is to discuss lessons learned from deploying and testing this system in an ongoing national study with PD patients. 43 people with PD and 16 age- and sex-matched control participants have been enrolled in the study, with study recruitment continuing. Participants completed four assessment visits, one per week at times that fit their schedule. Issues related to internet connection were anticipated, but were not as impactful as we anticipated. Some issues related to PD were anticipated, including the potential for cognitive impairment to make it difficult for people with PD to execute a session independently with the dialog agent. This was not a large problem in the study. One PD-related issue that was unanticipated was problems with the system voice activity detection routine which occasionally terminated tasks early due to low vocal intensity or long pauses in the speech of people with PD. Throughout the course of the study, we have adjusted criteria for stopping tasks to mitigate these issues. Issues related to caregiver interactions were not anticipated. Caregivers were often involved in helping participants with PD to connect to the system and completing the assessment. Occasionally the web cameras picked up the caregiver and acoustic samples sometimes included speech produced by the caregiver. This study demonstrates the feasibility of collecting acoustic and video data from people with PD for the purposes of speech and motor assessments.

Motor Speech Conference