Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy
Abstract Introduction We investigated the agreement between automated and gold‐standard manual transcriptions of telephone chatbot‐based semantic verbal fluency testing. Methods We examined 78 cases from the Screening over Speech in Unselected Populations for Clinical Trials in AD (PROSPECT‐AD) stud...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-10-01
|
| Series: | Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring |
| Subjects: | |
| Online Access: | https://doi.org/10.1002/dad2.70011 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850041038151876608 |
|---|---|
| author | Alexandra König Stefanie Köhler Johannes Tröger Emrah Düzel Wenzel Glanz Michaela Butryn Elisa Mallick Josef Priller Slawek Altenstein Annika Spottke Okka Kimmich Björn Falkenburger Antje Osterrath Jens Wiltfang Claudia Bartels Ingo Kilimann Christoph Laske Matthias H. Munk Sandra Roeske Ingo Frommann Daniel C. Hoffmann Frank Jessen Michael Wagner Nicklas Linz Stefan Teipel |
| author_facet | Alexandra König Stefanie Köhler Johannes Tröger Emrah Düzel Wenzel Glanz Michaela Butryn Elisa Mallick Josef Priller Slawek Altenstein Annika Spottke Okka Kimmich Björn Falkenburger Antje Osterrath Jens Wiltfang Claudia Bartels Ingo Kilimann Christoph Laske Matthias H. Munk Sandra Roeske Ingo Frommann Daniel C. Hoffmann Frank Jessen Michael Wagner Nicklas Linz Stefan Teipel |
| author_sort | Alexandra König |
| collection | DOAJ |
| description | Abstract Introduction We investigated the agreement between automated and gold‐standard manual transcriptions of telephone chatbot‐based semantic verbal fluency testing. Methods We examined 78 cases from the Screening over Speech in Unselected Populations for Clinical Trials in AD (PROSPECT‐AD) study, including cognitively normal individuals and individuals with subjective cognitive decline, mild cognitive impairment, and dementia. We used Bayesian Bland–Altman analysis of word count and the qualitative features of semantic cluster size, cluster switches, and word frequencies. Results We found high levels of agreement for word count, with a 93% probability of a newly observed difference being below the minimally important difference. The qualitative features had fair levels of agreement. Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode. Discussion Our results support the use of automated speech recognition particularly for the assessment of quantitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes. Highlights High levels of agreement were found between automated and gold‐standard manual transcriptions of telephone chatbot‐based semantic verbal fluency testing, particularly for word count. The qualitative features had fair levels of agreement. Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode. Automated speech recognition for the assessment of quantitative and qualitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes, seems feasible and reliable. |
| format | Article |
| id | doaj-art-4dbd398884b4485dba14ea3ee8c3609d |
| institution | DOAJ |
| issn | 2352-8729 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | Wiley |
| record_format | Article |
| series | Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring |
| spelling | doaj-art-4dbd398884b4485dba14ea3ee8c3609d2025-08-20T02:55:53ZengWileyAlzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring2352-87292024-10-01164n/an/a10.1002/dad2.70011Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracyAlexandra König0Stefanie Köhler1Johannes Tröger2Emrah Düzel3Wenzel Glanz4Michaela Butryn5Elisa Mallick6Josef Priller7Slawek Altenstein8Annika Spottke9Okka Kimmich10Björn Falkenburger11Antje Osterrath12Jens Wiltfang13Claudia Bartels14Ingo Kilimann15Christoph Laske16Matthias H. Munk17Sandra Roeske18Ingo Frommann19Daniel C. Hoffmann20Frank Jessen21Michael Wagner22Nicklas Linz23Stefan Teipel24ki:elements GmbH Saarbrücken GermanyGerman Center for Neurodegenerative Diseases (DZNE), Rostock Greifswald GermanyUniversité Côte d'Azur, Centre Hospitalier et Universitaire, Clinique Gériatrique du Cerveau et du Mouvement, Centre Mémoire de Ressources et de Recherche Nice FranceGerman Center for Neurodegenerative Diseases (DZNE) Magdeburg GermanyGerman Center for Neurodegenerative Diseases (DZNE) Magdeburg GermanyGerman Center for Neurodegenerative Diseases (DZNE) Magdeburg GermanyUniversité Côte d'Azur, Centre Hospitalier et Universitaire, Clinique Gériatrique du Cerveau et du Mouvement, Centre Mémoire de Ressources et de Recherche Nice FranceGerman Center for Neurodegenerative Diseases (DZNE) Berlin GermanyGerman Center for Neurodegenerative Diseases (DZNE) Berlin GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Dresden GermanyGerman Center for Neurodegenerative Diseases (DZNE) Dresden GermanyGerman Center for Neurodegenerative Diseases (DZNE) Göttingen GermanyDepartment of Psychiatry and Psychotherapy University Medical Center Goettingen University of Goettingen Göttingen GermanyGerman Center for Neurodegenerative Diseases (DZNE), Rostock Greifswald GermanyGerman Center for Neurodegenerative Diseases (DZNE) Tübingen GermanyGerman Center for Neurodegenerative Diseases (DZNE) Tübingen GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyGerman Center for Neurodegenerative Diseases (DZNE) Bonn GermanyUniversité Côte d'Azur, Centre Hospitalier et Universitaire, Clinique Gériatrique du Cerveau et du Mouvement, Centre Mémoire de Ressources et de Recherche Nice FranceGerman Center for Neurodegenerative Diseases (DZNE), Rostock Greifswald GermanyAbstract Introduction We investigated the agreement between automated and gold‐standard manual transcriptions of telephone chatbot‐based semantic verbal fluency testing. Methods We examined 78 cases from the Screening over Speech in Unselected Populations for Clinical Trials in AD (PROSPECT‐AD) study, including cognitively normal individuals and individuals with subjective cognitive decline, mild cognitive impairment, and dementia. We used Bayesian Bland–Altman analysis of word count and the qualitative features of semantic cluster size, cluster switches, and word frequencies. Results We found high levels of agreement for word count, with a 93% probability of a newly observed difference being below the minimally important difference. The qualitative features had fair levels of agreement. Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode. Discussion Our results support the use of automated speech recognition particularly for the assessment of quantitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes. Highlights High levels of agreement were found between automated and gold‐standard manual transcriptions of telephone chatbot‐based semantic verbal fluency testing, particularly for word count. The qualitative features had fair levels of agreement. Word count reached high levels of discrimination between cognitively impaired and unimpaired individuals, regardless of transcription mode. Automated speech recognition for the assessment of quantitative and qualitative speech features, even when using data from telephone calls with cognitively impaired individuals in their homes, seems feasible and reliable.https://doi.org/10.1002/dad2.70011automated speech recognitionBland–Altman analysisdementiareliabilityremote cognitive testingsemantic verbal fluency |
| spellingShingle | Alexandra König Stefanie Köhler Johannes Tröger Emrah Düzel Wenzel Glanz Michaela Butryn Elisa Mallick Josef Priller Slawek Altenstein Annika Spottke Okka Kimmich Björn Falkenburger Antje Osterrath Jens Wiltfang Claudia Bartels Ingo Kilimann Christoph Laske Matthias H. Munk Sandra Roeske Ingo Frommann Daniel C. Hoffmann Frank Jessen Michael Wagner Nicklas Linz Stefan Teipel Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring automated speech recognition Bland–Altman analysis dementia reliability remote cognitive testing semantic verbal fluency |
| title | Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy |
| title_full | Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy |
| title_fullStr | Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy |
| title_full_unstemmed | Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy |
| title_short | Automated remote speech‐based testing of individuals with cognitive decline: Bayesian agreement of transcription accuracy |
| title_sort | automated remote speech based testing of individuals with cognitive decline bayesian agreement of transcription accuracy |
| topic | automated speech recognition Bland–Altman analysis dementia reliability remote cognitive testing semantic verbal fluency |
| url | https://doi.org/10.1002/dad2.70011 |
| work_keys_str_mv | AT alexandrakonig automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT stefaniekohler automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT johannestroger automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT emrahduzel automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT wenzelglanz automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT michaelabutryn automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT elisamallick automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT josefpriller automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT slawekaltenstein automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT annikaspottke automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT okkakimmich automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT bjornfalkenburger automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT antjeosterrath automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT jenswiltfang automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT claudiabartels automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT ingokilimann automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT christophlaske automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT matthiashmunk automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT sandraroeske automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT ingofrommann automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT danielchoffmann automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT frankjessen automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT michaelwagner automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT nicklaslinz automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy AT stefanteipel automatedremotespeechbasedtestingofindividualswithcognitivedeclinebayesianagreementoftranscriptionaccuracy |