Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies
The swift development of artificial intelligence (AI) technology has triggered substantial changes, particularly evident in the emergence of chat-based services driven by large language models. With the increasing number of users utilizing these services, understanding and analysing user satisfactio...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2025-02-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.241687 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832540485494767616 |
|---|---|
| author | Hae Sun Jung Jang Hyun Kim Haein Lee |
| author_facet | Hae Sun Jung Jang Hyun Kim Haein Lee |
| author_sort | Hae Sun Jung |
| collection | DOAJ |
| description | The swift development of artificial intelligence (AI) technology has triggered substantial changes, particularly evident in the emergence of chat-based services driven by large language models. With the increasing number of users utilizing these services, understanding and analysing user satisfaction becomes crucial for service improvement. While previous studies have explored leveraging online reviews as indicators of user satisfaction, efficiently collecting and analysing extensive datasets remain a challenge. This research aims to address this challenge by proposing a framework to handle extensive review datasets from the Google Play Store, employing natural language processing with machine learning techniques for sentiment analysis. Specifically, the authors collect review data of chat-based AI applications and perform filtering through majority voting of multiple unsupervised sentiment analyses. This framework is a proposed methodology for eliminating inconsistencies between ratings and contents. Subsequently, the authors conduct supervised sentiment analysis using various machine learning and deep learning algorithms. The experimental results confirm the effectiveness of the proposed approach showing improvement in prediction accuracy with cost efficiency. In summary, the findings of this study enhance the predictive performance of user satisfaction for improving service quality in chat-based AI applications and provide valuable insights for the advancement of next-generation chat-based AI services. |
| format | Article |
| id | doaj-art-bdbb3a25cbe14ff286fcbb897d352d26 |
| institution | Kabale University |
| issn | 2054-5703 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj-art-bdbb3a25cbe14ff286fcbb897d352d262025-02-05T00:05:16ZengThe Royal SocietyRoyal Society Open Science2054-57032025-02-0112210.1098/rsos.241687Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistenciesHae Sun Jung0Jang Hyun Kim1Haein Lee2Department of Applied Artificial Intelligence, Sungkyunkwan University, Seoul 03063, Republic of KoreaDepartment of Interaction Science, Sungkyunkwan University, Seoul 03063, Republic of KoreaDepartment of Applied Artificial Intelligence, Sungkyunkwan University, Seoul 03063, Republic of KoreaThe swift development of artificial intelligence (AI) technology has triggered substantial changes, particularly evident in the emergence of chat-based services driven by large language models. With the increasing number of users utilizing these services, understanding and analysing user satisfaction becomes crucial for service improvement. While previous studies have explored leveraging online reviews as indicators of user satisfaction, efficiently collecting and analysing extensive datasets remain a challenge. This research aims to address this challenge by proposing a framework to handle extensive review datasets from the Google Play Store, employing natural language processing with machine learning techniques for sentiment analysis. Specifically, the authors collect review data of chat-based AI applications and perform filtering through majority voting of multiple unsupervised sentiment analyses. This framework is a proposed methodology for eliminating inconsistencies between ratings and contents. Subsequently, the authors conduct supervised sentiment analysis using various machine learning and deep learning algorithms. The experimental results confirm the effectiveness of the proposed approach showing improvement in prediction accuracy with cost efficiency. In summary, the findings of this study enhance the predictive performance of user satisfaction for improving service quality in chat-based AI applications and provide valuable insights for the advancement of next-generation chat-based AI services.https://royalsocietypublishing.org/doi/10.1098/rsos.241687chat-based AIsentiment analysisnatural language processinguser satisfactionBERT |
| spellingShingle | Hae Sun Jung Jang Hyun Kim Haein Lee Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies Royal Society Open Science chat-based AI sentiment analysis natural language processing user satisfaction BERT |
| title | Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies |
| title_full | Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies |
| title_fullStr | Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies |
| title_full_unstemmed | Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies |
| title_short | Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies |
| title_sort | refining the prediction of user satisfaction on chat based ai applications with unsupervised filtering of rating text inconsistencies |
| topic | chat-based AI sentiment analysis natural language processing user satisfaction BERT |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.241687 |
| work_keys_str_mv | AT haesunjung refiningthepredictionofusersatisfactiononchatbasedaiapplicationswithunsupervisedfilteringofratingtextinconsistencies AT janghyunkim refiningthepredictionofusersatisfactiononchatbasedaiapplicationswithunsupervisedfilteringofratingtextinconsistencies AT haeinlee refiningthepredictionofusersatisfactiononchatbasedaiapplicationswithunsupervisedfilteringofratingtextinconsistencies |