From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists
IntroductionArtificial intelligence (AI) has tremendous potential for use in psychology. Among the many applications that may benefit from development of AI applications is narrative-personality assessment. Use of these tools and research methods is notably time-consuming and resource intensive. AI...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-02-01
|
| Series: | Frontiers in Artificial Intelligence |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/frai.2025.1398885/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832087102100078592 |
|---|---|
| author | Barry Dauphin Caleb Siefert |
| author_facet | Barry Dauphin Caleb Siefert |
| author_sort | Barry Dauphin |
| collection | DOAJ |
| description | IntroductionArtificial intelligence (AI) has tremendous potential for use in psychology. Among the many applications that may benefit from development of AI applications is narrative-personality assessment. Use of these tools and research methods is notably time-consuming and resource intensive. AI has potential to address these issues in ways that would greatly reduce clinician and researcher burden. Nonetheless, it is unclear if current AI models are sufficiently sophisticated to perform the complex downstream tasks, such as narrative assessment.MethodologyThe purpose of this study is to explore if an expert-refined prompt generation process can enable AI-empowered chatbots to reliably and accurately rate narratives using the Social Cognition and Object Relations scales – Global Rating Method (SCORS-G). Experts generated prompt inputs by engaging in a detailed review of SCORS-G training materials. Prompts were then improved using an systematic process in which experts worked with Llama-2-70b to refine prompts. The utility of the prompts was then tested on two AI-empowered chatbots, ChatGPT-4 (OpenAI, 2023) and CLAUDE-2-100k, that were not used in the prompt refinement process.ResultsResults showed that the refined prompts allowed chatbots to reliably rate narratives at the global level, though accuracy varied across subscales. Averaging ratings from two chatbots notably improved reliability for the global score and all subscale scores. Experimentation indicated that expert-refined prompts outperformed basic prompts regarding interrater reliability and absolute agreement with gold standard ratings. Only the expert-refined prompts were able to generate acceptable single-rater interrater reliability estimates.DiscussionFindings suggest that AI could significantly reduce the time and resource burdens on clinicians and researchers using narrative rating systems like the SCORS-G. Limitations and implications for future research are discussed. |
| format | Article |
| id | doaj-art-250f14e08b6e4e57a74bcf771f06a1db |
| institution | Kabale University |
| issn | 2624-8212 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Artificial Intelligence |
| spelling | doaj-art-250f14e08b6e4e57a74bcf771f06a1db2025-02-06T07:09:33ZengFrontiers Media S.A.Frontiers in Artificial Intelligence2624-82122025-02-01810.3389/frai.2025.13988851398885From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologistsBarry Dauphin0Caleb Siefert1Department of Psychology, University of Detroit-Mercy, Detroit, MI, United StatesBehavioral Sciences, Psychology, University of Michigan-Dearborn, Dearborn, MI, United StatesIntroductionArtificial intelligence (AI) has tremendous potential for use in psychology. Among the many applications that may benefit from development of AI applications is narrative-personality assessment. Use of these tools and research methods is notably time-consuming and resource intensive. AI has potential to address these issues in ways that would greatly reduce clinician and researcher burden. Nonetheless, it is unclear if current AI models are sufficiently sophisticated to perform the complex downstream tasks, such as narrative assessment.MethodologyThe purpose of this study is to explore if an expert-refined prompt generation process can enable AI-empowered chatbots to reliably and accurately rate narratives using the Social Cognition and Object Relations scales – Global Rating Method (SCORS-G). Experts generated prompt inputs by engaging in a detailed review of SCORS-G training materials. Prompts were then improved using an systematic process in which experts worked with Llama-2-70b to refine prompts. The utility of the prompts was then tested on two AI-empowered chatbots, ChatGPT-4 (OpenAI, 2023) and CLAUDE-2-100k, that were not used in the prompt refinement process.ResultsResults showed that the refined prompts allowed chatbots to reliably rate narratives at the global level, though accuracy varied across subscales. Averaging ratings from two chatbots notably improved reliability for the global score and all subscale scores. Experimentation indicated that expert-refined prompts outperformed basic prompts regarding interrater reliability and absolute agreement with gold standard ratings. Only the expert-refined prompts were able to generate acceptable single-rater interrater reliability estimates.DiscussionFindings suggest that AI could significantly reduce the time and resource burdens on clinicians and researchers using narrative rating systems like the SCORS-G. Limitations and implications for future research are discussed.https://www.frontiersin.org/articles/10.3389/frai.2025.1398885/fullSocial Cognition and Object Relations Scalesartificial intelligencenarrative researchpsychopathologypersonalitypersonality assessment |
| spellingShingle | Barry Dauphin Caleb Siefert From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists Frontiers in Artificial Intelligence Social Cognition and Object Relations Scales artificial intelligence narrative research psychopathology personality personality assessment |
| title | From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists |
| title_full | From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists |
| title_fullStr | From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists |
| title_full_unstemmed | From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists |
| title_short | From Llama to language: prompt-engineering allows general-purpose artificial intelligence to rate narratives like expert psychologists |
| title_sort | from llama to language prompt engineering allows general purpose artificial intelligence to rate narratives like expert psychologists |
| topic | Social Cognition and Object Relations Scales artificial intelligence narrative research psychopathology personality personality assessment |
| url | https://www.frontiersin.org/articles/10.3389/frai.2025.1398885/full |
| work_keys_str_mv | AT barrydauphin fromllamatolanguagepromptengineeringallowsgeneralpurposeartificialintelligencetoratenarrativeslikeexpertpsychologists AT calebsiefert fromllamatolanguagepromptengineeringallowsgeneralpurposeartificialintelligencetoratenarrativeslikeexpertpsychologists |