Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development

IntroductionThe emergence of artificial intelligence and the widespread availability of large language model open the door to text analysis at scale leveraging on complex classification instructions. This case study explores the possibility of using available large language models to measure ego dev...

Full description

Saved in:
Bibliographic Details
Main Author: Xavier Bronlet
Format: Article
Language:English
Published: Frontiers Media S.A. 2025-02-01
Series:Frontiers in Psychology
Subjects:
Online Access:https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1488102/full
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832087150470889472
author Xavier Bronlet
author_facet Xavier Bronlet
author_sort Xavier Bronlet
collection DOAJ
description IntroductionThe emergence of artificial intelligence and the widespread availability of large language model open the door to text analysis at scale leveraging on complex classification instructions. This case study explores the possibility of using available large language models to measure ego development at scale and establish a methodology that can be applied to other classification instructions. Ego consists of the traits that influence how a person perceives and engages with the world, while ego development is a crucial aspect of adult personality growth, influencing behaviors and decisions in both personal and professional contexts. Accurate assessments of ego development stages are vital for creating effective strategies in organizational psychology and corporate analytics.MethodsThis case study investigates the agreement between expert and automated classifications of ego development stages, aiming to evaluate the potential of automation in this domain leveraging artificial intelligence and large language models. Cohen’s kappa statistic has been used to measure the agreement between classifications made by experts and those generated by an automated process leveraging large language models.ResultsThe comparison between the scoring of experts and large language models yielded a weighted Kappa value of 0.779, indicating a substantial level of agreement that is statistically meaningful and unlikely to be due to chance.DiscussionWhile this suggests valuable scoring that leverages large language models, it also highlights the opportunity for further refinement to closely match expert assessments. We observed low variability in aggregated values, demonstrating that the automated process functions effectively at scale. The robustness of aggregated data is particularly evident when calculating ego development scores for individuals, groups, corporate units, and entire corporations. This capability underscores the utility of the automated system for high-level evaluations and decision-making leveraging on a solid indicator. While the classification system developed in this case study shows promise, targeted enhancements may help to achieve a level of accuracy and reliability that improves alignment with experts’ evaluations for single sentences. The methodology developed in this case study appears to be useful to support other evaluations at scale that leverage large language models using other maps of classifications.
format Article
id doaj-art-948e8c2572c042cfb77769281a45706b
institution Kabale University
issn 1664-1078
language English
publishDate 2025-02-01
publisher Frontiers Media S.A.
record_format Article
series Frontiers in Psychology
spelling doaj-art-948e8c2572c042cfb77769281a45706b2025-02-06T07:10:28ZengFrontiers Media S.A.Frontiers in Psychology1664-10782025-02-011610.3389/fpsyg.2025.14881021488102Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego developmentXavier BronletIntroductionThe emergence of artificial intelligence and the widespread availability of large language model open the door to text analysis at scale leveraging on complex classification instructions. This case study explores the possibility of using available large language models to measure ego development at scale and establish a methodology that can be applied to other classification instructions. Ego consists of the traits that influence how a person perceives and engages with the world, while ego development is a crucial aspect of adult personality growth, influencing behaviors and decisions in both personal and professional contexts. Accurate assessments of ego development stages are vital for creating effective strategies in organizational psychology and corporate analytics.MethodsThis case study investigates the agreement between expert and automated classifications of ego development stages, aiming to evaluate the potential of automation in this domain leveraging artificial intelligence and large language models. Cohen’s kappa statistic has been used to measure the agreement between classifications made by experts and those generated by an automated process leveraging large language models.ResultsThe comparison between the scoring of experts and large language models yielded a weighted Kappa value of 0.779, indicating a substantial level of agreement that is statistically meaningful and unlikely to be due to chance.DiscussionWhile this suggests valuable scoring that leverages large language models, it also highlights the opportunity for further refinement to closely match expert assessments. We observed low variability in aggregated values, demonstrating that the automated process functions effectively at scale. The robustness of aggregated data is particularly evident when calculating ego development scores for individuals, groups, corporate units, and entire corporations. This capability underscores the utility of the automated system for high-level evaluations and decision-making leveraging on a solid indicator. While the classification system developed in this case study shows promise, targeted enhancements may help to achieve a level of accuracy and reliability that improves alignment with experts’ evaluations for single sentences. The methodology developed in this case study appears to be useful to support other evaluations at scale that leverage large language models using other maps of classifications.https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1488102/fulllarge language modelautomated classificationego developmentCohen’s kappamethods
spellingShingle Xavier Bronlet
Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
Frontiers in Psychology
large language model
automated classification
ego development
Cohen’s kappa
methods
title Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
title_full Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
title_fullStr Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
title_full_unstemmed Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
title_short Leveraging on large language model to classify sentences: a case study applying STAGES scoring methodology for sentence completion test on ego development
title_sort leveraging on large language model to classify sentences a case study applying stages scoring methodology for sentence completion test on ego development
topic large language model
automated classification
ego development
Cohen’s kappa
methods
url https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1488102/full
work_keys_str_mv AT xavierbronlet leveragingonlargelanguagemodeltoclassifysentencesacasestudyapplyingstagesscoringmethodologyforsentencecompletiontestonegodevelopment