Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests
In this article, we explore the use of Large Language Models (LLMs) for predicting factor loadings in personality tests through the semantic analysis of test items. By leveraging text embeddings generated from LLMs, we evaluate the semantic similarity of test items and their alignment with hypothesi...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-01-01
|
| Series: | Current Research in Behavioral Sciences |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666518225000014 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576414081089536 |
|---|---|
| author | Nicola Milano Maria Luongo Michela Ponticorvo Davide Marocco |
| author_facet | Nicola Milano Maria Luongo Michela Ponticorvo Davide Marocco |
| author_sort | Nicola Milano |
| collection | DOAJ |
| description | In this article, we explore the use of Large Language Models (LLMs) for predicting factor loadings in personality tests through the semantic analysis of test items. By leveraging text embeddings generated from LLMs, we evaluate the semantic similarity of test items and their alignment with hypothesized factorial structures without depending on human response data. Our methodology involves using embeddings from four different personality test to examine correlations between item semantics and their grouping in principal factors. Our results indicate that LLM-derived embeddings can effectively capture semantic similarities among test items, showing moderate to high correlation with the factorial structure produced by humans respondents in all tests, potentially serving as a valid measure of content validity for initial survey design and refinement. This approach offers valuable insights into the robustness of embedding techniques in psychological evaluations, showing a significant correlation with traditional test structures and providing a novel perspective on test item analysis. |
| format | Article |
| id | doaj-art-779d8b47fc9a4db190ebcf4f421f94fd |
| institution | Kabale University |
| issn | 2666-5182 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Current Research in Behavioral Sciences |
| spelling | doaj-art-779d8b47fc9a4db190ebcf4f421f94fd2025-01-31T05:12:29ZengElsevierCurrent Research in Behavioral Sciences2666-51822025-01-018100168Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality testsNicola Milano0Maria Luongo1Michela Ponticorvo2Davide Marocco3Corresponding author.; Department of Humanistic Studies, Natural and Artificial Cognition Laboratory “Orazio Miglino”, University of Naples Federico II, via Porta di Massa 1, Naples 80125, ItalyDepartment of Humanistic Studies, Natural and Artificial Cognition Laboratory “Orazio Miglino”, University of Naples Federico II, via Porta di Massa 1, Naples 80125, ItalyDepartment of Humanistic Studies, Natural and Artificial Cognition Laboratory “Orazio Miglino”, University of Naples Federico II, via Porta di Massa 1, Naples 80125, ItalyDepartment of Humanistic Studies, Natural and Artificial Cognition Laboratory “Orazio Miglino”, University of Naples Federico II, via Porta di Massa 1, Naples 80125, ItalyIn this article, we explore the use of Large Language Models (LLMs) for predicting factor loadings in personality tests through the semantic analysis of test items. By leveraging text embeddings generated from LLMs, we evaluate the semantic similarity of test items and their alignment with hypothesized factorial structures without depending on human response data. Our methodology involves using embeddings from four different personality test to examine correlations between item semantics and their grouping in principal factors. Our results indicate that LLM-derived embeddings can effectively capture semantic similarities among test items, showing moderate to high correlation with the factorial structure produced by humans respondents in all tests, potentially serving as a valid measure of content validity for initial survey design and refinement. This approach offers valuable insights into the robustness of embedding techniques in psychological evaluations, showing a significant correlation with traditional test structures and providing a novel perspective on test item analysis.http://www.sciencedirect.com/science/article/pii/S2666518225000014Semantic similarityLanguage modelsMachine learningContent validityDimensionality reductionTest items analysis |
| spellingShingle | Nicola Milano Maria Luongo Michela Ponticorvo Davide Marocco Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests Current Research in Behavioral Sciences Semantic similarity Language models Machine learning Content validity Dimensionality reduction Test items analysis |
| title | Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests |
| title_full | Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests |
| title_fullStr | Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests |
| title_full_unstemmed | Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests |
| title_short | Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests |
| title_sort | semantic analysis of test items through large language model embeddings predicts a priori factorial structure of personality tests |
| topic | Semantic similarity Language models Machine learning Content validity Dimensionality reduction Test items analysis |
| url | http://www.sciencedirect.com/science/article/pii/S2666518225000014 |
| work_keys_str_mv | AT nicolamilano semanticanalysisoftestitemsthroughlargelanguagemodelembeddingspredictsapriorifactorialstructureofpersonalitytests AT marialuongo semanticanalysisoftestitemsthroughlargelanguagemodelembeddingspredictsapriorifactorialstructureofpersonalitytests AT michelaponticorvo semanticanalysisoftestitemsthroughlargelanguagemodelembeddingspredictsapriorifactorialstructureofpersonalitytests AT davidemarocco semanticanalysisoftestitemsthroughlargelanguagemodelembeddingspredictsapriorifactorialstructureofpersonalitytests |