Cognitive phantoms in large language models through the lens of latent variables
Large language models (LLMs) increasingly reach real-world applications, necessitating a better understanding of their behaviour. Their size and complexity complicate traditional assessment methods, causing the emergence of alternative approaches inspired by the field of psychology. Recent studies a...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-05-01
|
| Series: | Computers in Human Behavior: Artificial Humans |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2949882125000453 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850246198115434496 |
|---|---|
| author | Sanne Peereboom Inga Schwabe Bennett Kleinberg |
| author_facet | Sanne Peereboom Inga Schwabe Bennett Kleinberg |
| author_sort | Sanne Peereboom |
| collection | DOAJ |
| description | Large language models (LLMs) increasingly reach real-world applications, necessitating a better understanding of their behaviour. Their size and complexity complicate traditional assessment methods, causing the emergence of alternative approaches inspired by the field of psychology. Recent studies administering psychometric questionnaires to LLMs report human-like traits in LLMs, potentially influencing LLM behaviour. However, this approach suffers from a validity problem: it presupposes that these traits exist in LLMs and that they are measurable with tools designed for humans. Typical procedures rarely acknowledge the validity problem in LLMs, comparing and interpreting average LLM scores. This study investigates this problem by comparing latent structures of personality between humans and three LLMs using two validated personality questionnaires. Findings suggest that questionnaires designed for humans do not validly measure similar constructs in LLMs, and that these constructs may not exist in LLMs at all, highlighting the need for psychometric analyses of LLM responses to avoid chasing cognitive phantoms. |
| format | Article |
| id | doaj-art-e397ed41e7fb4d6f990bba6402fb6fbd |
| institution | OA Journals |
| issn | 2949-8821 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers in Human Behavior: Artificial Humans |
| spelling | doaj-art-e397ed41e7fb4d6f990bba6402fb6fbd2025-08-20T01:59:14ZengElsevierComputers in Human Behavior: Artificial Humans2949-88212025-05-01410016110.1016/j.chbah.2025.100161Cognitive phantoms in large language models through the lens of latent variablesSanne Peereboom0Inga Schwabe1Bennett Kleinberg2Tilburg University, Department of Methodology and Statistics, Professor Cobbenhagenlaan 125, Tilburg, 5037 DB, the Netherlands; Corresponding author.Tilburg University, Department of Methodology and Statistics, Professor Cobbenhagenlaan 125, Tilburg, 5037 DB, the NetherlandsTilburg University, Department of Methodology and Statistics, Professor Cobbenhagenlaan 125, Tilburg, 5037 DB, the Netherlands; University College London, Department of Security and Crime Science, 35 Tavistock Square, London, WC1H 9EZ, England, UKLarge language models (LLMs) increasingly reach real-world applications, necessitating a better understanding of their behaviour. Their size and complexity complicate traditional assessment methods, causing the emergence of alternative approaches inspired by the field of psychology. Recent studies administering psychometric questionnaires to LLMs report human-like traits in LLMs, potentially influencing LLM behaviour. However, this approach suffers from a validity problem: it presupposes that these traits exist in LLMs and that they are measurable with tools designed for humans. Typical procedures rarely acknowledge the validity problem in LLMs, comparing and interpreting average LLM scores. This study investigates this problem by comparing latent structures of personality between humans and three LLMs using two validated personality questionnaires. Findings suggest that questionnaires designed for humans do not validly measure similar constructs in LLMs, and that these constructs may not exist in LLMs at all, highlighting the need for psychometric analyses of LLM responses to avoid chasing cognitive phantoms.http://www.sciencedirect.com/science/article/pii/S2949882125000453Large language modelsPsychometricsMachine behaviourLatent variable modellingValidity |
| spellingShingle | Sanne Peereboom Inga Schwabe Bennett Kleinberg Cognitive phantoms in large language models through the lens of latent variables Computers in Human Behavior: Artificial Humans Large language models Psychometrics Machine behaviour Latent variable modelling Validity |
| title | Cognitive phantoms in large language models through the lens of latent variables |
| title_full | Cognitive phantoms in large language models through the lens of latent variables |
| title_fullStr | Cognitive phantoms in large language models through the lens of latent variables |
| title_full_unstemmed | Cognitive phantoms in large language models through the lens of latent variables |
| title_short | Cognitive phantoms in large language models through the lens of latent variables |
| title_sort | cognitive phantoms in large language models through the lens of latent variables |
| topic | Large language models Psychometrics Machine behaviour Latent variable modelling Validity |
| url | http://www.sciencedirect.com/science/article/pii/S2949882125000453 |
| work_keys_str_mv | AT sannepeereboom cognitivephantomsinlargelanguagemodelsthroughthelensoflatentvariables AT ingaschwabe cognitivephantomsinlargelanguagemodelsthroughthelensoflatentvariables AT bennettkleinberg cognitivephantomsinlargelanguagemodelsthroughthelensoflatentvariables |