A multidimensional Bayesian IRT method for discovering misconceptions from concept test data
We present an exploratory method for discovering likely misconceptions from multiple-choice concept test data, as well as preliminary evidence that this method recovers known misconceptions from real student responses. Our procedure is based on a Bayesian implementation of the Multidimensional Nomin...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-01-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1506320/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832582908672475136 |
|---|---|
| author | Martin Segado Aaron Adair John Stewart Yunfei Ma Byron Drury David Pritchard |
| author_facet | Martin Segado Aaron Adair John Stewart Yunfei Ma Byron Drury David Pritchard |
| author_sort | Martin Segado |
| collection | DOAJ |
| description | We present an exploratory method for discovering likely misconceptions from multiple-choice concept test data, as well as preliminary evidence that this method recovers known misconceptions from real student responses. Our procedure is based on a Bayesian implementation of the Multidimensional Nominal Categories IRT model (MNCM) combined with standard factor-analytic rotation methods; by analyzing student responses at the level of individual distractors rather than at the level of entire questions, this approach is able to highlight multiple likely misconceptions for subsequent investigation without requiring any manual labeling of test content. We explore the performance of the Bayesian MNCM on synthetic data and find that it is able to recover multidimensional item parameters consistently at achievable sample sizes. These studies demonstrate the method's robustness to overfitting and ability to perform automatic dimensionality assessment and selection. The method also compares favorably to existing IRT software implementing marginal maximum likelihood estimation which we use as a validation benchmark. We then apply our method to approximately 10,000 students' responses to a research-designed concept test: the Force Concept Inventory. In addition to a broad first dimension strongly correlated with overall test score, we discover thirteen additional dimensions which load on smaller sets of distractors; we discuss two as examples, showing that these are consistent with already-known misconceptions in Newtonian mechanics. While work remains to validate our findings, our hope is that future applications of this method could aid in the refinement of existing concept inventories or the development of new ones, enable the discovery of previously-unknown student misconceptions across a variety of disciplines, and—by leveraging the method's ability to quantify the prevalence of particular misconceptions—provide opportunities for targeted instruction at both the individual and classroom level. |
| format | Article |
| id | doaj-art-831b23af80bd455ebe1f1b065a740d3f |
| institution | Kabale University |
| issn | 1664-1078 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj-art-831b23af80bd455ebe1f1b065a740d3f2025-01-29T06:45:53ZengFrontiers Media S.A.Frontiers in Psychology1664-10782025-01-011610.3389/fpsyg.2025.15063201506320A multidimensional Bayesian IRT method for discovering misconceptions from concept test dataMartin Segado0Aaron Adair1John Stewart2Yunfei Ma3Byron Drury4David Pritchard5Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, United StatesDepartment of Physics, Massachusetts Institute of Technology, Cambridge, MA, United StatesDepartment of Physics and Astronomy, West Virginia University, Morgantown, WV, United StatesDepartment of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, United StatesDepartment of Physics, Massachusetts Institute of Technology, Cambridge, MA, United StatesDepartment of Physics, Massachusetts Institute of Technology, Cambridge, MA, United StatesWe present an exploratory method for discovering likely misconceptions from multiple-choice concept test data, as well as preliminary evidence that this method recovers known misconceptions from real student responses. Our procedure is based on a Bayesian implementation of the Multidimensional Nominal Categories IRT model (MNCM) combined with standard factor-analytic rotation methods; by analyzing student responses at the level of individual distractors rather than at the level of entire questions, this approach is able to highlight multiple likely misconceptions for subsequent investigation without requiring any manual labeling of test content. We explore the performance of the Bayesian MNCM on synthetic data and find that it is able to recover multidimensional item parameters consistently at achievable sample sizes. These studies demonstrate the method's robustness to overfitting and ability to perform automatic dimensionality assessment and selection. The method also compares favorably to existing IRT software implementing marginal maximum likelihood estimation which we use as a validation benchmark. We then apply our method to approximately 10,000 students' responses to a research-designed concept test: the Force Concept Inventory. In addition to a broad first dimension strongly correlated with overall test score, we discover thirteen additional dimensions which load on smaller sets of distractors; we discuss two as examples, showing that these are consistent with already-known misconceptions in Newtonian mechanics. While work remains to validate our findings, our hope is that future applications of this method could aid in the refinement of existing concept inventories or the development of new ones, enable the discovery of previously-unknown student misconceptions across a variety of disciplines, and—by leveraging the method's ability to quantify the prevalence of particular misconceptions—provide opportunities for targeted instruction at both the individual and classroom level.https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1506320/fullitem response theorystudent misconceptionsmultiple-choice questionsdistractor analysismultidimensional nominal categories modelmean-field variational inference |
| spellingShingle | Martin Segado Aaron Adair John Stewart Yunfei Ma Byron Drury David Pritchard A multidimensional Bayesian IRT method for discovering misconceptions from concept test data Frontiers in Psychology item response theory student misconceptions multiple-choice questions distractor analysis multidimensional nominal categories model mean-field variational inference |
| title | A multidimensional Bayesian IRT method for discovering misconceptions from concept test data |
| title_full | A multidimensional Bayesian IRT method for discovering misconceptions from concept test data |
| title_fullStr | A multidimensional Bayesian IRT method for discovering misconceptions from concept test data |
| title_full_unstemmed | A multidimensional Bayesian IRT method for discovering misconceptions from concept test data |
| title_short | A multidimensional Bayesian IRT method for discovering misconceptions from concept test data |
| title_sort | multidimensional bayesian irt method for discovering misconceptions from concept test data |
| topic | item response theory student misconceptions multiple-choice questions distractor analysis multidimensional nominal categories model mean-field variational inference |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2025.1506320/full |
| work_keys_str_mv | AT martinsegado amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT aaronadair amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT johnstewart amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT yunfeima amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT byrondrury amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT davidpritchard amultidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT martinsegado multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT aaronadair multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT johnstewart multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT yunfeima multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT byrondrury multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata AT davidpritchard multidimensionalbayesianirtmethodfordiscoveringmisconceptionsfromconcepttestdata |