Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability
This study examined a historical mixture model approach to the evaluation of ratings made in “gold standard” and two-rater 2×2 contingency tables. Peirce's i and the derived i average were discussed in relation to a widely used index of reliability in the behavioral sciences, Cohen's κ....
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2010-01-01
|
| Series: | Journal of Probability and Statistics |
| Online Access: | http://dx.doi.org/10.1155/2010/480364 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832564391314194432 |
|---|---|
| author | Beau Abar Eric Loken |
| author_facet | Beau Abar Eric Loken |
| author_sort | Beau Abar |
| collection | DOAJ |
| description | This study examined a historical mixture model approach to the evaluation of ratings made in “gold standard” and two-rater 2×2 contingency tables. Peirce's i and the derived i average were discussed in relation to a widely used index of reliability in the behavioral sciences, Cohen's κ. Sample size, population base rate of occurrence, the true “science of the method”, and guessing rates were manipulated across simulations. In “gold standard” situations, Peirce's i tended to recover the true reliability of ratings as well as better than κ. In two-rater situations, iave tended to recover the true reliability as well as better than κ in most situations. The empirical utility and potential theoretical benefits of mixture model methods in estimating reliability are discussed, as are the associations between the i statistics and other modern mixture model approaches. |
| format | Article |
| id | doaj-art-23b03a4f3a3b4109a92defb66c1a8a25 |
| institution | Kabale University |
| issn | 1687-952X 1687-9538 |
| language | English |
| publishDate | 2010-01-01 |
| publisher | Wiley |
| record_format | Article |
| series | Journal of Probability and Statistics |
| spelling | doaj-art-23b03a4f3a3b4109a92defb66c1a8a252025-02-03T01:11:09ZengWileyJournal of Probability and Statistics1687-952X1687-95382010-01-01201010.1155/2010/480364480364Peirce's i and Cohen's κ for 2×2 Measures of Rater ReliabilityBeau Abar0Eric Loken1Departement of Human Development and Family Studies, The Pennsylvania State University, University Park, PA 16802, USADepartement of Human Development and Family Studies, The Pennsylvania State University, University Park, PA 16802, USAThis study examined a historical mixture model approach to the evaluation of ratings made in “gold standard” and two-rater 2×2 contingency tables. Peirce's i and the derived i average were discussed in relation to a widely used index of reliability in the behavioral sciences, Cohen's κ. Sample size, population base rate of occurrence, the true “science of the method”, and guessing rates were manipulated across simulations. In “gold standard” situations, Peirce's i tended to recover the true reliability of ratings as well as better than κ. In two-rater situations, iave tended to recover the true reliability as well as better than κ in most situations. The empirical utility and potential theoretical benefits of mixture model methods in estimating reliability are discussed, as are the associations between the i statistics and other modern mixture model approaches.http://dx.doi.org/10.1155/2010/480364 |
| spellingShingle | Beau Abar Eric Loken Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability Journal of Probability and Statistics |
| title | Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability |
| title_full | Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability |
| title_fullStr | Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability |
| title_full_unstemmed | Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability |
| title_short | Peirce's i and Cohen's κ for 2×2 Measures of Rater Reliability |
| title_sort | peirce s i and cohen s κ for 2 2 measures of rater reliability |
| url | http://dx.doi.org/10.1155/2010/480364 |
| work_keys_str_mv | AT beauabar peircesiandcohenskfor22measuresofraterreliability AT ericloken peircesiandcohenskfor22measuresofraterreliability |