Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values
As measures of interobserver agreement for both nominal and ordinal categories, Cohen’s kappa coefficients appear to be the most widely used with simple and meaningful interpretations. However, for negative coefficient values when (the probability of) observed disagreement exceeds chance-expected di...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2015-01-01
|
| Series: | Journal of Probability and Statistics |
| Online Access: | http://dx.doi.org/10.1155/2015/751803 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832556342458449920 |
|---|---|
| author | Tarald O. Kvålseth |
| author_facet | Tarald O. Kvålseth |
| author_sort | Tarald O. Kvålseth |
| collection | DOAJ |
| description | As measures of interobserver agreement for both nominal and ordinal categories, Cohen’s kappa coefficients appear to be the most widely used with simple and meaningful interpretations. However, for negative coefficient values when (the probability of) observed disagreement exceeds chance-expected disagreement, no fixed lower bounds exist for the kappa coefficients and their interpretations are no longer meaningful and may be entirely misleading. In this paper, alternative measures of disagreement (or negative agreement) are proposed as simple corrections or modifications of Cohen’s kappa coefficients. The new coefficients have a fixed lower bound of −1 that can be attained irrespective of the marginal distributions. A coefficient is formulated for the case when the classification categories are nominal and a weighted coefficient is proposed for ordinal categories. Besides coefficients for the overall disagreement across categories, disagreement coefficients for individual categories are presented. Statistical inference procedures are developed and numerical examples are provided. |
| format | Article |
| id | doaj-art-76edf271057b494183b01a31e2a10ce4 |
| institution | Kabale University |
| issn | 1687-952X 1687-9538 |
| language | English |
| publishDate | 2015-01-01 |
| publisher | Wiley |
| record_format | Article |
| series | Journal of Probability and Statistics |
| spelling | doaj-art-76edf271057b494183b01a31e2a10ce42025-02-03T05:45:38ZengWileyJournal of Probability and Statistics1687-952X1687-95382015-01-01201510.1155/2015/751803751803Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative ValuesTarald O. Kvålseth0Departments of Mechanical Engineering and Industrial & Systems Engineering, University of Minnesota, Minneapolis, MN 55455, USAAs measures of interobserver agreement for both nominal and ordinal categories, Cohen’s kappa coefficients appear to be the most widely used with simple and meaningful interpretations. However, for negative coefficient values when (the probability of) observed disagreement exceeds chance-expected disagreement, no fixed lower bounds exist for the kappa coefficients and their interpretations are no longer meaningful and may be entirely misleading. In this paper, alternative measures of disagreement (or negative agreement) are proposed as simple corrections or modifications of Cohen’s kappa coefficients. The new coefficients have a fixed lower bound of −1 that can be attained irrespective of the marginal distributions. A coefficient is formulated for the case when the classification categories are nominal and a weighted coefficient is proposed for ordinal categories. Besides coefficients for the overall disagreement across categories, disagreement coefficients for individual categories are presented. Statistical inference procedures are developed and numerical examples are provided.http://dx.doi.org/10.1155/2015/751803 |
| spellingShingle | Tarald O. Kvålseth Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values Journal of Probability and Statistics |
| title | Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values |
| title_full | Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values |
| title_fullStr | Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values |
| title_full_unstemmed | Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values |
| title_short | Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values |
| title_sort | measurement of interobserver disagreement correction of cohen s kappa for negative values |
| url | http://dx.doi.org/10.1155/2015/751803 |
| work_keys_str_mv | AT taraldokvalseth measurementofinterobserverdisagreementcorrectionofcohenskappafornegativevalues |