Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric
Abstract The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner’s (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluatin...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2025-01-01
|
| Series: | Language Testing in Asia |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40468-024-00338-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832571471019376640 |
|---|---|
| author | Yuichiro Yokouchi |
| author_facet | Yuichiro Yokouchi |
| author_sort | Yuichiro Yokouchi |
| collection | DOAJ |
| description | Abstract The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner’s (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluating multiple binary-choice descriptors. Additionally, learners receive easy-to-understand feedback. The most important advantage is that the number of feedback types can be increased to the nth power of 2 (n = number of items). Therefore, this study compared the validity, reliability, and practicality of the PDT and grid-style rubrics. Three Japanese university English instructors participated in the preliminary work to create the rubric, and the contributors scrutinized the information to create a pilot version. Five raters (two English lecturers and three student raters) then scored 64 Japanese university students’ performances in four English-speaking tasks. Next, analyses were conducted using the many-facet Rasch measurement and generalizability theory to observe the validity and reliability of the PDT. The results confirmed that both the PDT and the grid had validity within the range that could be examined and that the reliability was sufficiently high. However, interviews with raters revealed that the PDT had superior practicality. Additionally, the PDT took less time to complete the evaluation than the grid and was judged to be an effective form of the rubric. This article is part of a study aimed at demonstrating the usefulness of the PDT by creating and evaluating a pilot version. It was used to partially examine the validity and reliability of the PDT. The remaining aspects of validation shall be presented in a separate article. |
| format | Article |
| id | doaj-art-b65248c5ecc342dc91d2e656bfe6ce31 |
| institution | Kabale University |
| issn | 2229-0443 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Language Testing in Asia |
| spelling | doaj-art-b65248c5ecc342dc91d2e656bfe6ce312025-02-02T12:36:32ZengSpringerOpenLanguage Testing in Asia2229-04432025-01-0115112610.1186/s40468-024-00338-5Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubricYuichiro Yokouchi0Fukushima UniversityAbstract The performance decision tree (PDT; Fulcher et al., 2011) is a rubric style that is applicable to performance assessment, with origins in Upshur and Turner’s (1995) empirically derived binary-choice, boundary-definition (EBB) scale. It is easier for raters to assess performance by evaluating multiple binary-choice descriptors. Additionally, learners receive easy-to-understand feedback. The most important advantage is that the number of feedback types can be increased to the nth power of 2 (n = number of items). Therefore, this study compared the validity, reliability, and practicality of the PDT and grid-style rubrics. Three Japanese university English instructors participated in the preliminary work to create the rubric, and the contributors scrutinized the information to create a pilot version. Five raters (two English lecturers and three student raters) then scored 64 Japanese university students’ performances in four English-speaking tasks. Next, analyses were conducted using the many-facet Rasch measurement and generalizability theory to observe the validity and reliability of the PDT. The results confirmed that both the PDT and the grid had validity within the range that could be examined and that the reliability was sufficiently high. However, interviews with raters revealed that the PDT had superior practicality. Additionally, the PDT took less time to complete the evaluation than the grid and was judged to be an effective form of the rubric. This article is part of a study aimed at demonstrating the usefulness of the PDT by creating and evaluating a pilot version. It was used to partially examine the validity and reliability of the PDT. The remaining aspects of validation shall be presented in a separate article.https://doi.org/10.1186/s40468-024-00338-5Performance decision tree (PDT)RubricPerformance assessmentMany-facet Rasch measurement (MFRM)Generalizability theory |
| spellingShingle | Yuichiro Yokouchi Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric Language Testing in Asia Performance decision tree (PDT) Rubric Performance assessment Many-facet Rasch measurement (MFRM) Generalizability theory |
| title | Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric |
| title_full | Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric |
| title_fullStr | Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric |
| title_full_unstemmed | Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric |
| title_short | Revisiting the effectiveness of a performance decision tree-style rubric compared to a grid-style rubric |
| title_sort | revisiting the effectiveness of a performance decision tree style rubric compared to a grid style rubric |
| topic | Performance decision tree (PDT) Rubric Performance assessment Many-facet Rasch measurement (MFRM) Generalizability theory |
| url | https://doi.org/10.1186/s40468-024-00338-5 |
| work_keys_str_mv | AT yuichiroyokouchi revisitingtheeffectivenessofaperformancedecisiontreestylerubriccomparedtoagridstylerubric |