Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study
Objectives To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.Design Qualitative text analysis supplemented by quantit...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2021-07-01
|
| Series: | BMJ Open |
| Online Access: | https://bmjopen.bmj.com/content/11/7/e047019.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850284947095420928 |
|---|---|
| author | Maurice A Lembeck Jette Led Sørensen Hanne Pedersen Ellen Astrid Holm Shaymaa Jaafar Lafta Al-Bayati Toke Seierøe Barfod Emilie Ramberg Åse Kathrine Klemmensen |
| author_facet | Maurice A Lembeck Jette Led Sørensen Hanne Pedersen Ellen Astrid Holm Shaymaa Jaafar Lafta Al-Bayati Toke Seierøe Barfod Emilie Ramberg Åse Kathrine Klemmensen |
| author_sort | Maurice A Lembeck |
| collection | DOAJ |
| description | Objectives To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.Design Qualitative text analysis supplemented by quantitative descriptive analysis.Setting Internal Medicine Departments in Zealand, Denmark.Participants 48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).Intervention This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either ‘no reasons for concern’ or ‘possibly some concern’, thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.Results Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7–1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.Conclusions A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees’ performances based on the MSF reports. |
| format | Article |
| id | doaj-art-638257a33e204af6acc473d7ee2d3d2f |
| institution | OA Journals |
| issn | 2044-6055 |
| language | English |
| publishDate | 2021-07-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Open |
| spelling | doaj-art-638257a33e204af6acc473d7ee2d3d2f2025-08-20T01:47:25ZengBMJ Publishing GroupBMJ Open2044-60552021-07-0111710.1136/bmjopen-2020-047019Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method studyMaurice A Lembeck0Jette Led Sørensen1Hanne Pedersen2Ellen Astrid Holm3Shaymaa Jaafar Lafta Al-Bayati4Toke Seierøe Barfod5Emilie Ramberg6Åse Kathrine Klemmensen7Department of Internal Medicine, Nykobing F Sygehus, Nykobing Falster, DenmarkJuliane Marie Centre for Children, Women and Reproduction Section 4074, Rigshospitalet, University of Copenhagen, Copenhagen, DenmarkInternal Medicine, Zealand University Hospital Koge, Koge, DenmarkInternal Medicine, Zealand University Hospital Koge, Koge, DenmarkDepartment of Clinical Physiology and Nuclear Medicine, Zealand University Hospital Roskilde, Roskilde, DenmarkDepartment of Internal Medicine, Zealand University Hospital Roskilde, Roskilde, DenmarkDepartment of Internal Medicine, Nykøbing Falster Sygehus, Nykobing, DenmarkDepartment of Obstetrics and Gynecology, Rigshospitalet, Kobenhavn, DenmarkObjectives To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.Design Qualitative text analysis supplemented by quantitative descriptive analysis.Setting Internal Medicine Departments in Zealand, Denmark.Participants 48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).Intervention This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either ‘no reasons for concern’ or ‘possibly some concern’, thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.Results Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7–1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.Conclusions A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees’ performances based on the MSF reports.https://bmjopen.bmj.com/content/11/7/e047019.full |
| spellingShingle | Maurice A Lembeck Jette Led Sørensen Hanne Pedersen Ellen Astrid Holm Shaymaa Jaafar Lafta Al-Bayati Toke Seierøe Barfod Emilie Ramberg Åse Kathrine Klemmensen Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study BMJ Open |
| title | Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study |
| title_full | Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study |
| title_fullStr | Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study |
| title_full_unstemmed | Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study |
| title_short | Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study |
| title_sort | feasibility quality and validity of narrative multisource feedback in postgraduate training a mixed method study |
| url | https://bmjopen.bmj.com/content/11/7/e047019.full |
| work_keys_str_mv | AT mauricealembeck feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT jetteledsørensen feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT hannepedersen feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT ellenastridholm feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT shaymaajaafarlaftaalbayati feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT tokeseierøebarfod feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT emilieramberg feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy AT asekathrineklemmensen feasibilityqualityandvalidityofnarrativemultisourcefeedbackinpostgraduatetrainingamixedmethodstudy |