Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study
Abstract Background The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on exper...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-01-01
|
| Series: | BMC Medical Education |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12909-024-06631-4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832594638145323008 |
|---|---|
| author | Hervaldo Sampaio Carvalho Maria do Carmo Barros Melo Fabiana Maria Kakehasi Priscila Menezes Ferri Liu Fernanda Gontijo Minafra Silveira Santos Ricardo Luiz de Melo Martins Marilene Vale de Castro Monteiro Marcelo Esteves Chaves Campos Roberto Zonato Esteves Matheus Belloni Torsani Iolanda de Fátima Lopes Calvo Tiberio Akihito Inca Atahualpa Urdiales Andrea Mora De Marco Novellino Rosiane Viana Zuza Diniz Helena Ferreira Moura Maria Alice de Vilhena Toledo Célia Regina Machado Saldanha Andrea Pedrosa Ribeiro Alves Oliveira Patricia Zen Tempski Itamar de Souza Santos Milton de Arruda Martins |
| author_facet | Hervaldo Sampaio Carvalho Maria do Carmo Barros Melo Fabiana Maria Kakehasi Priscila Menezes Ferri Liu Fernanda Gontijo Minafra Silveira Santos Ricardo Luiz de Melo Martins Marilene Vale de Castro Monteiro Marcelo Esteves Chaves Campos Roberto Zonato Esteves Matheus Belloni Torsani Iolanda de Fátima Lopes Calvo Tiberio Akihito Inca Atahualpa Urdiales Andrea Mora De Marco Novellino Rosiane Viana Zuza Diniz Helena Ferreira Moura Maria Alice de Vilhena Toledo Célia Regina Machado Saldanha Andrea Pedrosa Ribeiro Alves Oliveira Patricia Zen Tempski Itamar de Souza Santos Milton de Arruda Martins |
| author_sort | Hervaldo Sampaio Carvalho |
| collection | DOAJ |
| description | Abstract Background The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on expert opinion, as tools for assessing the competencies needed under medical curriculum guidelines in Brazil. Methods In this multicenter study, the suitability levels of the OSCE and vOSCE for assessing the competencies needed under the Brazilian National Curriculum Guidelines (DCNs) were compared. The DCNs encompass five groups of domains, namely, health education; general; health management; public health needs; and individual health needs. Sixteen teaching experts served as judges. They responded to a questionnaire with the provided Likert scores indicating the ability of the OSCE and vOSCE to assess each type of competency. Virtual meetings were held via the Delphi method and a focus group-based approach to enable the experts to reach a consensus. Results A total of 200 items included in the guidelines (100 for the OSCE and 100 for the vOSCE) were evaluated across a total of 3,200 responses. Analysis via the Likert scale revealed a high proportion of positive evaluations for the use of the vOSCE, with kappa values ranging from 0.4 to 0.72. The values of the absolute agreement (positive view) between examiners and equivalence with the kappa statistic for the adoption of the vOSCE ranged from 0.38 to 0.72. Via the use of virtual meetings (Delphi and focus groups), consensus was reached regarding the capacity of the OSCE and vOSCE to evaluate 97% and 92% of these items, respectively. Conclusion The vOSCE can be employed to assess relevant competencies. However, it faces limitations regarding 8% (8/100) of the items, and some items (3%) cannot be assessed via either method. The difference in results between the two assessment methodologies (OSCE and vOSCE) is 5%. It is necessary to invest in the creation of instruments that can be used to apply the vOSCE and to conduct cost‒benefit analysis of its broader application in health education institutions. We conclude that the vOSCE is an effective tool for assessing most competencies of medical professionals and students required under Brazilian guidelines. |
| format | Article |
| id | doaj-art-9eb35273366442ef8933e0d3f09f8fd3 |
| institution | Kabale University |
| issn | 1472-6920 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Education |
| spelling | doaj-art-9eb35273366442ef8933e0d3f09f8fd32025-01-19T12:27:42ZengBMCBMC Medical Education1472-69202025-01-0125111310.1186/s12909-024-06631-4Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter studyHervaldo Sampaio Carvalho0Maria do Carmo Barros Melo1Fabiana Maria Kakehasi2Priscila Menezes Ferri Liu3Fernanda Gontijo Minafra Silveira Santos4Ricardo Luiz de Melo Martins5Marilene Vale de Castro Monteiro6Marcelo Esteves Chaves Campos7Roberto Zonato Esteves8Matheus Belloni Torsani9Iolanda de Fátima Lopes Calvo Tiberio10Akihito Inca Atahualpa Urdiales11Andrea Mora De Marco Novellino12Rosiane Viana Zuza Diniz13Helena Ferreira Moura14Maria Alice de Vilhena Toledo15Célia Regina Machado Saldanha16Andrea Pedrosa Ribeiro Alves Oliveira17Patricia Zen Tempski18Itamar de Souza Santos19Milton de Arruda Martins20Internal Medicine/Cardiology, Medical School, Brasília UniversityDepartment of Pediatrics, Federal University of Minas GeraisDepartment of Pediatrics, Federal University of Minas GeraisDepartment of Pediatrics, Federal University of Minas GeraisDepartment of Pediatrics, Federal University of Minas GeraisDepartment of Internal Medicine/Pulmonology, University of BrasíliaDepartment of Gynecology and Obstetrics, Federal University of Minas Geraiss GeraisDepartment of Surgery, Federal University of Minas GeraisInternal Medicine/Endocrinology, State University of MaringáCenter for Development of Medical Education, University of São PauloDepartment of Internal Medicine, University of São PauloDepartment of Medicine, Federal University of ParanáDepartment of Medicine, Federal University of Paraná Clinical Medicine Department, Federal University of Rio Grande do NorteDepartment of Internal Medicine, Brasília UniversityDepartment of Internal Medicine, Brasília UniversityFaculty of Medical and Health Sciences of Juiz de Fora, Suprema Surgical Clinic, Brasília UniversityUniversity of São PauloUniversity of São PauloUniversity of São PauloAbstract Background The COVID-19 pandemic has led to the popularization of information and communication technology in medical education. This study aimed to compare the objective structured clinical examination (OSCE) and the virtual objective structured clinical examination (vOSCE), based on expert opinion, as tools for assessing the competencies needed under medical curriculum guidelines in Brazil. Methods In this multicenter study, the suitability levels of the OSCE and vOSCE for assessing the competencies needed under the Brazilian National Curriculum Guidelines (DCNs) were compared. The DCNs encompass five groups of domains, namely, health education; general; health management; public health needs; and individual health needs. Sixteen teaching experts served as judges. They responded to a questionnaire with the provided Likert scores indicating the ability of the OSCE and vOSCE to assess each type of competency. Virtual meetings were held via the Delphi method and a focus group-based approach to enable the experts to reach a consensus. Results A total of 200 items included in the guidelines (100 for the OSCE and 100 for the vOSCE) were evaluated across a total of 3,200 responses. Analysis via the Likert scale revealed a high proportion of positive evaluations for the use of the vOSCE, with kappa values ranging from 0.4 to 0.72. The values of the absolute agreement (positive view) between examiners and equivalence with the kappa statistic for the adoption of the vOSCE ranged from 0.38 to 0.72. Via the use of virtual meetings (Delphi and focus groups), consensus was reached regarding the capacity of the OSCE and vOSCE to evaluate 97% and 92% of these items, respectively. Conclusion The vOSCE can be employed to assess relevant competencies. However, it faces limitations regarding 8% (8/100) of the items, and some items (3%) cannot be assessed via either method. The difference in results between the two assessment methodologies (OSCE and vOSCE) is 5%. It is necessary to invest in the creation of instruments that can be used to apply the vOSCE and to conduct cost‒benefit analysis of its broader application in health education institutions. We conclude that the vOSCE is an effective tool for assessing most competencies of medical professionals and students required under Brazilian guidelines.https://doi.org/10.1186/s12909-024-06631-4Medical EducationCompetency-Based EducationEducational MeasurementAudiovisual Aids |
| spellingShingle | Hervaldo Sampaio Carvalho Maria do Carmo Barros Melo Fabiana Maria Kakehasi Priscila Menezes Ferri Liu Fernanda Gontijo Minafra Silveira Santos Ricardo Luiz de Melo Martins Marilene Vale de Castro Monteiro Marcelo Esteves Chaves Campos Roberto Zonato Esteves Matheus Belloni Torsani Iolanda de Fátima Lopes Calvo Tiberio Akihito Inca Atahualpa Urdiales Andrea Mora De Marco Novellino Rosiane Viana Zuza Diniz Helena Ferreira Moura Maria Alice de Vilhena Toledo Célia Regina Machado Saldanha Andrea Pedrosa Ribeiro Alves Oliveira Patricia Zen Tempski Itamar de Souza Santos Milton de Arruda Martins Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study BMC Medical Education Medical Education Competency-Based Education Educational Measurement Audiovisual Aids |
| title | Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study |
| title_full | Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study |
| title_fullStr | Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study |
| title_full_unstemmed | Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study |
| title_short | Comparison of the adequacies of the OSCE and vOSCE to assess the competencies required under Brazilian medical curriculum guidelines: a multicenter study |
| title_sort | comparison of the adequacies of the osce and vosce to assess the competencies required under brazilian medical curriculum guidelines a multicenter study |
| topic | Medical Education Competency-Based Education Educational Measurement Audiovisual Aids |
| url | https://doi.org/10.1186/s12909-024-06631-4 |
| work_keys_str_mv | AT hervaldosampaiocarvalho comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT mariadocarmobarrosmelo comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT fabianamariakakehasi comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT priscilamenezesferriliu comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT fernandagontijominafrasilveirasantos comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT ricardoluizdemelomartins comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT marilenevaledecastromonteiro comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT marceloesteveschavescampos comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT robertozonatoesteves comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT matheusbellonitorsani comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT iolandadefatimalopescalvotiberio comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT akihitoincaatahualpaurdiales comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT andreamorademarconovellino comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT rosianevianazuzadiniz comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT helenaferreiramoura comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT mariaalicedevilhenatoledo comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT celiareginamachadosaldanha comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT andreapedrosaribeiroalvesoliveira comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT patriciazentempski comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT itamardesouzasantos comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy AT miltondearrudamartins comparisonoftheadequaciesoftheosceandvoscetoassessthecompetenciesrequiredunderbrazilianmedicalcurriculumguidelinesamulticenterstudy |