Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study
Abstract BackgroundOpenAI released versions ChatGPT-3.5 and GPT-4 between 2022 and 2023. GPT-3.5 has demonstrated proficiency in various examinations, particularly the United States Medical Licensing Examination. However, GPT-4 has more advanced capabilities. Objec...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-01-01
|
| Series: | JMIR Medical Education |
| Online Access: | https://mededu.jmir.org/2025/1/e56850 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832585111183294464 |
|---|---|
| author | Ying-Mei Wang Hung-Wei Shen Tzeng-Ji Chen Shu-Chiung Chiang Ting-Guan Lin |
| author_facet | Ying-Mei Wang Hung-Wei Shen Tzeng-Ji Chen Shu-Chiung Chiang Ting-Guan Lin |
| author_sort | Ying-Mei Wang |
| collection | DOAJ |
| description |
Abstract
BackgroundOpenAI released versions ChatGPT-3.5 and GPT-4 between 2022 and 2023. GPT-3.5 has demonstrated proficiency in various examinations, particularly the United States Medical Licensing Examination. However, GPT-4 has more advanced capabilities.
ObjectiveThis study aims to examine the efficacy of GPT-3.5 and GPT-4 within the Taiwan National Pharmacist Licensing Examination and to ascertain their utility and potential application in clinical pharmacy and education.
MethodsThe pharmacist examination in Taiwan consists of 2 stages: basic subjects and clinical subjects. In this study, exam questions were manually fed into the GPT-3.5 and GPT-4 models, and their responses were recorded; graphic-based questions were excluded. This study encompassed three steps: (1) determining the answering accuracy of GPT-3.5 and GPT-4, (2) categorizing question types and observing differences in model performance across these categories, and (3) comparing model performance on calculation and situational questions. Microsoft Excel and R software were used for statistical analyses.
ResultsGPT-4 achieved an accuracy rate of 72.9%, overshadowing GPT-3.5, which achieved 59.1% (PP
ConclusionsThis study demonstrates that GPT-4 outperforms GPT-3.5 in the Taiwan National Pharmacist Licensing Examination, particularly in basic subjects. While GPT-4 shows potential for use in clinical practice and pharmacy education, its limitations warrant caution. Future research should focus on refining prompts, improving model stability, integrating medical databases, and designing questions that better assess student competence and minimize guessing. |
| format | Article |
| id | doaj-art-03284571c2d84ba9ae90e09fbfc5de06 |
| institution | Kabale University |
| issn | 2369-3762 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Medical Education |
| spelling | doaj-art-03284571c2d84ba9ae90e09fbfc5de062025-01-27T02:52:29ZengJMIR PublicationsJMIR Medical Education2369-37622025-01-0111e56850e5685010.2196/56850Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation StudyYing-Mei Wanghttp://orcid.org/0009-0006-9823-5298Hung-Wei Shenhttp://orcid.org/0009-0002-6095-1185Tzeng-Ji Chenhttp://orcid.org/0000-0002-8350-0232Shu-Chiung Chianghttp://orcid.org/0000-0002-8795-2228Ting-Guan Linhttp://orcid.org/0009-0006-9138-9075 Abstract BackgroundOpenAI released versions ChatGPT-3.5 and GPT-4 between 2022 and 2023. GPT-3.5 has demonstrated proficiency in various examinations, particularly the United States Medical Licensing Examination. However, GPT-4 has more advanced capabilities. ObjectiveThis study aims to examine the efficacy of GPT-3.5 and GPT-4 within the Taiwan National Pharmacist Licensing Examination and to ascertain their utility and potential application in clinical pharmacy and education. MethodsThe pharmacist examination in Taiwan consists of 2 stages: basic subjects and clinical subjects. In this study, exam questions were manually fed into the GPT-3.5 and GPT-4 models, and their responses were recorded; graphic-based questions were excluded. This study encompassed three steps: (1) determining the answering accuracy of GPT-3.5 and GPT-4, (2) categorizing question types and observing differences in model performance across these categories, and (3) comparing model performance on calculation and situational questions. Microsoft Excel and R software were used for statistical analyses. ResultsGPT-4 achieved an accuracy rate of 72.9%, overshadowing GPT-3.5, which achieved 59.1% (PP ConclusionsThis study demonstrates that GPT-4 outperforms GPT-3.5 in the Taiwan National Pharmacist Licensing Examination, particularly in basic subjects. While GPT-4 shows potential for use in clinical practice and pharmacy education, its limitations warrant caution. Future research should focus on refining prompts, improving model stability, integrating medical databases, and designing questions that better assess student competence and minimize guessing.https://mededu.jmir.org/2025/1/e56850 |
| spellingShingle | Ying-Mei Wang Hung-Wei Shen Tzeng-Ji Chen Shu-Chiung Chiang Ting-Guan Lin Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study JMIR Medical Education |
| title | Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study |
| title_full | Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study |
| title_fullStr | Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study |
| title_full_unstemmed | Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study |
| title_short | Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study |
| title_sort | performance of chatgpt 3 5 and chatgpt 4 in the taiwan national pharmacist licensing examination comparative evaluation study |
| url | https://mededu.jmir.org/2025/1/e56850 |
| work_keys_str_mv | AT yingmeiwang performanceofchatgpt35andchatgpt4inthetaiwannationalpharmacistlicensingexaminationcomparativeevaluationstudy AT hungweishen performanceofchatgpt35andchatgpt4inthetaiwannationalpharmacistlicensingexaminationcomparativeevaluationstudy AT tzengjichen performanceofchatgpt35andchatgpt4inthetaiwannationalpharmacistlicensingexaminationcomparativeevaluationstudy AT shuchiungchiang performanceofchatgpt35andchatgpt4inthetaiwannationalpharmacistlicensingexaminationcomparativeevaluationstudy AT tingguanlin performanceofchatgpt35andchatgpt4inthetaiwannationalpharmacistlicensingexaminationcomparativeevaluationstudy |