Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification
Background:. This study assesses the effectiveness of large language models (LLMs) in simplifying complex language within orthopaedic patient education materials (PEMs) and identifies predictive factors for successful text transformation. Methods:. We transformed 48 orthopaedic PEMs using GPT-4, GPT...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wolters Kluwer
2025-03-01
|
| Series: | JBJS Open Access |
| Online Access: | http://journals.lww.com/jbjsoa/fulltext/10.2106/JBJS.OA.24.00007 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832589639322435584 |
|---|---|
| author | Saman Andalib, BS Sean S. Solomon, BS Bryce G. Picton, BS Aidin C. Spina, BS John A. Scolaro, MD Ariana M. Nelson, MD |
| author_facet | Saman Andalib, BS Sean S. Solomon, BS Bryce G. Picton, BS Aidin C. Spina, BS John A. Scolaro, MD Ariana M. Nelson, MD |
| author_sort | Saman Andalib, BS |
| collection | DOAJ |
| description | Background:. This study assesses the effectiveness of large language models (LLMs) in simplifying complex language within orthopaedic patient education materials (PEMs) and identifies predictive factors for successful text transformation.
Methods:. We transformed 48 orthopaedic PEMs using GPT-4, GPT-3.5, Claude 2, and Llama 2. The readability, quantified by the Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL) scores, was measured before and after transformation. Analysis included text characteristics such as syllable count, word length, and sentence length. Statistical and machine learning methods evaluated the correlations and predictive capacity of these features for transformation success.
Results:. All LLMs improved FKRE and FKGL scores (p < 0.01). GPT-4 showed superior performance, transforming PEMs to a seventh-grade reading level (mean FKGL, 6.72 ± 0.99), with higher FKRE and lower FKGL than other models. GPT-3.5, Claude 2, and Llama 2 significantly shortened sentences and overall text length (p < 0.01). Importantly, correlation analysis revealed that transformation success varied substantially with the model used, depending on original text factors such as word length and sentence complexity.
Conclusions:. LLMs successfully simplify orthopaedic PEMs, with GPT-4 leading in readability improvement. This study highlights the importance of initial text characteristics in determining the effectiveness of LLM transformations, offering insights for optimizing orthopaedic health literacy initiatives using artificial intelligence (AI).
Clinical Relevance:. This study provides critical insights into the ability of LLMs to simplify complex orthopaedic PEMs, enhancing their readability without compromising informational integrity. By identifying predictive factors for successful text transformation, this research supports the application of AI in improving health literacy, potentially leading to better patient comprehension and outcomes in orthopaedic care. |
| format | Article |
| id | doaj-art-7c5bffaa50e9475fa5df47a976f49f27 |
| institution | Kabale University |
| issn | 2472-7245 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Wolters Kluwer |
| record_format | Article |
| series | JBJS Open Access |
| spelling | doaj-art-7c5bffaa50e9475fa5df47a976f49f272025-01-24T09:20:07ZengWolters KluwerJBJS Open Access2472-72452025-03-0110110.2106/JBJS.OA.24.00007JBJSOA2400007Source Characteristics Influence AI-Enabled Orthopaedic Text SimplificationSaman Andalib, BS0Sean S. Solomon, BS1Bryce G. Picton, BS2Aidin C. Spina, BS3John A. Scolaro, MD4Ariana M. Nelson, MD51 University of California, Irvine, School of Medicine, Irvine, California1 University of California, Irvine, School of Medicine, Irvine, California1 University of California, Irvine, School of Medicine, Irvine, California1 University of California, Irvine, School of Medicine, Irvine, California2 Department of Orthopaedic Surgery, University of California, Irvine, Medical Center, Orange, California3 Department of Anesthesiology, University of California, Irvine, Medical Center, Orange, CaliforniaBackground:. This study assesses the effectiveness of large language models (LLMs) in simplifying complex language within orthopaedic patient education materials (PEMs) and identifies predictive factors for successful text transformation. Methods:. We transformed 48 orthopaedic PEMs using GPT-4, GPT-3.5, Claude 2, and Llama 2. The readability, quantified by the Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL) scores, was measured before and after transformation. Analysis included text characteristics such as syllable count, word length, and sentence length. Statistical and machine learning methods evaluated the correlations and predictive capacity of these features for transformation success. Results:. All LLMs improved FKRE and FKGL scores (p < 0.01). GPT-4 showed superior performance, transforming PEMs to a seventh-grade reading level (mean FKGL, 6.72 ± 0.99), with higher FKRE and lower FKGL than other models. GPT-3.5, Claude 2, and Llama 2 significantly shortened sentences and overall text length (p < 0.01). Importantly, correlation analysis revealed that transformation success varied substantially with the model used, depending on original text factors such as word length and sentence complexity. Conclusions:. LLMs successfully simplify orthopaedic PEMs, with GPT-4 leading in readability improvement. This study highlights the importance of initial text characteristics in determining the effectiveness of LLM transformations, offering insights for optimizing orthopaedic health literacy initiatives using artificial intelligence (AI). Clinical Relevance:. This study provides critical insights into the ability of LLMs to simplify complex orthopaedic PEMs, enhancing their readability without compromising informational integrity. By identifying predictive factors for successful text transformation, this research supports the application of AI in improving health literacy, potentially leading to better patient comprehension and outcomes in orthopaedic care.http://journals.lww.com/jbjsoa/fulltext/10.2106/JBJS.OA.24.00007 |
| spellingShingle | Saman Andalib, BS Sean S. Solomon, BS Bryce G. Picton, BS Aidin C. Spina, BS John A. Scolaro, MD Ariana M. Nelson, MD Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification JBJS Open Access |
| title | Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification |
| title_full | Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification |
| title_fullStr | Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification |
| title_full_unstemmed | Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification |
| title_short | Source Characteristics Influence AI-Enabled Orthopaedic Text Simplification |
| title_sort | source characteristics influence ai enabled orthopaedic text simplification |
| url | http://journals.lww.com/jbjsoa/fulltext/10.2106/JBJS.OA.24.00007 |
| work_keys_str_mv | AT samanandalibbs sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification AT seanssolomonbs sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification AT brycegpictonbs sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification AT aidincspinabs sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification AT johnascolaromd sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification AT arianamnelsonmd sourcecharacteristicsinfluenceaienabledorthopaedictextsimplification |