LLM-driven multimodal target volume contouring in radiation oncology
Abstract Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of large language models (LLMs) that...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2024-10-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-024-53387-y |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832594607695724544 |
|---|---|
| author | Yujin Oh Sangjoon Park Hwa Kyung Byun Yeona Cho Ik Jae Lee Jin Sung Kim Jong Chul Ye |
| author_facet | Yujin Oh Sangjoon Park Hwa Kyung Byun Yeona Cho Ik Jae Lee Jin Sung Kim Jong Chul Ye |
| author_sort | Yujin Oh |
| collection | DOAJ |
| description | Abstract Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of large language models (LLMs) that can facilitate the integration of the textural information and images, here we present an LLM-driven multimodal artificial intelligence (AI), namely LLMSeg, that utilizes the clinical information and is applicable to the challenging task of 3-dimensional context-aware target volume delineation for radiation oncology. We validate our proposed LLMSeg within the context of breast cancer radiotherapy using external validation and data-insufficient environments, which attributes highly conducive to real-world applications. We demonstrate that the proposed multimodal LLMSeg exhibits markedly improved performance compared to conventional unimodal AI models, particularly exhibiting robust generalization performance and data-efficiency. |
| format | Article |
| id | doaj-art-2a0f76528c044de2845e8ca1e5f6dc07 |
| institution | Kabale University |
| issn | 2041-1723 |
| language | English |
| publishDate | 2024-10-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-2a0f76528c044de2845e8ca1e5f6dc072025-01-19T12:29:33ZengNature PortfolioNature Communications2041-17232024-10-0115111410.1038/s41467-024-53387-yLLM-driven multimodal target volume contouring in radiation oncologyYujin Oh0Sangjoon Park1Hwa Kyung Byun2Yeona Cho3Ik Jae Lee4Jin Sung Kim5Jong Chul Ye6Department of Radiology, Massachusetts General Hospital (MGH) and Harvard Medical SchoolDepartment of Radiation Oncology, Yonsei University College of MedicineDepartment of Radiation Oncology, Yongin Severance HospitalDepartment of Radiation Oncology, Gangnam Severance HospitalDepartment of Radiation Oncology, Yonsei University College of MedicineDepartment of Radiation Oncology, Yonsei University College of MedicineKim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)Abstract Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of large language models (LLMs) that can facilitate the integration of the textural information and images, here we present an LLM-driven multimodal artificial intelligence (AI), namely LLMSeg, that utilizes the clinical information and is applicable to the challenging task of 3-dimensional context-aware target volume delineation for radiation oncology. We validate our proposed LLMSeg within the context of breast cancer radiotherapy using external validation and data-insufficient environments, which attributes highly conducive to real-world applications. We demonstrate that the proposed multimodal LLMSeg exhibits markedly improved performance compared to conventional unimodal AI models, particularly exhibiting robust generalization performance and data-efficiency.https://doi.org/10.1038/s41467-024-53387-y |
| spellingShingle | Yujin Oh Sangjoon Park Hwa Kyung Byun Yeona Cho Ik Jae Lee Jin Sung Kim Jong Chul Ye LLM-driven multimodal target volume contouring in radiation oncology Nature Communications |
| title | LLM-driven multimodal target volume contouring in radiation oncology |
| title_full | LLM-driven multimodal target volume contouring in radiation oncology |
| title_fullStr | LLM-driven multimodal target volume contouring in radiation oncology |
| title_full_unstemmed | LLM-driven multimodal target volume contouring in radiation oncology |
| title_short | LLM-driven multimodal target volume contouring in radiation oncology |
| title_sort | llm driven multimodal target volume contouring in radiation oncology |
| url | https://doi.org/10.1038/s41467-024-53387-y |
| work_keys_str_mv | AT yujinoh llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT sangjoonpark llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT hwakyungbyun llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT yeonacho llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT ikjaelee llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT jinsungkim llmdrivenmultimodaltargetvolumecontouringinradiationoncology AT jongchulye llmdrivenmultimodaltargetvolumecontouringinradiationoncology |