Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy
The advancement of neural radiance fields (NeRFs) has facilitated the high-quality 3D reconstruction of complex scenes. However, for most NeRFs, reconstructing 3D tissues from endoscopy images poses significant challenges due to the occlusion of soft tissue regions by invalid pixels, deformations in...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/25/2/565 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832587526248857600 |
|---|---|
| author | Jinhua Liu Yongsheng Shi Dongjin Huang Jiantao Qu |
| author_facet | Jinhua Liu Yongsheng Shi Dongjin Huang Jiantao Qu |
| author_sort | Jinhua Liu |
| collection | DOAJ |
| description | The advancement of neural radiance fields (NeRFs) has facilitated the high-quality 3D reconstruction of complex scenes. However, for most NeRFs, reconstructing 3D tissues from endoscopy images poses significant challenges due to the occlusion of soft tissue regions by invalid pixels, deformations in soft tissue, and poor image quality, which severely limits their application in endoscopic scenarios. To address the above issues, we propose a novel framework to reconstruct high-fidelity soft tissue scenes from low-quality endoscopic images. We first construct an EndoTissue dataset of soft tissue regions in endoscopic images and fine-tune the Segment Anything Model (SAM) based on EndoTissue to obtain a potent segmentation network. Given a sequence of monocular endoscopic images, this segmentation network can quickly obtain the tissue mask images. Additionally, we incorporate tissue masks into a dynamic scene reconstruction method called Tensor4D to effectively guide the reconstruction of 3D deformable soft tissues. Finally, we propose adopting the image enhancement model EDAU-Net to improve the quality of the rendered views. The experimental results show that our method can effectively focus on the soft tissue regions in the image, achieving higher fidelity in detail and geometric structural integrity in reconstruction compared to state-of-the-art algorithms. Feedback from the user study indicates high participant scores for our method. |
| format | Article |
| id | doaj-art-ca1de3bdae574e70a3e4d512a557bef4 |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-ca1de3bdae574e70a3e4d512a557bef42025-01-24T13:49:22ZengMDPI AGSensors1424-82202025-01-0125256510.3390/s25020565Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in EndoscopyJinhua Liu0Yongsheng Shi1Dongjin Huang2Jiantao Qu3Shanghai Film Academy, Shanghai University, Shanghai 200072, ChinaShanghai Film Academy, Shanghai University, Shanghai 200072, ChinaShanghai Film Academy, Shanghai University, Shanghai 200072, ChinaShanghai Film Academy, Shanghai University, Shanghai 200072, ChinaThe advancement of neural radiance fields (NeRFs) has facilitated the high-quality 3D reconstruction of complex scenes. However, for most NeRFs, reconstructing 3D tissues from endoscopy images poses significant challenges due to the occlusion of soft tissue regions by invalid pixels, deformations in soft tissue, and poor image quality, which severely limits their application in endoscopic scenarios. To address the above issues, we propose a novel framework to reconstruct high-fidelity soft tissue scenes from low-quality endoscopic images. We first construct an EndoTissue dataset of soft tissue regions in endoscopic images and fine-tune the Segment Anything Model (SAM) based on EndoTissue to obtain a potent segmentation network. Given a sequence of monocular endoscopic images, this segmentation network can quickly obtain the tissue mask images. Additionally, we incorporate tissue masks into a dynamic scene reconstruction method called Tensor4D to effectively guide the reconstruction of 3D deformable soft tissues. Finally, we propose adopting the image enhancement model EDAU-Net to improve the quality of the rendered views. The experimental results show that our method can effectively focus on the soft tissue regions in the image, achieving higher fidelity in detail and geometric structural integrity in reconstruction compared to state-of-the-art algorithms. Feedback from the user study indicates high participant scores for our method.https://www.mdpi.com/1424-8220/25/2/565endoscopic image3D reconstructionneural radiance fieldssoft tissue dynamicsimage segmentation |
| spellingShingle | Jinhua Liu Yongsheng Shi Dongjin Huang Jiantao Qu Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy Sensors endoscopic image 3D reconstruction neural radiance fields soft tissue dynamics image segmentation |
| title | Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy |
| title_full | Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy |
| title_fullStr | Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy |
| title_full_unstemmed | Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy |
| title_short | Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy |
| title_sort | neural radiance fields for high fidelity soft tissue reconstruction in endoscopy |
| topic | endoscopic image 3D reconstruction neural radiance fields soft tissue dynamics image segmentation |
| url | https://www.mdpi.com/1424-8220/25/2/565 |
| work_keys_str_mv | AT jinhualiu neuralradiancefieldsforhighfidelitysofttissuereconstructioninendoscopy AT yongshengshi neuralradiancefieldsforhighfidelitysofttissuereconstructioninendoscopy AT dongjinhuang neuralradiancefieldsforhighfidelitysofttissuereconstructioninendoscopy AT jiantaoqu neuralradiancefieldsforhighfidelitysofttissuereconstructioninendoscopy |