Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion
In autonomous vehicles (AV), sensor fusion methods have proven to be effective in merging data from multiple sensors and enhancing their perception capabilities. In the context of sensor fusion, the distinct strengths of multi-sensors, such as LiDAR, RGB, Thermal sensors, etc., can be leveraged to m...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of Vehicular Technology |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10841396/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832540517952389120 |
|---|---|
| author | Mehmood Nawaz Sheheryar Khan Muhammad Daud Muhammad Asim Ghazanfar Ali Anwar Ali Raza Shahid Ho Pui Aaron HO Tom Chan Daniel Pak Kong Wu Yuan |
| author_facet | Mehmood Nawaz Sheheryar Khan Muhammad Daud Muhammad Asim Ghazanfar Ali Anwar Ali Raza Shahid Ho Pui Aaron HO Tom Chan Daniel Pak Kong Wu Yuan |
| author_sort | Mehmood Nawaz |
| collection | DOAJ |
| description | In autonomous vehicles (AV), sensor fusion methods have proven to be effective in merging data from multiple sensors and enhancing their perception capabilities. In the context of sensor fusion, the distinct strengths of multi-sensors, such as LiDAR, RGB, Thermal sensors, etc., can be leveraged to mitigate the impact of challenges imposed by extreme weather conditions. In this paper, we address multi-sensor fusion in AVs and present a comprehensive integration of a thermal sensor aimed at enhancing the cognitive robustness of AVs. Thermal sensors possess an impressive capability to detect objects and hazards that may be imperceptible to traditional visible light sensors. When integrated with RGB and LiDAR sensors, the thermal sensor becomes highly beneficial for detecting and locating objects in adverse weather conditions. The proposed deep learning-assisted multi-sensor fusion technique consists of two parts: (1) visual information fusion and (2) object detection using LiDAR, RGB, and Thermal sensors. The visual fusion framework employs a CNN (convolutional neural network) inspired by a domain image fusion algorithm. The object detection framework uses the modified version of the YoloV8 model, which exhibits high accuracy in real-time detection. In the YoloV8 model, we adjusted the network architecture to incorporate additional convolutional layers and altered the loss function to enhance detection accuracy in foggy and rainy conditions. The proposed technique is effective and adaptable in challenging conditions, such as night or dark mode, smoke, and heavy rain. The experimental results of the proposed method demonstrate enhanced efficiency and cognitive robustness compared to state-of-the-art fusion and detection techniques. This is evident from tests conducted on two public datasets (FLIR and TarDAL) and one private dataset (CUHK). |
| format | Article |
| id | doaj-art-335463d0952649b7b8d7bb5ab4cff0a2 |
| institution | Kabale University |
| issn | 2644-1330 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Vehicular Technology |
| spelling | doaj-art-335463d0952649b7b8d7bb5ab4cff0a22025-02-05T00:01:20ZengIEEEIEEE Open Journal of Vehicular Technology2644-13302025-01-01642644110.1109/OJVT.2025.352949510841396Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera FusionMehmood Nawaz0https://orcid.org/0000-0002-0978-2163Sheheryar Khan1https://orcid.org/0000-0002-1975-4334Muhammad Daud2Muhammad Asim3https://orcid.org/0000-0003-2145-4880Ghazanfar Ali Anwar4Ali Raza Shahid5https://orcid.org/0000-0002-6877-1405Ho Pui Aaron HO6https://orcid.org/0000-0001-5001-5911Tom Chan7https://orcid.org/0009-0007-2052-5508Daniel Pak Kong8https://orcid.org/0000-0003-3891-1363Wu Yuan9https://orcid.org/0000-0001-9405-519XDepartment of Biomedical Engineering, The Chinese University of Hong Kong, Hong KongDivision of Science, Engineering, and Health Studies (SEHS), School of Professional Education & Executive Development, The Hong Kong Polytechnic University, Kowloon, Hong KongDepartment of Agriculture, Bahauddin Zakariya University, Multan, PakistanDivision of Science, Engineering, and Health Studies (SEHS), School of Professional Education & Executive Development, The Hong Kong Polytechnic University, Kowloon, Hong KongDivision of Science, Engineering, and Health Studies (SEHS), School of Professional Education & Executive Development, The Hong Kong Polytechnic University, Kowloon, Hong KongDivision of Science, Engineering, and Health Studies (SEHS), School of Professional Education & Executive Development, The Hong Kong Polytechnic University, Kowloon, Hong KongDepartment of Biomedical Engineering, The Chinese University of Hong Kong, Hong KongDivision of Science, Engineering, and Health Studies (SEHS), School of Professional Education & Executive Development, The Hong Kong Polytechnic University, Kowloon, Hong KongDepartment of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hong KongDepartment of Biomedical Engineering, The Chinese University of Hong Kong, Hong KongIn autonomous vehicles (AV), sensor fusion methods have proven to be effective in merging data from multiple sensors and enhancing their perception capabilities. In the context of sensor fusion, the distinct strengths of multi-sensors, such as LiDAR, RGB, Thermal sensors, etc., can be leveraged to mitigate the impact of challenges imposed by extreme weather conditions. In this paper, we address multi-sensor fusion in AVs and present a comprehensive integration of a thermal sensor aimed at enhancing the cognitive robustness of AVs. Thermal sensors possess an impressive capability to detect objects and hazards that may be imperceptible to traditional visible light sensors. When integrated with RGB and LiDAR sensors, the thermal sensor becomes highly beneficial for detecting and locating objects in adverse weather conditions. The proposed deep learning-assisted multi-sensor fusion technique consists of two parts: (1) visual information fusion and (2) object detection using LiDAR, RGB, and Thermal sensors. The visual fusion framework employs a CNN (convolutional neural network) inspired by a domain image fusion algorithm. The object detection framework uses the modified version of the YoloV8 model, which exhibits high accuracy in real-time detection. In the YoloV8 model, we adjusted the network architecture to incorporate additional convolutional layers and altered the loss function to enhance detection accuracy in foggy and rainy conditions. The proposed technique is effective and adaptable in challenging conditions, such as night or dark mode, smoke, and heavy rain. The experimental results of the proposed method demonstrate enhanced efficiency and cognitive robustness compared to state-of-the-art fusion and detection techniques. This is evident from tests conducted on two public datasets (FLIR and TarDAL) and one private dataset (CUHK).https://ieeexplore.ieee.org/document/10841396/Sensor fusionThermal cameraObject detectionObject recognitionconvolution neural network |
| spellingShingle | Mehmood Nawaz Sheheryar Khan Muhammad Daud Muhammad Asim Ghazanfar Ali Anwar Ali Raza Shahid Ho Pui Aaron HO Tom Chan Daniel Pak Kong Wu Yuan Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion IEEE Open Journal of Vehicular Technology Sensor fusion Thermal camera Object detection Object recognition convolution neural network |
| title | Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion |
| title_full | Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion |
| title_fullStr | Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion |
| title_full_unstemmed | Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion |
| title_short | Improving Autonomous Vehicle Cognitive Robustness in Extreme Weather With Deep Learning and Thermal Camera Fusion |
| title_sort | improving autonomous vehicle cognitive robustness in extreme weather with deep learning and thermal camera fusion |
| topic | Sensor fusion Thermal camera Object detection Object recognition convolution neural network |
| url | https://ieeexplore.ieee.org/document/10841396/ |
| work_keys_str_mv | AT mehmoodnawaz improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT sheheryarkhan improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT muhammaddaud improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT muhammadasim improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT ghazanfaralianwar improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT alirazashahid improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT hopuiaaronho improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT tomchan improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT danielpakkong improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion AT wuyuan improvingautonomousvehiclecognitiverobustnessinextremeweatherwithdeeplearningandthermalcamerafusion |