Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing
In recent intelligent transportation systems (ITS), it is important to recognize pedestrians and avoid collisions. Various sensors are used to detect pedestrians, and some research on pedestrian detection uses a visible light (RGB) camera and a far-infrared (FIR) camera. FIR cameras are significantl...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Open Journal of Intelligent Transportation Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10770282/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832590336783810560 |
|---|---|
| author | Masato Okuda Kota Yoshida Takeshi Fujino |
| author_facet | Masato Okuda Kota Yoshida Takeshi Fujino |
| author_sort | Masato Okuda |
| collection | DOAJ |
| description | In recent intelligent transportation systems (ITS), it is important to recognize pedestrians and avoid collisions. Various sensors are used to detect pedestrians, and some research on pedestrian detection uses a visible light (RGB) camera and a far-infrared (FIR) camera. FIR cameras are significantly affected by ambient temperatures such as summer and winter. However, few studies have focused on this property when evaluating pedestrian detection accuracy. Therefore, this paper investigates the effect of temperature change in real-time multispectral pedestrian detection. We created an original dataset with three subsets, Hot, Intermediate, and Cold, and evaluated temperature effects by changing the subsets during training and testing. We first evaluated YOLOv8s-4ch, which simply extended the input layer of YOLOv8 from 3 channels of RGB to 4 channels of RGB-FIR. To further improve detection performance, we built a new model called YOLOv8s-2stream. This model has two backbones for RGB and FIR, and fuses their feature maps in each resolution. We found that the model trained on a specific temperature subset dropped the test accuracy in other subsets. On the other hand, when training using a Mix set covering all temperature sets (Hot, Inter., Cold), the model achieved the highest accuracy through all conditions. Moreover, our YOLOv8s-2stream has improved by 3.9 points of accuracy (AP@0.5:0.95) compared to YOLOv8s-4ch, and achieved 73 FPS inference speed on Jetson. |
| format | Article |
| id | doaj-art-77e2ae15d12345fbad5c26660d69ae2f |
| institution | Kabale University |
| issn | 2687-7813 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Intelligent Transportation Systems |
| spelling | doaj-art-77e2ae15d12345fbad5c26660d69ae2f2025-01-24T00:02:56ZengIEEEIEEE Open Journal of Intelligent Transportation Systems2687-78132024-01-01579780910.1109/OJITS.2024.350791710770282Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature ChangingMasato Okuda0https://orcid.org/0009-0005-6952-4114Kota Yoshida1https://orcid.org/0000-0003-1293-6415Takeshi Fujino2Graduate School of Science and Engineering, Ritsumeikan University, Shiga, JapanDepartment of Science and Engineering, Ritsumeikan University, Shiga, JapanDepartment of Science and Engineering, Ritsumeikan University, Shiga, JapanIn recent intelligent transportation systems (ITS), it is important to recognize pedestrians and avoid collisions. Various sensors are used to detect pedestrians, and some research on pedestrian detection uses a visible light (RGB) camera and a far-infrared (FIR) camera. FIR cameras are significantly affected by ambient temperatures such as summer and winter. However, few studies have focused on this property when evaluating pedestrian detection accuracy. Therefore, this paper investigates the effect of temperature change in real-time multispectral pedestrian detection. We created an original dataset with three subsets, Hot, Intermediate, and Cold, and evaluated temperature effects by changing the subsets during training and testing. We first evaluated YOLOv8s-4ch, which simply extended the input layer of YOLOv8 from 3 channels of RGB to 4 channels of RGB-FIR. To further improve detection performance, we built a new model called YOLOv8s-2stream. This model has two backbones for RGB and FIR, and fuses their feature maps in each resolution. We found that the model trained on a specific temperature subset dropped the test accuracy in other subsets. On the other hand, when training using a Mix set covering all temperature sets (Hot, Inter., Cold), the model achieved the highest accuracy through all conditions. Moreover, our YOLOv8s-2stream has improved by 3.9 points of accuracy (AP@0.5:0.95) compared to YOLOv8s-4ch, and achieved 73 FPS inference speed on Jetson.https://ieeexplore.ieee.org/document/10770282/Object detectionpedestrian detectiondeep learningfar-infraredsensor fusion |
| spellingShingle | Masato Okuda Kota Yoshida Takeshi Fujino Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing IEEE Open Journal of Intelligent Transportation Systems Object detection pedestrian detection deep learning far-infrared sensor fusion |
| title | Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing |
| title_full | Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing |
| title_fullStr | Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing |
| title_full_unstemmed | Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing |
| title_short | Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing |
| title_sort | realtime multispectral pedestrian detection with visible and far infrared under ambient temperature changing |
| topic | Object detection pedestrian detection deep learning far-infrared sensor fusion |
| url | https://ieeexplore.ieee.org/document/10770282/ |
| work_keys_str_mv | AT masatookuda realtimemultispectralpedestriandetectionwithvisibleandfarinfraredunderambienttemperaturechanging AT kotayoshida realtimemultispectralpedestriandetectionwithvisibleandfarinfraredunderambienttemperaturechanging AT takeshifujino realtimemultispectralpedestriandetectionwithvisibleandfarinfraredunderambienttemperaturechanging |