Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos
Near-miss traffic incidents, positioned just above "unsafe acts" on the safety triangle theory, offer crucial predictive insights for preventing crashes. However, these incidents are often underrepresented in traffic safety research, which tends to focus primarily on actual crashes. This s...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of the Computer Society |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10820995/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832583991092314112 |
|---|---|
| author | Shadi Jaradat Mohammed Elhenawy Huthaifa I. Ashqar Alexander Paz Richi Nayak |
| author_facet | Shadi Jaradat Mohammed Elhenawy Huthaifa I. Ashqar Alexander Paz Richi Nayak |
| author_sort | Shadi Jaradat |
| collection | DOAJ |
| description | Near-miss traffic incidents, positioned just above "unsafe acts" on the safety triangle theory, offer crucial predictive insights for preventing crashes. However, these incidents are often underrepresented in traffic safety research, which tends to focus primarily on actual crashes. This study introduces a novel AI-based framework designed to detect and analyze near-miss and crash events in crowdsourced dashcam footage. The framework consists of two key components: a deep learning model to segment video streams and identify potential near-miss or crash incidents and a multimodal large language model (MLLM) to further analyze and extract narrative information from the identified events. We evaluated three deep learning models—CNN, Vision Transformers (ViTs), and CNN+LSTM—on a dataset specifically curated for three-class classification (crashes, near-misses, and normal driving events). CNN achieved the highest accuracy (90%) and F1-score (89%) at the frame level. At the event level, ViTs delivered a strong performance with a test accuracy of 77.27% and an F1-score of 67.37%, while CNN+LSTM, although lower in overall performance, demonstrated significant potential with a test accuracy of 78.1% and an F1-score of 68.69%. For a deeper analysis, we applied GPT-4o to process critical safety events (near-misses and crashes), utilizing both zero-shot and few-shot learning for narrative generation and feature extraction. The zero-shot learning method performed better, achieving an accuracy of 81.2% and an F1-score of 81.9%. This study underscores the potential of combining deep learning with MLLMs to enhance traffic safety analysis by integrating near-miss data as a key predictive layer. Our approach highlights the importance of leveraging near-miss incidents to proactively enhance road safety, thereby reducing the likelihood of crashes through early intervention and better event understanding. |
| format | Article |
| id | doaj-art-9eb20ccdc47746fe917cac873414274c |
| institution | Kabale University |
| issn | 2644-1268 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of the Computer Society |
| spelling | doaj-art-9eb20ccdc47746fe917cac873414274c2025-01-28T00:02:05ZengIEEEIEEE Open Journal of the Computer Society2644-12682025-01-01622323510.1109/OJCS.2025.352556010820995Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced VideosShadi Jaradat0https://orcid.org/0000-0001-6322-2792Mohammed Elhenawy1https://orcid.org/0000-0003-2634-4576Huthaifa I. Ashqar2Alexander Paz3https://orcid.org/0000-0002-1217-9808Richi Nayak4https://orcid.org/0000-0002-9954-0159Queensland University of Technology, Brisbane, QLD, AustraliaQueensland University of Technology, Brisbane, QLD, AustraliaArab American University, Jenin, PalestineQueensland University of Technology, Brisbane, QLD, AustraliaQueensland University of Technology, Brisbane, QLD, AustraliaNear-miss traffic incidents, positioned just above "unsafe acts" on the safety triangle theory, offer crucial predictive insights for preventing crashes. However, these incidents are often underrepresented in traffic safety research, which tends to focus primarily on actual crashes. This study introduces a novel AI-based framework designed to detect and analyze near-miss and crash events in crowdsourced dashcam footage. The framework consists of two key components: a deep learning model to segment video streams and identify potential near-miss or crash incidents and a multimodal large language model (MLLM) to further analyze and extract narrative information from the identified events. We evaluated three deep learning models—CNN, Vision Transformers (ViTs), and CNN+LSTM—on a dataset specifically curated for three-class classification (crashes, near-misses, and normal driving events). CNN achieved the highest accuracy (90%) and F1-score (89%) at the frame level. At the event level, ViTs delivered a strong performance with a test accuracy of 77.27% and an F1-score of 67.37%, while CNN+LSTM, although lower in overall performance, demonstrated significant potential with a test accuracy of 78.1% and an F1-score of 68.69%. For a deeper analysis, we applied GPT-4o to process critical safety events (near-misses and crashes), utilizing both zero-shot and few-shot learning for narrative generation and feature extraction. The zero-shot learning method performed better, achieving an accuracy of 81.2% and an F1-score of 81.9%. This study underscores the potential of combining deep learning with MLLMs to enhance traffic safety analysis by integrating near-miss data as a key predictive layer. Our approach highlights the importance of leveraging near-miss incidents to proactively enhance road safety, thereby reducing the likelihood of crashes through early intervention and better event understanding.https://ieeexplore.ieee.org/document/10820995/Near-miss detectioncrowdsourceproactive approachconvolutional neural network (CNN)vision transformerLSTM |
| spellingShingle | Shadi Jaradat Mohammed Elhenawy Huthaifa I. Ashqar Alexander Paz Richi Nayak Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos IEEE Open Journal of the Computer Society Near-miss detection crowdsource proactive approach convolutional neural network (CNN) vision transformer LSTM |
| title | Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos |
| title_full | Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos |
| title_fullStr | Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos |
| title_full_unstemmed | Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos |
| title_short | Leveraging Deep Learning and Multimodal Large Language Models for Near-Miss Detection Using Crowdsourced Videos |
| title_sort | leveraging deep learning and multimodal large language models for near miss detection using crowdsourced videos |
| topic | Near-miss detection crowdsource proactive approach convolutional neural network (CNN) vision transformer LSTM |
| url | https://ieeexplore.ieee.org/document/10820995/ |
| work_keys_str_mv | AT shadijaradat leveragingdeeplearningandmultimodallargelanguagemodelsfornearmissdetectionusingcrowdsourcedvideos AT mohammedelhenawy leveragingdeeplearningandmultimodallargelanguagemodelsfornearmissdetectionusingcrowdsourcedvideos AT huthaifaiashqar leveragingdeeplearningandmultimodallargelanguagemodelsfornearmissdetectionusingcrowdsourcedvideos AT alexanderpaz leveragingdeeplearningandmultimodallargelanguagemodelsfornearmissdetectionusingcrowdsourcedvideos AT richinayak leveragingdeeplearningandmultimodallargelanguagemodelsfornearmissdetectionusingcrowdsourcedvideos |