MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network

While deep learning techniques have been extensively employed in malware detection, there is a notable challenge in effectively embedding malware features. Current neural network methods primarily capture superficial characteristics, lacking in-depth semantic exploration of functions and failing to...

Full description

Saved in:
Bibliographic Details
Main Authors: Wenjie Guo, Wenbiao Du, Xiuqi Yang, Jingfeng Xue, Yong Wang, Weijie Han, Jingjing Hu
Format: Article
Language:English
Published: MDPI AG 2025-01-01
Series:Sensors
Subjects:
Online Access:https://www.mdpi.com/1424-8220/25/2/374
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832587475321618432
author Wenjie Guo
Wenbiao Du
Xiuqi Yang
Jingfeng Xue
Yong Wang
Weijie Han
Jingjing Hu
author_facet Wenjie Guo
Wenbiao Du
Xiuqi Yang
Jingfeng Xue
Yong Wang
Weijie Han
Jingjing Hu
author_sort Wenjie Guo
collection DOAJ
description While deep learning techniques have been extensively employed in malware detection, there is a notable challenge in effectively embedding malware features. Current neural network methods primarily capture superficial characteristics, lacking in-depth semantic exploration of functions and failing to preserve structural information at the file level. Motivated by the aforementioned challenges, this paper introduces MalHAPGNN, a novel framework for malware detection that leverages a hierarchical attention pooling graph neural network based on enhanced call graphs. Firstly, to ensure semantic richness, a Bidirectional Encoder Representations from Transformers-based (BERT) attribute-enhanced function embedding method is proposed for the extraction of node attributes in the function call graph. Subsequently, this work designs a hierarchical graph neural network that integrates attention mechanisms and pooling operations, complemented by function node sampling and structural learning strategies. This framework delivers a comprehensive profile of malicious code across semantic, syntactic, and structural dimensions. Extensive experiments conducted on the Kaggle and VirusShare datasets have demonstrated that the proposed framework outperforms other graph neural network (GNN)-based malware detection methods.
format Article
id doaj-art-08ceed74ba984ba282bc6844ad58cce1
institution Kabale University
issn 1424-8220
language English
publishDate 2025-01-01
publisher MDPI AG
record_format Article
series Sensors
spelling doaj-art-08ceed74ba984ba282bc6844ad58cce12025-01-24T13:48:41ZengMDPI AGSensors1424-82202025-01-0125237410.3390/s25020374MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural NetworkWenjie Guo0Wenbiao Du1Xiuqi Yang2Jingfeng Xue3Yong Wang4Weijie Han5Jingjing Hu6School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaSchool of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaSchool of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaSchool of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaSchool of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaSchool of Space Information, Space Engineering University, Beijing 100084, ChinaSchool of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, ChinaWhile deep learning techniques have been extensively employed in malware detection, there is a notable challenge in effectively embedding malware features. Current neural network methods primarily capture superficial characteristics, lacking in-depth semantic exploration of functions and failing to preserve structural information at the file level. Motivated by the aforementioned challenges, this paper introduces MalHAPGNN, a novel framework for malware detection that leverages a hierarchical attention pooling graph neural network based on enhanced call graphs. Firstly, to ensure semantic richness, a Bidirectional Encoder Representations from Transformers-based (BERT) attribute-enhanced function embedding method is proposed for the extraction of node attributes in the function call graph. Subsequently, this work designs a hierarchical graph neural network that integrates attention mechanisms and pooling operations, complemented by function node sampling and structural learning strategies. This framework delivers a comprehensive profile of malicious code across semantic, syntactic, and structural dimensions. Extensive experiments conducted on the Kaggle and VirusShare datasets have demonstrated that the proposed framework outperforms other graph neural network (GNN)-based malware detection methods.https://www.mdpi.com/1424-8220/25/2/374malware detectionmalware embeddinggraph neural networkrepresentation learninggraph pooling mechanism
spellingShingle Wenjie Guo
Wenbiao Du
Xiuqi Yang
Jingfeng Xue
Yong Wang
Weijie Han
Jingjing Hu
MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
Sensors
malware detection
malware embedding
graph neural network
representation learning
graph pooling mechanism
title MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
title_full MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
title_fullStr MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
title_full_unstemmed MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
title_short MalHAPGNN: An Enhanced Call Graph-Based Malware Detection Framework Using Hierarchical Attention Pooling Graph Neural Network
title_sort malhapgnn an enhanced call graph based malware detection framework using hierarchical attention pooling graph neural network
topic malware detection
malware embedding
graph neural network
representation learning
graph pooling mechanism
url https://www.mdpi.com/1424-8220/25/2/374
work_keys_str_mv AT wenjieguo malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT wenbiaodu malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT xiuqiyang malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT jingfengxue malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT yongwang malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT weijiehan malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork
AT jingjinghu malhapgnnanenhancedcallgraphbasedmalwaredetectionframeworkusinghierarchicalattentionpoolinggraphneuralnetwork