Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design
This paper presents an improved deep reinforcement learning-based (DRL) approach for end-to-end models using a Graph Neural Network(GNN). The proposed method aims to improve end-to-end deep Q learning with a GNN by decomposing the GNN-based Q-network structure into two sub-streams to separately esti...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10854435/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832087959485022208 |
|---|---|
| author | Tianchen Zhou Yutaka Yakuwa Natsuki Okamura Hiroyuki Hochigai Takayuki Kuroda Ikuko Eguchi Yairi |
| author_facet | Tianchen Zhou Yutaka Yakuwa Natsuki Okamura Hiroyuki Hochigai Takayuki Kuroda Ikuko Eguchi Yairi |
| author_sort | Tianchen Zhou |
| collection | DOAJ |
| description | This paper presents an improved deep reinforcement learning-based (DRL) approach for end-to-end models using a Graph Neural Network(GNN). The proposed method aims to improve end-to-end deep Q learning with a GNN by decomposing the GNN-based Q-network structure into two sub-streams to separately estimate the global state value and the state-dependent action advantage instead. By doing that, our dueling GNN architecture can independently learn which states are valuable or not. This is achieved by utilizing the graph-dependent global-state value rather than relying on the effect of each action for each state. This approach provides a more accurate approximation of the Q-value. With better Q-value approximation, the network can deal with the problem of massive state space with sparse rewards and significantly achieve higher learning efficiency without imposing any change to the underlying reinforcement learning algorithm. The proposed method was introduced into an automated ICT system design model. The automated ICT system design model faces a fundamental challenge characterized by prolonged learning times, primarily attributable to the tendency to overestimate particular configurations owing to the scarcity of rewards despite the vast exploration space encompassing numerous possible combinations of ICT system components. The results reveal that the proposed architecture effectively improves the learning efficiency of the DRL model without imposing any changes to the underlying reinforcement learning algorithm. |
| format | Article |
| id | doaj-art-329c7210afa4460bb849a2ef045a5fc3 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-329c7210afa4460bb849a2ef045a5fc32025-02-06T00:00:36ZengIEEEIEEE Access2169-35362025-01-0113218702187910.1109/ACCESS.2025.353424610854435Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System DesignTianchen Zhou0https://orcid.org/0009-0005-6769-3109Yutaka Yakuwa1https://orcid.org/0009-0001-4262-4239Natsuki Okamura2Hiroyuki Hochigai3https://orcid.org/0009-0001-8645-4470Takayuki Kuroda4Ikuko Eguchi Yairi5https://orcid.org/0000-0001-7522-0663Department of Information Science, Sophia University, Tokyo, JapanNEC Corporation, Kawasaki, JapanDepartment of Information Science, Sophia University, Tokyo, JapanDepartment of Information Science, Sophia University, Tokyo, JapanNEC Corporation, Kawasaki, JapanDepartment of Information Science, Sophia University, Tokyo, JapanThis paper presents an improved deep reinforcement learning-based (DRL) approach for end-to-end models using a Graph Neural Network(GNN). The proposed method aims to improve end-to-end deep Q learning with a GNN by decomposing the GNN-based Q-network structure into two sub-streams to separately estimate the global state value and the state-dependent action advantage instead. By doing that, our dueling GNN architecture can independently learn which states are valuable or not. This is achieved by utilizing the graph-dependent global-state value rather than relying on the effect of each action for each state. This approach provides a more accurate approximation of the Q-value. With better Q-value approximation, the network can deal with the problem of massive state space with sparse rewards and significantly achieve higher learning efficiency without imposing any change to the underlying reinforcement learning algorithm. The proposed method was introduced into an automated ICT system design model. The automated ICT system design model faces a fundamental challenge characterized by prolonged learning times, primarily attributable to the tendency to overestimate particular configurations owing to the scarcity of rewards despite the vast exploration space encompassing numerous possible combinations of ICT system components. The results reveal that the proposed architecture effectively improves the learning efficiency of the DRL model without imposing any changes to the underlying reinforcement learning algorithm.https://ieeexplore.ieee.org/document/10854435/Machine learningdeep reinforcement learningsystem designdesign automation |
| spellingShingle | Tianchen Zhou Yutaka Yakuwa Natsuki Okamura Hiroyuki Hochigai Takayuki Kuroda Ikuko Eguchi Yairi Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design IEEE Access Machine learning deep reinforcement learning system design design automation |
| title | Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design |
| title_full | Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design |
| title_fullStr | Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design |
| title_full_unstemmed | Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design |
| title_short | Dueling Network Architecture for GNN in the Deep Reinforcement Learning for the Automated ICT System Design |
| title_sort | dueling network architecture for gnn in the deep reinforcement learning for the automated ict system design |
| topic | Machine learning deep reinforcement learning system design design automation |
| url | https://ieeexplore.ieee.org/document/10854435/ |
| work_keys_str_mv | AT tianchenzhou duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign AT yutakayakuwa duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign AT natsukiokamura duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign AT hiroyukihochigai duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign AT takayukikuroda duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign AT ikukoeguchiyairi duelingnetworkarchitectureforgnninthedeepreinforcementlearningfortheautomatedictsystemdesign |