Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control
A number of deep reinforcement-learning (RL) approaches propose to control traffic signals. Compared to traditional approaches, RL approaches can learn from higher-dimensionality input road and vehicle sensors and better adapt to varying traffic conditions resulting in reduced travel times (in simul...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Open Journal of Intelligent Transportation Systems |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10315958/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832590347404836864 |
|---|---|
| author | Tianyu Shi Francois-Xavier Devailly Denis Larocque Laurent Charlin |
| author_facet | Tianyu Shi Francois-Xavier Devailly Denis Larocque Laurent Charlin |
| author_sort | Tianyu Shi |
| collection | DOAJ |
| description | A number of deep reinforcement-learning (RL) approaches propose to control traffic signals. Compared to traditional approaches, RL approaches can learn from higher-dimensionality input road and vehicle sensors and better adapt to varying traffic conditions resulting in reduced travel times (in simulation). However, these RL methods require training from massive traffic sensor data. To offset this relative inefficiency, some recent RL methods have the ability to first learn from small-scale networks and then generalize to unseen city-scale networks without additional retraining (zero-shot transfer). In this work, we study the robustness of such methods along two axes. First, sensor failures and GPS occlusions create missing-data challenges and we show that recent methods remain brittle in the face of these missing data. Second, we provide a more systematic study of the generalization ability of RL methods to new networks with different traffic regimes. Again, we identify the limitations of recent approaches. We then propose using a combination of distributional and vanilla reinforcement learning through a policy ensemble. Building upon the state-of-the-art previous model which uses a decentralized approach for large-scale traffic signal control with graph convolutional networks (GCNs), we first learn models using a distributional reinforcement learning (DisRL) approach. In particular, we use implicit quantile networks (IQN) to model the state-action return distribution with quantile regression. For traffic signal control problems, an ensemble of standard RL and DisRL yields superior performance across different scenarios, including different levels of missing sensor data and traffic flow patterns. Furthermore, the learning scheme of the resulting model can improve zero-shot transferability to different road network structures, including both synthetic networks and real-world networks (e.g., Luxembourg, Manhattan). We conduct extensive experiments to compare our approach to multi-agent reinforcement learning and traditional transportation approaches. Results show that the proposed method improves robustness and generalizability in the face of missing data, varying road networks, and traffic flows. |
| format | Article |
| id | doaj-art-9c2c683dbf4a4cc98f2824e41e3b8b20 |
| institution | Kabale University |
| issn | 2687-7813 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Intelligent Transportation Systems |
| spelling | doaj-art-9c2c683dbf4a4cc98f2824e41e3b8b202025-01-24T00:02:38ZengIEEEIEEE Open Journal of Intelligent Transportation Systems2687-78132024-01-01521510.1109/OJITS.2023.333168910315958Improving the Generalizability and Robustness of Large-Scale Traffic Signal ControlTianyu Shi0https://orcid.org/0000-0003-4271-0871Francois-Xavier Devailly1https://orcid.org/0000-0002-5861-0675Denis Larocque2https://orcid.org/0000-0002-7372-7943Laurent Charlin3https://orcid.org/0000-0002-6545-9459Department of Civil Engineering, University of Toronto, Toronto, CanadaDepartment of Decision Sciences, HEC Montreal, Montreal, CanadaDepartment of Decision Sciences, HEC Montreal, Montreal, CanadaDepartment of Decision Sciences, HEC Montreal, Montreal, CanadaA number of deep reinforcement-learning (RL) approaches propose to control traffic signals. Compared to traditional approaches, RL approaches can learn from higher-dimensionality input road and vehicle sensors and better adapt to varying traffic conditions resulting in reduced travel times (in simulation). However, these RL methods require training from massive traffic sensor data. To offset this relative inefficiency, some recent RL methods have the ability to first learn from small-scale networks and then generalize to unseen city-scale networks without additional retraining (zero-shot transfer). In this work, we study the robustness of such methods along two axes. First, sensor failures and GPS occlusions create missing-data challenges and we show that recent methods remain brittle in the face of these missing data. Second, we provide a more systematic study of the generalization ability of RL methods to new networks with different traffic regimes. Again, we identify the limitations of recent approaches. We then propose using a combination of distributional and vanilla reinforcement learning through a policy ensemble. Building upon the state-of-the-art previous model which uses a decentralized approach for large-scale traffic signal control with graph convolutional networks (GCNs), we first learn models using a distributional reinforcement learning (DisRL) approach. In particular, we use implicit quantile networks (IQN) to model the state-action return distribution with quantile regression. For traffic signal control problems, an ensemble of standard RL and DisRL yields superior performance across different scenarios, including different levels of missing sensor data and traffic flow patterns. Furthermore, the learning scheme of the resulting model can improve zero-shot transferability to different road network structures, including both synthetic networks and real-world networks (e.g., Luxembourg, Manhattan). We conduct extensive experiments to compare our approach to multi-agent reinforcement learning and traditional transportation approaches. Results show that the proposed method improves robustness and generalizability in the face of missing data, varying road networks, and traffic flows.https://ieeexplore.ieee.org/document/10315958/Distributional reinforcement learninggraph neural networkspolicy ensemblerobustnessgeneralizabilitytraffic signal control |
| spellingShingle | Tianyu Shi Francois-Xavier Devailly Denis Larocque Laurent Charlin Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control IEEE Open Journal of Intelligent Transportation Systems Distributional reinforcement learning graph neural networks policy ensemble robustness generalizability traffic signal control |
| title | Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control |
| title_full | Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control |
| title_fullStr | Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control |
| title_full_unstemmed | Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control |
| title_short | Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control |
| title_sort | improving the generalizability and robustness of large scale traffic signal control |
| topic | Distributional reinforcement learning graph neural networks policy ensemble robustness generalizability traffic signal control |
| url | https://ieeexplore.ieee.org/document/10315958/ |
| work_keys_str_mv | AT tianyushi improvingthegeneralizabilityandrobustnessoflargescaletrafficsignalcontrol AT francoisxavierdevailly improvingthegeneralizabilityandrobustnessoflargescaletrafficsignalcontrol AT denislarocque improvingthegeneralizabilityandrobustnessoflargescaletrafficsignalcontrol AT laurentcharlin improvingthegeneralizabilityandrobustnessoflargescaletrafficsignalcontrol |