Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward
Robot control using reinforcement learning has become popular, but its learning process often terminates midway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termi...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-03-01
|
| Series: | Results in Control and Optimization |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666720725000165 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832583860505804800 |
|---|---|
| author | Taisuke Kobayashi |
| author_facet | Taisuke Kobayashi |
| author_sort | Taisuke Kobayashi |
| collection | DOAJ |
| description | Robot control using reinforcement learning has become popular, but its learning process often terminates midway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentional implicit underestimation or overestimation occurs, depending on the reward design in the normal states. If the termination by failure is highly valued with the unintentional overestimation, the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in the practical use of TD learning to review the exception handling at termination. Therefore, this paper proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. This intentional underestimation is heuristically derived with the assumption of two-step transition to absorbing state. In addition, the degree of underestimation is adjusted according to the degree of steadiness at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulation results showed that the proposed method improves the success rate for 24 tasks with different reward designs from 10/24 in the conventional method to 20/24. Real-robot experiments also demonstrated that the proposed method enables to learn the optimal policy even in the case that the conventional method fails. |
| format | Article |
| id | doaj-art-a5368f809bb840d998b46323f0c04882 |
| institution | Kabale University |
| issn | 2666-7207 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Results in Control and Optimization |
| spelling | doaj-art-a5368f809bb840d998b46323f0c048822025-01-28T04:14:57ZengElsevierResults in Control and Optimization2666-72072025-03-0118100530Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed rewardTaisuke Kobayashi0Corresponding author.; National Institute of Informatics (NII) and The Graduate University for Advanced Studies (SOKENDAI), Tokyo, JapanRobot control using reinforcement learning has become popular, but its learning process often terminates midway through an episode for safety and time-saving reasons. This study addresses the problem of the most popular exception handling that temporal-difference (TD) learning performs at such termination. That is, by forcibly assuming zero value after termination, unintentional implicit underestimation or overestimation occurs, depending on the reward design in the normal states. If the termination by failure is highly valued with the unintentional overestimation, the wrong policy may be acquired. Although this problem can be avoided by paying attention to the reward design, it is essential in the practical use of TD learning to review the exception handling at termination. Therefore, this paper proposes a method to intentionally underestimate the value after termination to avoid learning failures due to the unintentional overestimation. This intentional underestimation is heuristically derived with the assumption of two-step transition to absorbing state. In addition, the degree of underestimation is adjusted according to the degree of steadiness at termination, thereby preventing excessive exploration due to the intentional underestimation. Simulation results showed that the proposed method improves the success rate for 24 tasks with different reward designs from 10/24 in the conventional method to 20/24. Real-robot experiments also demonstrated that the proposed method enables to learn the optimal policy even in the case that the conventional method fails.http://www.sciencedirect.com/science/article/pii/S2666720725000165Temporal-difference learningArbitrariness of reward designException handling at episode terminationIntentional underestimation of terminal value |
| spellingShingle | Taisuke Kobayashi Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward Results in Control and Optimization Temporal-difference learning Arbitrariness of reward design Exception handling at episode termination Intentional underestimation of terminal value |
| title | Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward |
| title_full | Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward |
| title_fullStr | Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward |
| title_full_unstemmed | Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward |
| title_short | Intentionally-underestimated value function at terminal state for temporal-difference learning with mis-designed reward |
| title_sort | intentionally underestimated value function at terminal state for temporal difference learning with mis designed reward |
| topic | Temporal-difference learning Arbitrariness of reward design Exception handling at episode termination Intentional underestimation of terminal value |
| url | http://www.sciencedirect.com/science/article/pii/S2666720725000165 |
| work_keys_str_mv | AT taisukekobayashi intentionallyunderestimatedvaluefunctionatterminalstatefortemporaldifferencelearningwithmisdesignedreward |