MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform
To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10843684/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576747842830336 |
|---|---|
| author | Ryutaro Kurai Takefumi Hiraki Yuichi Hiroi Yutaro Hirao Monica Perusquia-Hernandez Hideaki Uchiyama Kiyoshi Kiyokawa |
| author_facet | Ryutaro Kurai Takefumi Hiraki Yuichi Hiroi Yutaro Hirao Monica Perusquia-Hernandez Hideaki Uchiyama Kiyoshi Kiyokawa |
| author_sort | Ryutaro Kurai |
| collection | DOAJ |
| description | To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms. |
| format | Article |
| id | doaj-art-38b68bb1f87347119dcb19cb91fc7b32 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-38b68bb1f87347119dcb19cb91fc7b322025-01-31T00:01:34ZengIEEEIEEE Access2169-35362025-01-0113191321914310.1109/ACCESS.2025.353043910843684MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse PlatformRyutaro Kurai0Takefumi Hiraki1https://orcid.org/0000-0002-5767-3607Yuichi Hiroi2https://orcid.org/0000-0001-8567-6947Yutaro Hirao3Monica Perusquia-Hernandez4Hideaki Uchiyama5https://orcid.org/0000-0002-6119-1184Kiyoshi Kiyokawa6https://orcid.org/0000-0003-2260-1707Cluster, Inc., Shinagawa, Tokyo, JapanCluster Metaverse Lab, Shingawa, Tokyo, JapanCluster Metaverse Lab, Shingawa, Tokyo, JapanGraduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara, JapanGraduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara, JapanGraduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara, JapanGraduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Nara, JapanTo create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.https://ieeexplore.ieee.org/document/10843684/Large-language modellow-code programmingmetaverse platformvirtual reality |
| spellingShingle | Ryutaro Kurai Takefumi Hiraki Yuichi Hiroi Yutaro Hirao Monica Perusquia-Hernandez Hideaki Uchiyama Kiyoshi Kiyokawa MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform IEEE Access Large-language model low-code programming metaverse platform virtual reality |
| title | MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform |
| title_full | MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform |
| title_fullStr | MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform |
| title_full_unstemmed | MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform |
| title_short | MagicItem: Dynamic Behavior Design of Virtual Objects With Large Language Models in a Commercial Metaverse Platform |
| title_sort | magicitem dynamic behavior design of virtual objects with large language models in a commercial metaverse platform |
| topic | Large-language model low-code programming metaverse platform virtual reality |
| url | https://ieeexplore.ieee.org/document/10843684/ |
| work_keys_str_mv | AT ryutarokurai magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT takefumihiraki magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT yuichihiroi magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT yutarohirao magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT monicaperusquiahernandez magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT hideakiuchiyama magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform AT kiyoshikiyokawa magicitemdynamicbehaviordesignofvirtualobjectswithlargelanguagemodelsinacommercialmetaverseplatform |