MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery
This paper introduces a novel approach for learning natural language descriptions of scenery in Minecraft. We apply techniques from Computer Vision and Natural Language Processing to create an AI framework called MineObserver for assessing the accuracy of learner-generated descriptions of science-re...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2022-05-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Subjects: | |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/130729 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850271222326099968 |
|---|---|
| author | Jay Mahajan Samuel Hum Jeff Ginger H. Chad Lane |
| author_facet | Jay Mahajan Samuel Hum Jeff Ginger H. Chad Lane |
| author_sort | Jay Mahajan |
| collection | DOAJ |
| description | This paper introduces a novel approach for learning natural language descriptions of scenery in Minecraft. We apply techniques from Computer Vision and Natural Language Processing to create an AI framework called MineObserver for assessing the accuracy of learner-generated descriptions of science-related images. The ultimate purpose of the system is to automatically assess the accuracy of learner observations, written in natural language, made during science learning activities that take place in Minecraft. Eventually, MineObserver will be used as part of a pedagogical agent framework for providing in-game support for learning. Preliminary results are mixed, but promising with approximately 62% of images in our test set being properly classified by our image captioning approach. Broadly, our work suggests that computer vision techniques work as expected in Minecraft and can serve as a basis for assessing learner observations. |
| format | Article |
| id | doaj-art-e37bbf25787f4e198d8c316c6fc3fdf3 |
| institution | OA Journals |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2022-05-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-e37bbf25787f4e198d8c316c6fc3fdf32025-08-20T01:52:18ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622022-05-013510.32473/flairs.v35i.13072966928MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft ImageryJay Mahajan0Samuel Hum1Jeff Ginger2H. Chad Lane3University of Illinois at Urbana ChampaignUniversity of Illinois at Urbana ChampaignUniversity of Illinois at Urbana ChampaignUniversity of Illinois at Urbana ChampaignThis paper introduces a novel approach for learning natural language descriptions of scenery in Minecraft. We apply techniques from Computer Vision and Natural Language Processing to create an AI framework called MineObserver for assessing the accuracy of learner-generated descriptions of science-related images. The ultimate purpose of the system is to automatically assess the accuracy of learner observations, written in natural language, made during science learning activities that take place in Minecraft. Eventually, MineObserver will be used as part of a pedagogical agent framework for providing in-game support for learning. Preliminary results are mixed, but promising with approximately 62% of images in our test set being properly classified by our image captioning approach. Broadly, our work suggests that computer vision techniques work as expected in Minecraft and can serve as a basis for assessing learner observations.https://journals.flvc.org/FLAIRS/article/view/130729computer visionnatural language processingpedagogical agent |
| spellingShingle | Jay Mahajan Samuel Hum Jeff Ginger H. Chad Lane MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery Proceedings of the International Florida Artificial Intelligence Research Society Conference computer vision natural language processing pedagogical agent |
| title | MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery |
| title_full | MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery |
| title_fullStr | MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery |
| title_full_unstemmed | MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery |
| title_short | MineObserver: A Deep Learning Framework for Assessing Natural Language Descriptions of Minecraft Imagery |
| title_sort | mineobserver a deep learning framework for assessing natural language descriptions of minecraft imagery |
| topic | computer vision natural language processing pedagogical agent |
| url | https://journals.flvc.org/FLAIRS/article/view/130729 |
| work_keys_str_mv | AT jaymahajan mineobserveradeeplearningframeworkforassessingnaturallanguagedescriptionsofminecraftimagery AT samuelhum mineobserveradeeplearningframeworkforassessingnaturallanguagedescriptionsofminecraftimagery AT jeffginger mineobserveradeeplearningframeworkforassessingnaturallanguagedescriptionsofminecraftimagery AT hchadlane mineobserveradeeplearningframeworkforassessingnaturallanguagedescriptionsofminecraftimagery |