Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images
Psychological studies highlight the importance of combining new knowledge with one’s prior experience. Hence personalization for a learner plays a key role for vocabulary acquisition. However, this faces two challenges: probing a learner’s experiences in their lives and craftin...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10851279/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832575627389042688 |
|---|---|
| author | Kanta Yamaoka Ko Watanabe Koichi Kise Andreas Dengel Shoya Ishimaru |
| author_facet | Kanta Yamaoka Ko Watanabe Koichi Kise Andreas Dengel Shoya Ishimaru |
| author_sort | Kanta Yamaoka |
| collection | DOAJ |
| description | Psychological studies highlight the importance of combining new knowledge with one’s prior experience. Hence personalization for a learner plays a key role for vocabulary acquisition. However, this faces two challenges: probing a learner’s experiences in their lives and crafting tailored material for every different individual. With the prevalence of visual social media, such as Instagram, people share their photos from favorite moments, providing rich contexts, and emerging generative AI would create learning material in an effortless fashion. We prototyped an online vocabulary exploration system, which displays a learner’s selected photos from their Instagram along with a generated sentence using image recognition and a language model, GPT-3. The system lets a learner find new words that are strongly tied to their daily life with the approximated context. We carried out our within-subject design evaluation of the system with 23 participants with three conditions: contexts grounded with learner’s Instagram photos, contexts grounded from general images, and text-only modality. From learners’ recall task accuracy, we found that having a context grounded with a learner’s social image allowed them to find difficult words to quickly learn than having context generated by someone’s image, or text only modality—although this finding is statistically insignificant. The Zipf frequency comparisons revealed that generally having image-based context allowed learners to extract difficult vocabulary than having text-only context. We also discuss quantitative and qualitative results regarding participants’ acceptance of the personalization system using their personal photos from social media. Generally, they reported positive impressions for our system such as high engagement. While our system prioritizes user privacy with opt-in data control and secure design, we explore additional ethical considerations. This paves the way for a future where personalized language learning, grounded in real-world experiences and generative AI, benefits learners. |
| format | Article |
| id | doaj-art-f466ba84e76d4d708fe1e41b4b2cb054 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-f466ba84e76d4d708fe1e41b4b2cb0542025-01-31T23:05:06ZengIEEEIEEE Access2169-35362025-01-0113204562047110.1109/ACCESS.2025.353307610851279Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media ImagesKanta Yamaoka0https://orcid.org/0000-0002-0141-7858Ko Watanabe1https://orcid.org/0000-0003-0252-1785Koichi Kise2https://orcid.org/0000-0001-5779-6968Andreas Dengel3https://orcid.org/0000-0002-6100-8255Shoya Ishimaru4https://orcid.org/0000-0002-5374-1510College of Engineering, Osaka Prefecture University, Osaka, JapanDepartment of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, GermanyGraduate School of Informatics, Osaka Metropolitan University, Osaka, JapanDepartment of Computer Science, RPTU Kaiserslautern-Landau, Kaiserslautern, GermanyGraduate School of Informatics, Osaka Metropolitan University, Osaka, JapanPsychological studies highlight the importance of combining new knowledge with one’s prior experience. Hence personalization for a learner plays a key role for vocabulary acquisition. However, this faces two challenges: probing a learner’s experiences in their lives and crafting tailored material for every different individual. With the prevalence of visual social media, such as Instagram, people share their photos from favorite moments, providing rich contexts, and emerging generative AI would create learning material in an effortless fashion. We prototyped an online vocabulary exploration system, which displays a learner’s selected photos from their Instagram along with a generated sentence using image recognition and a language model, GPT-3. The system lets a learner find new words that are strongly tied to their daily life with the approximated context. We carried out our within-subject design evaluation of the system with 23 participants with three conditions: contexts grounded with learner’s Instagram photos, contexts grounded from general images, and text-only modality. From learners’ recall task accuracy, we found that having a context grounded with a learner’s social image allowed them to find difficult words to quickly learn than having context generated by someone’s image, or text only modality—although this finding is statistically insignificant. The Zipf frequency comparisons revealed that generally having image-based context allowed learners to extract difficult vocabulary than having text-only context. We also discuss quantitative and qualitative results regarding participants’ acceptance of the personalization system using their personal photos from social media. Generally, they reported positive impressions for our system such as high engagement. While our system prioritizes user privacy with opt-in data control and secure design, we explore additional ethical considerations. This paves the way for a future where personalized language learning, grounded in real-world experiences and generative AI, benefits learners.https://ieeexplore.ieee.org/document/10851279/Context-aware language learningHCIlarge language modelsgenerative AI |
| spellingShingle | Kanta Yamaoka Ko Watanabe Koichi Kise Andreas Dengel Shoya Ishimaru Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images IEEE Access Context-aware language learning HCI large language models generative AI |
| title | Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images |
| title_full | Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images |
| title_fullStr | Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images |
| title_full_unstemmed | Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images |
| title_short | Img2Vocab: Explore Words Tied to Your Life With LLMs and Social Media Images |
| title_sort | img2vocab explore words tied to your life with llms and social media images |
| topic | Context-aware language learning HCI large language models generative AI |
| url | https://ieeexplore.ieee.org/document/10851279/ |
| work_keys_str_mv | AT kantayamaoka img2vocabexplorewordstiedtoyourlifewithllmsandsocialmediaimages AT kowatanabe img2vocabexplorewordstiedtoyourlifewithllmsandsocialmediaimages AT koichikise img2vocabexplorewordstiedtoyourlifewithllmsandsocialmediaimages AT andreasdengel img2vocabexplorewordstiedtoyourlifewithllmsandsocialmediaimages AT shoyaishimaru img2vocabexplorewordstiedtoyourlifewithllmsandsocialmediaimages |