From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model
The role of image data in knowledge extraction and representation has become increasingly significant. This study introduces a novel methodology, termed Image to Graph via Large Language Model (ImgGraph-LLM), which constructs a knowledge graph for each image in a dataset. Unlike existing methods tha...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/16/5/335 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850257782981263360 |
|---|---|
| author | Lei Chen Zhenyu Chen Wei Yang Shi Liu Yong Li |
| author_facet | Lei Chen Zhenyu Chen Wei Yang Shi Liu Yong Li |
| author_sort | Lei Chen |
| collection | DOAJ |
| description | The role of image data in knowledge extraction and representation has become increasingly significant. This study introduces a novel methodology, termed Image to Graph via Large Language Model (ImgGraph-LLM), which constructs a knowledge graph for each image in a dataset. Unlike existing methods that rely on text descriptions or multimodal data to build a comprehensive knowledge graph, our approach focuses solely on unlabeled individual image data, representing a distinct form of unsupervised knowledge graph construction. To tackle the challenge of generating a knowledge graph from individual images in an unsupervised manner, we first design two self-supervised operations to generate training data from unlabeled images. We then propose an iterative fine-tuning process that uses this self-supervised information, enabling the fine-tuned LLM to recognize the triplets needed to construct the knowledge graph. To improve the accuracy of triplet extraction, we introduce filtering strategies that effectively remove low-confidence training data. Finally, experiments on two large-scale real-world datasets demonstrate the superiority of our proposed model. |
| format | Article |
| id | doaj-art-e9dff79d70a842eebc3c20994c6560d1 |
| institution | OA Journals |
| issn | 2078-2489 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj-art-e9dff79d70a842eebc3c20994c6560d12025-08-20T01:56:19ZengMDPI AGInformation2078-24892025-04-0116533510.3390/info16050335From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language ModelLei Chen0Zhenyu Chen1Wei Yang2Shi Liu3Yong Li4Department of Electrical Engineering, Tsinghua University, Beijing 100084, ChinaBig Data Center, State Grid Corporation of China, Beijing 100052, ChinaBig Data Center, State Grid Corporation of China, Beijing 100052, ChinaBig Data Center, State Grid Corporation of China, Beijing 100052, ChinaDepartment of Electrical Engineering, Tsinghua University, Beijing 100084, ChinaThe role of image data in knowledge extraction and representation has become increasingly significant. This study introduces a novel methodology, termed Image to Graph via Large Language Model (ImgGraph-LLM), which constructs a knowledge graph for each image in a dataset. Unlike existing methods that rely on text descriptions or multimodal data to build a comprehensive knowledge graph, our approach focuses solely on unlabeled individual image data, representing a distinct form of unsupervised knowledge graph construction. To tackle the challenge of generating a knowledge graph from individual images in an unsupervised manner, we first design two self-supervised operations to generate training data from unlabeled images. We then propose an iterative fine-tuning process that uses this self-supervised information, enabling the fine-tuned LLM to recognize the triplets needed to construct the knowledge graph. To improve the accuracy of triplet extraction, we introduce filtering strategies that effectively remove low-confidence training data. Finally, experiments on two large-scale real-world datasets demonstrate the superiority of our proposed model.https://www.mdpi.com/2078-2489/16/5/335knowledge graphunsupervised learninglarge language model |
| spellingShingle | Lei Chen Zhenyu Chen Wei Yang Shi Liu Yong Li From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model Information knowledge graph unsupervised learning large language model |
| title | From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model |
| title_full | From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model |
| title_fullStr | From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model |
| title_full_unstemmed | From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model |
| title_short | From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model |
| title_sort | from pixels to insights unsupervised knowledge graph generation with large language model |
| topic | knowledge graph unsupervised learning large language model |
| url | https://www.mdpi.com/2078-2489/16/5/335 |
| work_keys_str_mv | AT leichen frompixelstoinsightsunsupervisedknowledgegraphgenerationwithlargelanguagemodel AT zhenyuchen frompixelstoinsightsunsupervisedknowledgegraphgenerationwithlargelanguagemodel AT weiyang frompixelstoinsightsunsupervisedknowledgegraphgenerationwithlargelanguagemodel AT shiliu frompixelstoinsightsunsupervisedknowledgegraphgenerationwithlargelanguagemodel AT yongli frompixelstoinsightsunsupervisedknowledgegraphgenerationwithlargelanguagemodel |