Digital chefs and intelligent cooking systems based on multimodal large language model
A digital chef and an intelligent cooking method were proposed to achieve high-quality, precise cooking results. In the offline phase, visual, auditory and thermal sensors record professional chefs' continuous cooking operations. The collected frame-by-frame images and multi-round Q&A t...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
POSTS&TELECOM PRESS Co., LTD
2024-12-01
|
| Series: | 智能科学与技术学报 |
| Subjects: | |
| Online Access: | http://www.cjist.com.cn/zh/article/doi/10.11959/j.issn.2096-6652.202448/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832586368948109312 |
|---|---|
| author | LI Xinyuan LI Bai SUN Yueshuo ZHANG Tantan TIAN Yonglin YIN Zhuyan WANG Fei-Yue |
| author_facet | LI Xinyuan LI Bai SUN Yueshuo ZHANG Tantan TIAN Yonglin YIN Zhuyan WANG Fei-Yue |
| author_sort | LI Xinyuan |
| collection | DOAJ |
| description | A digital chef and an intelligent cooking method were proposed to achieve high-quality, precise cooking results. In the offline phase, visual, auditory and thermal sensors record professional chefs' continuous cooking operations. The collected frame-by-frame images and multi-round Q&A texts form a culinary expert knowledge base. A low-rank adaptation method was applied to fine-tune a pretrained multimodal large language model, enabling it to understand cooking intentions. In the online phase, real-time sensory data were converted into image-text inputs for the fine-tuned model, which then generated cooking instructions to guide users through the cooking steps. A hardware-software cooking system was implemented and tested with a pan-frying steak task. Experimental results show that the fine-tuned system effectively controls the steak's doneness and quality, and significantly improves the accuracy and rationality of cooking instructions compared to the model before fine-tuning. |
| format | Article |
| id | doaj-art-c771741ac1014dc98fa4d946e92abbe3 |
| institution | Kabale University |
| issn | 2096-6652 |
| language | zho |
| publishDate | 2024-12-01 |
| publisher | POSTS&TELECOM PRESS Co., LTD |
| record_format | Article |
| series | 智能科学与技术学报 |
| spelling | doaj-art-c771741ac1014dc98fa4d946e92abbe32025-01-25T19:00:53ZzhoPOSTS&TELECOM PRESS Co., LTD智能科学与技术学报2096-66522024-12-01642944481046628Digital chefs and intelligent cooking systems based on multimodal large language modelLI XinyuanLI BaiSUN YueshuoZHANG TantanTIAN YonglinYIN ZhuyanWANG Fei-YueA digital chef and an intelligent cooking method were proposed to achieve high-quality, precise cooking results. In the offline phase, visual, auditory and thermal sensors record professional chefs' continuous cooking operations. The collected frame-by-frame images and multi-round Q&A texts form a culinary expert knowledge base. A low-rank adaptation method was applied to fine-tune a pretrained multimodal large language model, enabling it to understand cooking intentions. In the online phase, real-time sensory data were converted into image-text inputs for the fine-tuned model, which then generated cooking instructions to guide users through the cooking steps. A hardware-software cooking system was implemented and tested with a pan-frying steak task. Experimental results show that the fine-tuned system effectively controls the steak's doneness and quality, and significantly improves the accuracy and rationality of cooking instructions compared to the model before fine-tuning.http://www.cjist.com.cn/zh/article/doi/10.11959/j.issn.2096-6652.202448/multimodal large language modeldigital chefintelligent cookingcooking robotexpert systemartificial intelligence |
| spellingShingle | LI Xinyuan LI Bai SUN Yueshuo ZHANG Tantan TIAN Yonglin YIN Zhuyan WANG Fei-Yue Digital chefs and intelligent cooking systems based on multimodal large language model 智能科学与技术学报 multimodal large language model digital chef intelligent cooking cooking robot expert system artificial intelligence |
| title | Digital chefs and intelligent cooking systems based on multimodal large language model |
| title_full | Digital chefs and intelligent cooking systems based on multimodal large language model |
| title_fullStr | Digital chefs and intelligent cooking systems based on multimodal large language model |
| title_full_unstemmed | Digital chefs and intelligent cooking systems based on multimodal large language model |
| title_short | Digital chefs and intelligent cooking systems based on multimodal large language model |
| title_sort | digital chefs and intelligent cooking systems based on multimodal large language model |
| topic | multimodal large language model digital chef intelligent cooking cooking robot expert system artificial intelligence |
| url | http://www.cjist.com.cn/zh/article/doi/10.11959/j.issn.2096-6652.202448/ |
| work_keys_str_mv | AT lixinyuan digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT libai digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT sunyueshuo digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT zhangtantan digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT tianyonglin digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT yinzhuyan digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel AT wangfeiyue digitalchefsandintelligentcookingsystemsbasedonmultimodallargelanguagemodel |