Communication System Using Natural Language for Robotic Laparoscope Guidance
To help with the critical nurse staffing shortages in hospitals worldwide, robotic assistants are designed to handle frequently required tasks in the digital operating room (DOR), such as the guidance of the laparoscopic camera. To enable fluent collaboration between robots and clinicians, an intuit...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
De Gruyter
2024-09-01
|
| Series: | Current Directions in Biomedical Engineering |
| Subjects: | |
| Online Access: | https://doi.org/10.1515/cdbme-2024-1066 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832570465496858624 |
|---|---|
| author | Pan Yan Bernhard Lukas Fan Cheng Beckendorf Lukas Wilhelm Dirk Feußner Hubertus Groh Georg |
| author_facet | Pan Yan Bernhard Lukas Fan Cheng Beckendorf Lukas Wilhelm Dirk Feußner Hubertus Groh Georg |
| author_sort | Pan Yan |
| collection | DOAJ |
| description | To help with the critical nurse staffing shortages in hospitals worldwide, robotic assistants are designed to handle frequently required tasks in the digital operating room (DOR), such as the guidance of the laparoscopic camera. To enable fluent collaboration between robots and clinicians, an intuitive and efficient communication interface is needed to allow for interaction using natural language. However, the demanding requirements of the surgical domain make it challenging to develop suitable solutions. A variety of different vocabulary or phrases may be used for expressing the same command. At the same time, surgical workflows may be highly dynamic - especially in emergency situations - and thus the system must be able to grasp the user’s intent both quickly and with high accuracy. This is especially true as only some clinicians may be authorized to request certain tasks, depending on their rank or field of expertise. To solve these challenges, our proposed communication system uses the fine-tuned deep learning model to recognize the speaker information, and the robot assistant takes action only when it detects the commands from the responsible clinician. Also, our proposed conversational functions enable the finetuned large language models to understand the current natural language command given previous command history. In this work, we present a communication system to recognize the speaking person and understand the intent of conversational commands quickly and accurately. |
| format | Article |
| id | doaj-art-e18d4b009617478f9efe55afb93e9542 |
| institution | Kabale University |
| issn | 2364-5504 |
| language | English |
| publishDate | 2024-09-01 |
| publisher | De Gruyter |
| record_format | Article |
| series | Current Directions in Biomedical Engineering |
| spelling | doaj-art-e18d4b009617478f9efe55afb93e95422025-02-02T15:45:00ZengDe GruyterCurrent Directions in Biomedical Engineering2364-55042024-09-01102545710.1515/cdbme-2024-1066Communication System Using Natural Language for Robotic Laparoscope GuidancePan Yan0Bernhard Lukas1Fan Cheng2Beckendorf Lukas3Wilhelm Dirk4Feußner Hubertus5Groh Georg6Social computing group, Technical University Munich,Munich, GermanyResearch Group MITI, Klinikum rechts der Isar, Technical University of Munich,Munich, GermanySocial computing group, Technical University Munich,Munich, GermanyResearch Group MITI, Klinikum rechts der Isar, Technical University of Munich,Munich, GermanyResearch Group MITI, Klinikum rechts der Isar, Technical University of Munich,Munich, GermanyResearch Group MITI, Klinikum rechts der Isar, Technical University of Munich,Munich, GermanySocial computing group, Technical University Munich,Munich, GermanyTo help with the critical nurse staffing shortages in hospitals worldwide, robotic assistants are designed to handle frequently required tasks in the digital operating room (DOR), such as the guidance of the laparoscopic camera. To enable fluent collaboration between robots and clinicians, an intuitive and efficient communication interface is needed to allow for interaction using natural language. However, the demanding requirements of the surgical domain make it challenging to develop suitable solutions. A variety of different vocabulary or phrases may be used for expressing the same command. At the same time, surgical workflows may be highly dynamic - especially in emergency situations - and thus the system must be able to grasp the user’s intent both quickly and with high accuracy. This is especially true as only some clinicians may be authorized to request certain tasks, depending on their rank or field of expertise. To solve these challenges, our proposed communication system uses the fine-tuned deep learning model to recognize the speaker information, and the robot assistant takes action only when it detects the commands from the responsible clinician. Also, our proposed conversational functions enable the finetuned large language models to understand the current natural language command given previous command history. In this work, we present a communication system to recognize the speaking person and understand the intent of conversational commands quickly and accurately.https://doi.org/10.1515/cdbme-2024-1066communication systemnatural language commandlarge language modelspeaker recognizer |
| spellingShingle | Pan Yan Bernhard Lukas Fan Cheng Beckendorf Lukas Wilhelm Dirk Feußner Hubertus Groh Georg Communication System Using Natural Language for Robotic Laparoscope Guidance Current Directions in Biomedical Engineering communication system natural language command large language model speaker recognizer |
| title | Communication System Using Natural Language for Robotic Laparoscope Guidance |
| title_full | Communication System Using Natural Language for Robotic Laparoscope Guidance |
| title_fullStr | Communication System Using Natural Language for Robotic Laparoscope Guidance |
| title_full_unstemmed | Communication System Using Natural Language for Robotic Laparoscope Guidance |
| title_short | Communication System Using Natural Language for Robotic Laparoscope Guidance |
| title_sort | communication system using natural language for robotic laparoscope guidance |
| topic | communication system natural language command large language model speaker recognizer |
| url | https://doi.org/10.1515/cdbme-2024-1066 |
| work_keys_str_mv | AT panyan communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT bernhardlukas communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT fancheng communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT beckendorflukas communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT wilhelmdirk communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT feußnerhubertus communicationsystemusingnaturallanguageforroboticlaparoscopeguidance AT grohgeorg communicationsystemusingnaturallanguageforroboticlaparoscopeguidance |