AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique
The efforts in the field of virtualizing the guitar into well-modelled software systems have faced a lot of practical limitations. The existing guitar simulation programs require additional devices such as Electromyography (EMG) controllers, or Musical Instrument Digital Interface (MIDI)-based recor...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
ICI Publishing House
2024-12-01
|
| Series: | Revista Română de Informatică și Automatică |
| Subjects: | |
| Online Access: | https://rria.ici.ro/documents/1235/art._10_Beulah_Panda_Nair.pdf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832593947850964992 |
|---|---|
| author | Beulah ARUL Shashank PANDA Tushar NAIR |
| author_facet | Beulah ARUL Shashank PANDA Tushar NAIR |
| author_sort | Beulah ARUL |
| collection | DOAJ |
| description | The efforts in the field of virtualizing the guitar into well-modelled software systems have faced a lot of practical limitations. The existing guitar simulation programs require additional devices such as Electromyography (EMG) controllers, or Musical Instrument Digital Interface (MIDI)-based recording devices. The EMG-based device is still a work in progress, the device is expensive and it was not very well received by the instrumentalists. There exists a gap in the bridge that joins physical instruments to their software counterparts. In this context, this paper aims to significantly remove the inaccuracies and drawbacks related to the existing solutions by accounting for the individual roles that each hand plays in the act of guitar strumming and consolidating them into a single system. The design of the proposed AirStrum system involves a multi-step process. Initially, a dataset is created by recording images of hand gestures corresponding to the playing of various chords on a guitar. The palm is detected, and its related skeleton image is generated using MediaPipe. Subsequently, a model based on a Convolutional Neural Network (CNN) is trained and validated using the employed dataset to adeptly recognize and classify guitar chords. Additionally, this model incorporates a velocity detection function for the strumming hand. Finally, the proposed system can play different sounds by inferring both the played chord and the strumming velocity from human actions. This comprehensive approach enables a sophisticated virtual guitar experience based on a system that responds dynamically to the users' gestures and strumming techniques. The conducted experiments demonstrate that AirStrum achieves an accuracy of 95.92%, and a brief preliminary survey related to its perceived utility and usability received a positive feedback rate of 58% from eight guitar players. |
| format | Article |
| id | doaj-art-9f8436f5488247578e478bae1c2e87a4 |
| institution | Kabale University |
| issn | 1220-1758 1841-4303 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | ICI Publishing House |
| record_format | Article |
| series | Revista Română de Informatică și Automatică |
| spelling | doaj-art-9f8436f5488247578e478bae1c2e87a42025-01-20T08:17:00ZengICI Publishing HouseRevista Română de Informatică și Automatică1220-17581841-43032024-12-0134412713910.33436/v34i4y202410AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique Beulah ARUL0Shashank PANDA1Tushar NAIR 2Department of Computer Science and Engineering, Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, Chennai, Tamil Nadu, India Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, Chennai, Tamil Nadu, India Sri Sivasubramaniya Nadar College of Engineering, Kalavakkam, Chennai, Tamil Nadu, India The efforts in the field of virtualizing the guitar into well-modelled software systems have faced a lot of practical limitations. The existing guitar simulation programs require additional devices such as Electromyography (EMG) controllers, or Musical Instrument Digital Interface (MIDI)-based recording devices. The EMG-based device is still a work in progress, the device is expensive and it was not very well received by the instrumentalists. There exists a gap in the bridge that joins physical instruments to their software counterparts. In this context, this paper aims to significantly remove the inaccuracies and drawbacks related to the existing solutions by accounting for the individual roles that each hand plays in the act of guitar strumming and consolidating them into a single system. The design of the proposed AirStrum system involves a multi-step process. Initially, a dataset is created by recording images of hand gestures corresponding to the playing of various chords on a guitar. The palm is detected, and its related skeleton image is generated using MediaPipe. Subsequently, a model based on a Convolutional Neural Network (CNN) is trained and validated using the employed dataset to adeptly recognize and classify guitar chords. Additionally, this model incorporates a velocity detection function for the strumming hand. Finally, the proposed system can play different sounds by inferring both the played chord and the strumming velocity from human actions. This comprehensive approach enables a sophisticated virtual guitar experience based on a system that responds dynamically to the users' gestures and strumming techniques. The conducted experiments demonstrate that AirStrum achieves an accuracy of 95.92%, and a brief preliminary survey related to its perceived utility and usability received a positive feedback rate of 58% from eight guitar players.https://rria.ici.ro/documents/1235/art._10_Beulah_Panda_Nair.pdfconvolutional neural networkchord classificationhand gesturemediapipesound bankvirtual guitarvelocity detection |
| spellingShingle | Beulah ARUL Shashank PANDA Tushar NAIR AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique Revista Română de Informatică și Automatică convolutional neural network chord classification hand gesture mediapipe sound bank virtual guitar velocity detection |
| title | AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique |
| title_full | AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique |
| title_fullStr | AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique |
| title_full_unstemmed | AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique |
| title_short | AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique |
| title_sort | airstrum a virtual guitar using real time hand gesture recognition and strumming technique |
| topic | convolutional neural network chord classification hand gesture mediapipe sound bank virtual guitar velocity detection |
| url | https://rria.ici.ro/documents/1235/art._10_Beulah_Panda_Nair.pdf |
| work_keys_str_mv | AT beulaharul airstrumavirtualguitarusingrealtimehandgesturerecognitionandstrummingtechnique AT shashankpanda airstrumavirtualguitarusingrealtimehandgesturerecognitionandstrummingtechnique AT tusharnair airstrumavirtualguitarusingrealtimehandgesturerecognitionandstrummingtechnique |