A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation
Word Sense Disambiguation is determining the correct sense of an ambiguous word within context. It plays a crucial role in natural language applications such as machine translation, question-answering, chatbots, information retrieval, sentiment analysis, and overall language comprehension. Recent ad...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10857270/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832088067503030272 |
|---|---|
| author | Robbel Habtamu Yigzaw Beakal Gizachew Assefa Elefelious Getachew Belay |
| author_facet | Robbel Habtamu Yigzaw Beakal Gizachew Assefa Elefelious Getachew Belay |
| author_sort | Robbel Habtamu Yigzaw |
| collection | DOAJ |
| description | Word Sense Disambiguation is determining the correct sense of an ambiguous word within context. It plays a crucial role in natural language applications such as machine translation, question-answering, chatbots, information retrieval, sentiment analysis, and overall language comprehension. Recent advancements in this area have focused on utilizing deep contextual models to address these challenges. However, despite this positive progress, semantical and syntactical ambiguity remains a challenge, especially when dealing with polysomy words, and it is considered an AI-complete problem. In this work, we propose an approach that integrates hierarchical attention mechanisms and BERT embeddings to enhance WSD performance. Our model, incorporating local and global attention, demonstrates significant improvements in accuracy, particularly in complex sentence structures. To the best of our knowledge, our model is the first to incorporate hierarchical attention mechanisms integrated with contextual embedding. We conducted experiment on publicly available datasets for English and Italian language. Experimental results show that our model achieves state-of-the-art results in WSD, surpassing baseline models up to 2.9% F1 accuracy on English WSD. Additionally, it demonstrates superior performance in Italian WSD, outperforming existing papers up to 0.7% F1 accuracy. We further adapted the model for Amharic word sense disambiguation. Despite the absence of a standard benchmark dataset for Amharic WSD, our model achieved an accuracy of 92.4% on a dataset we prepared ourselves. Our findings underscore the significance of linguistic features in contextual information capture for WSD. While Part-of-Speech (POS) tagging has a limited impact, word embeddings significantly influence performance. Local and global attention further improve results, particularly at the word level. Overall the results emphasize the importance of context in WSD, advancing context-aware natural language processing systems. |
| format | Article |
| id | doaj-art-baffb0279a844626a4a29b7001afed07 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-baffb0279a844626a4a29b7001afed072025-02-06T00:00:39ZengIEEEIEEE Access2169-35362025-01-0113217442175810.1109/ACCESS.2025.353630010857270A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense DisambiguationRobbel Habtamu Yigzaw0https://orcid.org/0009-0000-7432-1489Beakal Gizachew Assefa1https://orcid.org/0000-0001-9510-5216Elefelious Getachew Belay2https://orcid.org/0000-0001-8720-6295School of Information Technology and Engineering, Addis Ababa Institute of Technology, Addis Ababa University, Addis Ababa, EthiopiaSchool of Information Technology and Engineering, Addis Ababa Institute of Technology, Addis Ababa University, Addis Ababa, EthiopiaSchool of Information Technology and Engineering, Addis Ababa Institute of Technology, Addis Ababa University, Addis Ababa, EthiopiaWord Sense Disambiguation is determining the correct sense of an ambiguous word within context. It plays a crucial role in natural language applications such as machine translation, question-answering, chatbots, information retrieval, sentiment analysis, and overall language comprehension. Recent advancements in this area have focused on utilizing deep contextual models to address these challenges. However, despite this positive progress, semantical and syntactical ambiguity remains a challenge, especially when dealing with polysomy words, and it is considered an AI-complete problem. In this work, we propose an approach that integrates hierarchical attention mechanisms and BERT embeddings to enhance WSD performance. Our model, incorporating local and global attention, demonstrates significant improvements in accuracy, particularly in complex sentence structures. To the best of our knowledge, our model is the first to incorporate hierarchical attention mechanisms integrated with contextual embedding. We conducted experiment on publicly available datasets for English and Italian language. Experimental results show that our model achieves state-of-the-art results in WSD, surpassing baseline models up to 2.9% F1 accuracy on English WSD. Additionally, it demonstrates superior performance in Italian WSD, outperforming existing papers up to 0.7% F1 accuracy. We further adapted the model for Amharic word sense disambiguation. Despite the absence of a standard benchmark dataset for Amharic WSD, our model achieved an accuracy of 92.4% on a dataset we prepared ourselves. Our findings underscore the significance of linguistic features in contextual information capture for WSD. While Part-of-Speech (POS) tagging has a limited impact, word embeddings significantly influence performance. Local and global attention further improve results, particularly at the word level. Overall the results emphasize the importance of context in WSD, advancing context-aware natural language processing systems.https://ieeexplore.ieee.org/document/10857270/Contextual embeddingshierarchical attentionnatural language processingword sense disambiguation |
| spellingShingle | Robbel Habtamu Yigzaw Beakal Gizachew Assefa Elefelious Getachew Belay A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation IEEE Access Contextual embeddings hierarchical attention natural language processing word sense disambiguation |
| title | A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation |
| title_full | A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation |
| title_fullStr | A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation |
| title_full_unstemmed | A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation |
| title_short | A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation |
| title_sort | hybrid contextual embedding and hierarchical attention for improving the performance of word sense disambiguation |
| topic | Contextual embeddings hierarchical attention natural language processing word sense disambiguation |
| url | https://ieeexplore.ieee.org/document/10857270/ |
| work_keys_str_mv | AT robbelhabtamuyigzaw ahybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation AT beakalgizachewassefa ahybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation AT elefeliousgetachewbelay ahybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation AT robbelhabtamuyigzaw hybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation AT beakalgizachewassefa hybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation AT elefeliousgetachewbelay hybridcontextualembeddingandhierarchicalattentionforimprovingtheperformanceofwordsensedisambiguation |