Metamorphic testing of named entity recognition systems: A case study
Abstract Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable....
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2022-08-01
|
| Series: | IET Software |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/sfw2.12058 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832546650816512000 |
|---|---|
| author | Yezi Xu Zhi Quan Zhou Xiaoxia Zhang Jing Wang Mingyue Jiang |
| author_facet | Yezi Xu Zhi Quan Zhou Xiaoxia Zhang Jing Wang Mingyue Jiang |
| author_sort | Yezi Xu |
| collection | DOAJ |
| description | Abstract Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable. To address the oracle problem, this study proposes to apply metamorphic testing (MT). The authors conduct a case study with Litigant, an industrial NER system of the Ant Group, and show that MT can effectively detect real‐life bugs in the absence of an ideal oracle. The authors further investigate the causes for a series of entity recognition failures detected. Outcomes of this research further justify the application of MT to the natural language processing domain as well as provide hints for practitioners to improve the quality process of their NER systems. |
| format | Article |
| id | doaj-art-2549c35b1b2441fabcb98e1a6dc40f9d |
| institution | Kabale University |
| issn | 1751-8806 1751-8814 |
| language | English |
| publishDate | 2022-08-01 |
| publisher | Wiley |
| record_format | Article |
| series | IET Software |
| spelling | doaj-art-2549c35b1b2441fabcb98e1a6dc40f9d2025-02-03T06:47:34ZengWileyIET Software1751-88061751-88142022-08-0116438640410.1049/sfw2.12058Metamorphic testing of named entity recognition systems: A case studyYezi Xu0Zhi Quan Zhou1Xiaoxia Zhang2Jing Wang3Mingyue Jiang4Alibaba and Ant Group Ant Z‐Space Hangzhou ChinaAlibaba and Ant Group Ant Z‐Space Hangzhou ChinaAlibaba and Ant Group Ant Z‐Space Hangzhou ChinaAlibaba and Ant Group Ant Z‐Space Hangzhou ChinaSchool of Information Science and Technology Zhejiang Sci‐Tech University Hangzhou ChinaAbstract Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable. To address the oracle problem, this study proposes to apply metamorphic testing (MT). The authors conduct a case study with Litigant, an industrial NER system of the Ant Group, and show that MT can effectively detect real‐life bugs in the absence of an ideal oracle. The authors further investigate the causes for a series of entity recognition failures detected. Outcomes of this research further justify the application of MT to the natural language processing domain as well as provide hints for practitioners to improve the quality process of their NER systems.https://doi.org/10.1049/sfw2.12058metamorphic relationsmetamorphic testingnamed entity recognitionnatural language processing |
| spellingShingle | Yezi Xu Zhi Quan Zhou Xiaoxia Zhang Jing Wang Mingyue Jiang Metamorphic testing of named entity recognition systems: A case study IET Software metamorphic relations metamorphic testing named entity recognition natural language processing |
| title | Metamorphic testing of named entity recognition systems: A case study |
| title_full | Metamorphic testing of named entity recognition systems: A case study |
| title_fullStr | Metamorphic testing of named entity recognition systems: A case study |
| title_full_unstemmed | Metamorphic testing of named entity recognition systems: A case study |
| title_short | Metamorphic testing of named entity recognition systems: A case study |
| title_sort | metamorphic testing of named entity recognition systems a case study |
| topic | metamorphic relations metamorphic testing named entity recognition natural language processing |
| url | https://doi.org/10.1049/sfw2.12058 |
| work_keys_str_mv | AT yezixu metamorphictestingofnamedentityrecognitionsystemsacasestudy AT zhiquanzhou metamorphictestingofnamedentityrecognitionsystemsacasestudy AT xiaoxiazhang metamorphictestingofnamedentityrecognitionsystemsacasestudy AT jingwang metamorphictestingofnamedentityrecognitionsystemsacasestudy AT mingyuejiang metamorphictestingofnamedentityrecognitionsystemsacasestudy |