Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation
One of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10706846/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832590355196805120 |
|---|---|
| author | Jesus M. Fraile-Hernandez Anselmo Penas Pablo Moral |
| author_facet | Jesus M. Fraile-Hernandez Anselmo Penas Pablo Moral |
| author_sort | Jesus M. Fraile-Hernandez |
| collection | DOAJ |
| description | One of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that they promote. The research in this task involves a number of challenges such as the limited availability of labelled datasets, the subjectivity of the annotators and the time cost of annotation. This article present a definition of the Narrative Identification task, proposes an evaluation framework for Narrative Identification, and a methodology for the creation and annotation of Narrative Identification datasets taking into account the subjectivity of the task. Keeping in mind the goal of comparing systems performance, we explore how to reduce the annotation time while maintaining the reliability of the evaluation. Following this methodology, a set of eight tasks for narrative identification in the political domain has been developed in Spanish and English. Finally, we validated the evaluation framework by analysing its application to DIPROMATS 2024 shared task, together with the performance analysis of baseline and participant systems. |
| format | Article |
| id | doaj-art-bd033e6e732540508ca36dfa6a1ef7ae |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-bd033e6e732540508ca36dfa6a1ef7ae2025-01-24T00:01:50ZengIEEEIEEE Access2169-35362025-01-0113117341175310.1109/ACCESS.2024.347557910706846Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset CreationJesus M. Fraile-Hernandez0https://orcid.org/0009-0001-5474-4844Anselmo Penas1https://orcid.org/0000-0002-7867-0149Pablo Moral2https://orcid.org/0000-0003-3028-4369UNED NLP & IR Group, Universidad Nacional de Educación a Distancia (UNED), Madrid, SpainUNED NLP & IR Group, Universidad Nacional de Educación a Distancia (UNED), Madrid, SpainUNED NLP & IR Group, Universidad Nacional de Educación a Distancia (UNED), Madrid, SpainOne of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that they promote. The research in this task involves a number of challenges such as the limited availability of labelled datasets, the subjectivity of the annotators and the time cost of annotation. This article present a definition of the Narrative Identification task, proposes an evaluation framework for Narrative Identification, and a methodology for the creation and annotation of Narrative Identification datasets taking into account the subjectivity of the task. Keeping in mind the goal of comparing systems performance, we explore how to reduce the annotation time while maintaining the reliability of the evaluation. Following this methodology, a set of eight tasks for narrative identification in the political domain has been developed in Spanish and English. Finally, we validated the evaluation framework by analysing its application to DIPROMATS 2024 shared task, together with the performance analysis of baseline and participant systems.https://ieeexplore.ieee.org/document/10706846/Narrative identificationnatural language processingsocial media analysisevaluation methodologydatasets |
| spellingShingle | Jesus M. Fraile-Hernandez Anselmo Penas Pablo Moral Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation IEEE Access Narrative identification natural language processing social media analysis evaluation methodology datasets |
| title | Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation |
| title_full | Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation |
| title_fullStr | Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation |
| title_full_unstemmed | Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation |
| title_short | Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation |
| title_sort | automatic identification of narratives evaluation framework annotation methodology and dataset creation |
| topic | Narrative identification natural language processing social media analysis evaluation methodology datasets |
| url | https://ieeexplore.ieee.org/document/10706846/ |
| work_keys_str_mv | AT jesusmfrailehernandez automaticidentificationofnarrativesevaluationframeworkannotationmethodologyanddatasetcreation AT anselmopenas automaticidentificationofnarrativesevaluationframeworkannotationmethodologyanddatasetcreation AT pablomoral automaticidentificationofnarrativesevaluationframeworkannotationmethodologyanddatasetcreation |