MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning
This paper presents MoCoUTRL: a Momentum Contrastive Framework for Unsupervised Text Representation Learning. This model improves two aspects of recently popular contrastive learning algorithms in natural language processing (NLP). Firstly, MoCoUTRL employs multi-granularity semantic contrastive lea...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2023-12-01
|
| Series: | Connection Science |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/09540091.2023.2221406 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850211082404102144 |
|---|---|
| author | Ao Zou Wenning Hao Dawei Jin Gang Chen Feiyan Sun |
| author_facet | Ao Zou Wenning Hao Dawei Jin Gang Chen Feiyan Sun |
| author_sort | Ao Zou |
| collection | DOAJ |
| description | This paper presents MoCoUTRL: a Momentum Contrastive Framework for Unsupervised Text Representation Learning. This model improves two aspects of recently popular contrastive learning algorithms in natural language processing (NLP). Firstly, MoCoUTRL employs multi-granularity semantic contrastive learning objectives, enabling a more comprehensive understanding of the semantic features of samples. Secondly, MoCoUTRL uses a dynamic dictionary to act as the approximately ground-truth representation for each token, providing the pseudo labels for token-level contrastive learning. The MoCoUTRL can extend the use of pre-trained language models (PLM) and even large-scale language models (LLM) into a plug-and-play semantic feature extractor that can fuel multiple downstream tasks. Experimental results on several publicly available datasets and further theoretical analysis validate the effectiveness and interpretability of the proposed method in this paper. |
| format | Article |
| id | doaj-art-8d9dceb6c5ce4e9fa3fc2acb9be0756a |
| institution | OA Journals |
| issn | 0954-0091 1360-0494 |
| language | English |
| publishDate | 2023-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Connection Science |
| spelling | doaj-art-8d9dceb6c5ce4e9fa3fc2acb9be0756a2025-08-20T02:09:38ZengTaylor & Francis GroupConnection Science0954-00911360-04942023-12-0135110.1080/09540091.2023.22214062221406MoCoUTRL: a momentum contrastive framework for unsupervised text representation learningAo Zou0Wenning Hao1Dawei Jin2Gang Chen3Feiyan Sun4Command and Control Engineering College, Army Engineering University of PLACommand and Control Engineering College, Army Engineering University of PLACommand and Control Engineering College, Army Engineering University of PLACommand and Control Engineering College, Army Engineering University of PLACommand and Control Engineering College, Army Engineering University of PLAThis paper presents MoCoUTRL: a Momentum Contrastive Framework for Unsupervised Text Representation Learning. This model improves two aspects of recently popular contrastive learning algorithms in natural language processing (NLP). Firstly, MoCoUTRL employs multi-granularity semantic contrastive learning objectives, enabling a more comprehensive understanding of the semantic features of samples. Secondly, MoCoUTRL uses a dynamic dictionary to act as the approximately ground-truth representation for each token, providing the pseudo labels for token-level contrastive learning. The MoCoUTRL can extend the use of pre-trained language models (PLM) and even large-scale language models (LLM) into a plug-and-play semantic feature extractor that can fuel multiple downstream tasks. Experimental results on several publicly available datasets and further theoretical analysis validate the effectiveness and interpretability of the proposed method in this paper.http://dx.doi.org/10.1080/09540091.2023.2221406natural language processingtext representation learningmomentum contrastalignmentuniformity |
| spellingShingle | Ao Zou Wenning Hao Dawei Jin Gang Chen Feiyan Sun MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning Connection Science natural language processing text representation learning momentum contrast alignment uniformity |
| title | MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning |
| title_full | MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning |
| title_fullStr | MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning |
| title_full_unstemmed | MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning |
| title_short | MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning |
| title_sort | mocoutrl a momentum contrastive framework for unsupervised text representation learning |
| topic | natural language processing text representation learning momentum contrast alignment uniformity |
| url | http://dx.doi.org/10.1080/09540091.2023.2221406 |
| work_keys_str_mv | AT aozou mocoutrlamomentumcontrastiveframeworkforunsupervisedtextrepresentationlearning AT wenninghao mocoutrlamomentumcontrastiveframeworkforunsupervisedtextrepresentationlearning AT daweijin mocoutrlamomentumcontrastiveframeworkforunsupervisedtextrepresentationlearning AT gangchen mocoutrlamomentumcontrastiveframeworkforunsupervisedtextrepresentationlearning AT feiyansun mocoutrlamomentumcontrastiveframeworkforunsupervisedtextrepresentationlearning |