SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification
Hierarchical text classification (HTC) is a challenging task that requires classifiers to solve a series of multi-label subtasks considering hierarchical dependencies among labels. Recent studies have introduced prompt tuning to create closer connections between the language model (LM) and the compl...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-01-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/27/2/128 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850081616218554368 |
|---|---|
| author | He Zhu Jinxiang Xia Ruomei Liu Bowen Deng |
| author_facet | He Zhu Jinxiang Xia Ruomei Liu Bowen Deng |
| author_sort | He Zhu |
| collection | DOAJ |
| description | Hierarchical text classification (HTC) is a challenging task that requires classifiers to solve a series of multi-label subtasks considering hierarchical dependencies among labels. Recent studies have introduced prompt tuning to create closer connections between the language model (LM) and the complex label hierarchy. However, we find that the model’s attention to the prompt gradually decreases as the prompt moves from the input to the output layer, revealing the limitations of previous prompt tuning methods for HTC. Given the success of prefix tuning-based studies in natural language understanding tasks, we introduce <b>S</b>tructural entro<b>P</b>y gu<b>I</b>ded p<b>R</b>ef<b>I</b>x <b>T</b>uning (<b>SPIRIT</b>). Specifically, we extract the essential structure of the label hierarchy via structural entropy minimization and decode the abstractive structural information as the prefix to prompt all intermediate layers in the LM. Additionally, a depth-wise reparameterization strategy is developed to enhance optimization and propagate the prefix throughout the LM layers. Extensive evaluation on four popular datasets demonstrates that SPIRIT achieves a state-of-the-art performance. |
| format | Article |
| id | doaj-art-8f1b0df5edea48a4b3dc4fa38e4cd39f |
| institution | DOAJ |
| issn | 1099-4300 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj-art-8f1b0df5edea48a4b3dc4fa38e4cd39f2025-08-20T02:44:42ZengMDPI AGEntropy1099-43002025-01-0127212810.3390/e27020128SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text ClassificationHe Zhu0Jinxiang Xia1Ruomei Liu2Bowen Deng3State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, No. 37 Xue Yuan Road, Hai Dian District, Beijing 100191, ChinaState Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, No. 37 Xue Yuan Road, Hai Dian District, Beijing 100191, ChinaState Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, No. 37 Xue Yuan Road, Hai Dian District, Beijing 100191, ChinaSchool of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou 215213, ChinaHierarchical text classification (HTC) is a challenging task that requires classifiers to solve a series of multi-label subtasks considering hierarchical dependencies among labels. Recent studies have introduced prompt tuning to create closer connections between the language model (LM) and the complex label hierarchy. However, we find that the model’s attention to the prompt gradually decreases as the prompt moves from the input to the output layer, revealing the limitations of previous prompt tuning methods for HTC. Given the success of prefix tuning-based studies in natural language understanding tasks, we introduce <b>S</b>tructural entro<b>P</b>y gu<b>I</b>ded p<b>R</b>ef<b>I</b>x <b>T</b>uning (<b>SPIRIT</b>). Specifically, we extract the essential structure of the label hierarchy via structural entropy minimization and decode the abstractive structural information as the prefix to prompt all intermediate layers in the LM. Additionally, a depth-wise reparameterization strategy is developed to enhance optimization and propagate the prefix throughout the LM layers. Extensive evaluation on four popular datasets demonstrates that SPIRIT achieves a state-of-the-art performance.https://www.mdpi.com/1099-4300/27/2/128information theorystructural entropyrepresentation learning |
| spellingShingle | He Zhu Jinxiang Xia Ruomei Liu Bowen Deng SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification Entropy information theory structural entropy representation learning |
| title | SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification |
| title_full | SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification |
| title_fullStr | SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification |
| title_full_unstemmed | SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification |
| title_short | SPIRIT: Structural Entropy Guided Prefix Tuning for Hierarchical Text Classification |
| title_sort | spirit structural entropy guided prefix tuning for hierarchical text classification |
| topic | information theory structural entropy representation learning |
| url | https://www.mdpi.com/1099-4300/27/2/128 |
| work_keys_str_mv | AT hezhu spiritstructuralentropyguidedprefixtuningforhierarchicaltextclassification AT jinxiangxia spiritstructuralentropyguidedprefixtuningforhierarchicaltextclassification AT ruomeiliu spiritstructuralentropyguidedprefixtuningforhierarchicaltextclassification AT bowendeng spiritstructuralentropyguidedprefixtuningforhierarchicaltextclassification |