Optimizing the Learnable RoPE Theta Parameter in Transformers by Zhigao Huang, Musheng Chen

Subjects: “…Rotary position embedding…”
Get full text

P-BERT: Toward Long Sequence Modeling by Enabling Language Representation With Prefix Sequence Compression, Soft Position Embedding, and Data Augmentation for Patent Relevance Asse... by Fei Wang, Xingchen Shi, Dongsheng Wang, Yinxia Lou

Subjects: Get full text

Enhancing the Transformer Model with a Convolutional Feature Extractor Block and Vector-Based Relative Position Embedding for Human Activity Recognition by Xin Guo, Young Kim, Xueli Ning, Se Dong Min

Subjects: Get full text

Positional embeddings and zero-shot learning using BERT for molecular-property prediction by Medard Edmund Mswahili, JunHa Hwang, Jagath C. Rajapakse, Kyuri Jo, Young-Seob Jeong

Subjects: Get full text

Hyperspectral Image Super-Resolution via Grouped Second-Order Spatial Features and Spectral Attention Network by Allen Patnaik, M. K. Bhuyan, Sultan Alfarhood, Mejdl Safran

Subjects: Get full text

Exploring the impact of fixed theta values in RoPE on character-level language model performance and efficiency by Zhigao Huang, Musheng Chen, Shiyan Zheng

Subjects: Get full text

An Ensemble of Vision-Language Transformer-Based Captioning Model With Rotatory Positional Embeddings by K. B. Sathyanarayana, Dinesh Naik

Subjects: Get full text