Optimizing the Learnable RoPE Theta Parameter in Transformers - Zhigao Huang; Musheng Chen (IEEE Access)
Rotary Position Embedding (RoPE) enhances Transformer models by encoding relative positions through a frequency parameter $θ $ , but conventional implementations fix $θ $ , constraining adaptability. We conduct the first systematic study of learnable RoPE $θ $ , introducing four optimization strategies—separate learning rates, layer-wise initialization, cosine annealing scheduling, and sigmoid-based constraints—to stabilize and refine positional learning. Our approach demonstrates modest but consistent benefits across multiple datasets including Tiny Shakespeare, WikiText-103, and IWSLT’14, achieving measurable gains in validation loss, perplexity, and BLEU scores relative to a fixed- $θ $ baseline while maintaining high inference throughput and requiring minimal architectural modifications. Ablation experiments quantify each strategy’s contribution and offer practical integration guidelines. This adaptive position encoding framework provides a foundation for large-scale pretraining and diverse sequence modeling applications.