Optimizing the Learnable RoPE Theta Parameter in Transformers by Zhigao Huang, Musheng Chen

Get full text

Exploring the impact of fixed theta values in RoPE on character-level language model performance and efficiency by Zhigao Huang, Musheng Chen, Shiyan Zheng

Get full text

Dynamic Mixture of Experts for Adaptive Computation in Character-Level Transformers by Zhigao Huang, Musheng Chen, Shiyan Zheng

Get full text

Spectral Adaptive Dropout: Frequency-Based Regularization for Improved Generalization by Zhigao Huang, Musheng Chen, Shiyan Zheng

Get full text

Spectral momentum integration: hybrid optimization of frequency and time domain gradients by Zhigao Huang, Musheng Chen, Shiyan Zheng

Get full text