Seeing the Sound: Multilingual Lip Sync for Real-Time Face-to-Face Translation by Amirkia Rafiei Oskooei, Mehmet S. Aktaş, Mustafa Keleş

“…Our research tackles the primary challenge of achieving accurate lip synchronization across culturally diverse languages, filling a significant gap in the literature by evaluating the generalizability of lip sync models beyond English. Specifically, we develop a novel evaluation framework combining quantitative lip sync error metrics and qualitative assessments by human observers. …”
Get full text