Lip Synchronization Modeling for Sinhala Speech

Abstract

Lip synchronization also known as visual speech animation, is the process of matching the speech with the lip movements. Visual speech animation has a great impact in the gaming and animation film industry, due to the reason that it provides the realistic experience to the users. Furthermore, this technology also supports better communication for deaf people. For most of the European languages, lip synchronizing models have been developed and used vastly in the entertainment industries. However, there are still no research experiments conducted towards the speech animation of the Sinhala language. This is due to the Less contribution towards research development and unavailability of resources. This research is focused on the problem of achieving a lip synchronization model for the Sinhala language. This project presents a study on how to map from acoustic speech to visual speech with the goal of generating perceptually natural speech animation. Initially, this study follows a deep learning approach and terminates due to not having enough video data to achieve a good performance. Next, the experiments on developing a visemes alphabet is carried out using a static visemes approach on a video data set created by the author. The implemented lip synchronizing model was evaluated using a subjective evaluation based on six different categories. The model achieved a 69% accuracy for the subjective evaluation using the static visemes approach. As an initiative research on speech animation for the Sinhala language, this model accurately animates individual words rather than long sentences.