Collaborative Hand Gesture and Face Movement Recognition for a Virtual Environment

Abstract

In the current days, most of the applications and researches are moving towards more immersive and virtual environments. To give the enhanced immersive experience to the user, simulation should be precisely accurate and realistic. In office environments and study environments, sometimes it is required to have a group conversation among several people in different locations. It is difficult to have video conferences in those circumstances because of issues like network bandwidth or privacy issues. It is also less effective to have only text based or audio based discussions as it lacks the realistic senses that make real discussions more useful. This study tries to address that by developing a virtual environment for teleconferencing called virtual conference. In the prototype of this study, it has implemented a virtual environment where users can have a live conference. There, the users can have a voice based discussion. All the users have a separate virtual environment where the current user becomes main avatar and other users become other avatars that are situated in front of them. All the users’ head movements are simulated in the virtual environment to give an enhanced immersive experience to the users. Users can initiate actions like private chats or silent virtual environment by hand and head gestures while attending virtual conference. In the process this study introduces a mechanism for six dimensional (6D) face pose detection in real time using positioning of three feature points of the face. It also has introduced Kalman filters to enhance performance of real-time face pose detection and gesture recognition. It implements a mechanism to detect view position of the user in display area using face pose and positioning of display and camera. According to user feedback, it is clear that users are more enthusiastic about this virtual environment and also about these kinds of virtual environments according to user feedbacks. It is possible to use this mechanism of face pose detection using three face features in proper lighting conditions for real-time face pose detection as it accurately detects 6D face pose. It is learnt that, it is necessary to have dynamic hand gestures than static hand gestures for more natural interaction. It is also clear that, it is possible to evaluate the user view position in real-time using this mechanism introduced in the study. This study has introduced more precise and immersive virtual environment where it is possible to conduct a virtual conference.