VirExp: Automated Expression and Gestures for Virtual Collaborative Environment

Abstract

Abstract In an era where virtual collaboration has become integral to professional, educational, and social interactions, the absence of physical expressiveness in digital communication presents a critical challenge. The ”VirExp” study addresses this by developing an innovative real-time pipeline capable of detecting facial expressions and upper body gestures through a standard webcam/inbuilt cameras and representing these as expressive animations via a virtual avatar. This research aims to bridge the expressive gap in virtual environments by translating natural human expressions into dynamic avatar movements, thus enhancing authenticity, expression fidelity, and user engagement in virtual collaboration. The system is designed to answer three core research questions: (1) What specific facial expressions and body gestures convey distinct expressions, and what are the corresponding sequences of skeletal points associated with these gestures? (2) In a real-time skeletal point sequence, how can we identify predefined facial expressions and body gesture patterns within near real-time and express them? (3) To what extent will the suggested solution perform and help users facilitate collaborative interactions in the virtual space? To achieve this, the research employs the Design Science Research Methodology, involving iterative development, rigorous evaluation, and empirical validation. As discussed in Chapter 4, for the technical implementation, skeletal data is captured from 25 participants using the MediaPipe Holistic library, which provides 543 facial and body landmarks. A total of 15,000 frames are collected for each expression. Machine learning models, LSTM, DTW, and Transformers, are trained to recognize expression-linked gesture patterns. As mentioned under Chapter 5, the best-performing model, LSTM-based architecture, achieved 91.67% accuracy. Real-time expression detection is integrated with avatar animation in Unity using Vroid and Mixamo avatars, facilitated by FastAPI for synchronization. User studies involving 30 participants evaluated the system’s performance on both technical metrics and experiential feedback. Expressions, including “High Laugh,” “Subtle Laugh,” “Surprise,” and “Neutral”, were captured and translated into avatar expressions. As discussed in Chapter 5, surveys demonstrated 93.33% user agreement with expression representation accuracy. This research contributes novel skeletal point patterns-based expression representation, introduces a low-cost, accessible framework for real-time avatar expressiveness without relying on sophisticated hardware, and provides empirical evidence of improved collaboration and communication in virtual spaces. Limitations include the focus on a defined set of expressions and exclusion of audio cues, while future work is suggested to expand expression classes, integrate cultural adaptability, and explore broader applications in education, therapy, and immersive metaverse contexts. Ultimately, ”VirExp” redefines how expressions are communicated in digital interactions, offering a technically robust and user-centered solution that transforms avatars from static representations into expression-responsive communicators in virtual collaboration. Key Words: Real-time expression recognition, virtual collaboration, facial expression detection, upper-body gestures, avatar animation, skeletal point tracking, virtual reality, avatar expressiveness, gesture-based communication, virtual environment