SeEar: Enhancing Learning Experience for the Deaf and Hard-of-Hearing (DHH) Students with Real-time AR Captioning Interfaces

Abstract

Abstract Deaf and hard-of-hearing (DHH) students encounter significant challenges within educational settings, primarily stemming from limited exposure to written and spoken language, a lack of tailored educational tools and methodologies, and restricted access to educational resources. We, therefore, adopted a User-Centered Design (UCD) process, collaborating with 8 DHH students and 2 Teachers of the Deaf (ToDs) from a School of Deaf from Sri Lanka to effectively co-design, develop and utilize a real-time captioning augmented reality (AR) system to their school settings, aiming to enhance their learning experience. We address four main research questions: how do we present and place real-time captions, how do we visualize wearer transcriptions and how do we visually represent inaccurate transcriptions from Automatic Speech Recognition (ASR) systems. User study with 24 DHH participants revealed a strong preference (87.5%) for our system, underscoring its potential to enhance learning experience. We share a comprehensive needs analysis, the UCD process, and insights from user feedback and evaluations, highlighting the effectiveness of tailored AR caption interfaces in improving education for DHH students. We discuss the implications of our findings for future development of educational technologies for DHH students. Our findings contribute to further exploration of the potential of tailored AR caption interfaces in addressing the educational challenges faced by DHH students and enhancing their learning experience in specialized educational settings.