SURVEILLING KEYBOARD ACTIVITY ON MOBILE DEVICES THROUGH ACOUSTIC AND INERTIAL SENSORS

Abstract

ABSTRACT Keystroke detection on mobile devices using multimodal sensor data has garnered significant interest within the domains of human-computer interaction, mobile security, and assistive technologies. This study investigates the detection and classification of keypress events on a virtual keyboard by leveraging acoustic signals alongside inertial sensor readings specifically accelerometer and gyroscope data collected under controlled, silent ambient conditions. The primary objective is to develop an end-to-end pipeline encompassing robust signal preprocessing, advanced feature extraction, and the application of machine learning models to accurately identify individual keypresses. The experimental dataset consists of audio recordings in .m4a format, each containing 30 keypresses per letter, spanning durations of 0 to 25 seconds. Initial analyses of raw audio signals revealed significant noise and artifacts that adversely impacted classification accuracy. To mitigate these challenges, a comprehensive preprocessing strategy was implemented, including audio format conversion, amplitude normalization, precise segmentation, and trimming. Feature extraction techniques incorporated both time-domain and frequency-domain characteristics. Supervised learning models were trained and evaluated using extracted features from both audio and inertial sensor modalities. Preliminary audio-only models utilizing conventional feature sets and classifiers achieved limited success, with Random Forest models reaching a baseline accuracy of approximately 21%. Through iterative optimization of preprocessing workflows, feature engineering, and classifier tuning, this accuracy was substantially enhanced, culminating in a peak classification accuracy of 94% for the Random Forest model applied to audio signals. Parallel classification efforts on inertial sensor data demonstrated a maximum accuracy of 80%, also achieved via Random Forest classifiers, indicating the significant discriminative power of motion sensor data in keystroke detection. The study highlights the complementary nature of audio and inertial sensor data and emphasizes the efficacy of ensemble-based machine learning methods for keypress recognition. Future research will focus on real-time implementation, scalability across device types, and deeper exploration of sensor fusion techniques to further enhance performance and robustness in diverse operating environments. Keywords: human-computer interaction, accelerometer, gyroscope, feature extraction, Random Forest, ensemble-based model, sensor fusion