Virtual Keyboard using Hand and Finger Tracking

Abstract

This research is presented with a vision based human-computer interaction system for text-entry using the colour and depth sensing capabilities with the Microsoft Kinect sensor. It turns a quadrangle shaped sheet of paper into a customizable keyboard layout for text-entry by tracking hand and finger motions. The keyboard layout configuration provides the ability to define regions on a sheet of paper as well as to mapping with specific keys. Then the calibration process involves the detection of the customized layout in the image. Further the detection of the layout is performed by scale-invariant feature point matching with SURF algorithm using OpenCV library. Additionally, the finger detection module uses depth segmentation to extract hand region and use it to detect the fingers. Then key press gestures are identified by the detected fingers by calculating the distance differences between the layout and the fingertip as well as the palm center and the fingertip as another approach. Subsequently, results are presented after evaluating the prototype in terms of system and user evaluations. The prototype scored the average of 75% of SUS questionnaires. Finally, it was presented with the future extensions and improvements to the prototype