Mathias did his master thesis about an innovative and new idea – using neural networks to estimate the hand pose while learning typing. Now we published his research in the International Journal of Interactive Mobile technologies.
Abstract:
This paper focuses on enhancing touch-typing skills through state-of-the-art deep learning technologies. It has been demonstrated that writing posture can be assessed using integrated keyboard detection and hand pose estimation in real-time browser environments. A unique dataset was created for training the object detection model to recognize individual keyboards. The object detection models were trained using various architectures and then assessed for both inference time and accuracy. In addition, a hand pose estimation was implemented to precisely recognize 21 knuckle points of the hand and compute their exact positions. In order to implement object detection and hand pose estimation, a stream transmission of the keyboard and hand scene is required. For this purpose, server-client communication via a QR code connection is implemented to transfer the stream between two mobile devices. Based on these deep learning technologies, a mobile web app was developed that offers 170 learning courses for touch-typing training. With the help of the Typing Learning Academy prototype, a study was conducted as part of this paper to evaluate the usability and utility of the learning app. This research demonstrates the potential for enhancing the development of writing skills through touch typing through the utilization of advanced deep learning technologies.
[full article @ journal’s homepage]
[full article @ ResearchGate]
Reference: Mattersberger, M., Wachtler, J., & Ebner, M. (2024). Handpose Estimation Based Learning Academy for Improving Typing Efficiency by Using Mobile Technologies. International Journal of Interactive Mobile Technologies (iJIM), 18(10), pp. 59–70. https://doi.org/10.3991/ijim.v18i10.49037