Custom Hand-Gestured Controlled Video Game using RGB Camera

Abstract

Video gaming is a rapidly evolving form of entertainment introducing young minds into the field and increasing competition within the industry. Due to the requirement of high-level passionate experience among players, game designers are forcing to create more realistic and enjoyable game plans. Gesture-based Monitoring (GM) in Human-Computer Interaction (HCI) plays a crucial role in increasing player’s enjoyment and delight during game play and chosen as a more significant modality. In GM hand gestures are complicated for a computer to recognize as they include more complex signs apart from simple leg and arm actions. The environment setup is also becoming complex due to deployment of sensors that are used to record the depth of an image and recognize complex activities. The present study proposed a method based on augmented reality that allows a player to experience a real- time gaming environment. The method uses a conventional RGB camera using media-pipe and LSTM deep learning models while ensuring the same level of recognition accuracy. The study found that using simple RNN for postures classification using landmarks gave an accuracy of 97. Using RNN with LSTM technique for gesture classification gave an output with an accuracy of 94 and under 1 second response time on application with simple games. Pipelining new models to the Mediapipe gives us a large future scope like pipelining action recognitions, behavioural models can enrich the virtual gaming itself.