Forward Hand Gesture Spotting and Prediction Using HMM-DNN Model

Mahmoud Elmezain; Majed M. Alwateer; Rasha El-Agamy; Elsayed Atlam; Hani M. Ibrahim

doi:10.3390/informatics10010001

journal article Open Access Dec 28, 2022

Forward Hand Gesture Spotting and Prediction Using HMM-DNN Model

Mahmoud Elmezain

Majed M. Alwateer Rasha El-Agamy Elsayed Atlam

Hani M. Ibrahim

Informatics Vol. 10 No. 1 pp. 1 · MDPI AG

View at Publisher Save 10.3390/informatics10010001

Abstract

Automatic key gesture detection and recognition are difficult tasks in Human–Computer Interaction due to the need to spot the start and the end points of the gesture of interest. By integrating Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs), the present research provides an autonomous technique that carries out hand gesture spotting and prediction simultaneously with no time delay. An HMM can be used to extract features, spot the meaning of gestures using a forward spotting mechanism with varying sliding window sizes, and then employ Deep Neural Networks to perform the recognition process. Therefore, a stochastic strategy for creating a non-gesture model using HMMs with no training data is suggested to accurately spot meaningful number gestures (0–9). The non-gesture model provides a confidence measure, which is utilized as an adaptive threshold to determine where meaningful gestures begin and stop in the input video stream. Furthermore, DNNs are extremely efficient and perform exceptionally well when it comes to real-time object detection. According to experimental results, the proposed method can successfully spot and predict significant motions with a reliability of 94.70%.

Topics

No keywords indexed for this article. Browse by subject →

References

33

[1]

Zhao, Y., Wang, W., and Wang, Y. (2011, January 9–11). A real-time hand gesture recognition method. Proceedings of the 2011 International Conference on Electronics, Communications and Control (ICECC), Ningbo, China. 10.1109/icecc.2011.6066597

[2]

Hand Gesture Recognition Based on Computer Vision: A Review of Techniques

Munir Oudah, Ali Al-Naji, Javaan Chahl

Journal of Imaging 10.3390/jimaging6080073

[3]

Raudonis "Recognition of basketball referee signals from real-time videos" J. Ambient Intell. Humaniz. Comput. (2019)

[4]

Muhammad "Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation" IEEE Access (2020) 10.1109/access.2020.3032140

[5]

Vaitkevičius, A., Taroza, M., Blažauskas, T., Damaševičius, R., Maskeliūnas, R., and Woźniak, M. (2019). Recognition of American Sign Language Gestures in a Virtual Reality Using Leap Motion. Appl. Sci., 9. 10.3390/app9030445

[6]

Rezende "Development and validation of a Brazilian sign language database for human gesture recognition" Neural Comput. Appl. (2021) 10.1007/s00521-021-05802-4

[7]

Afza "A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection" Image Vis. Comput. (2021) 10.1016/j.imavis.2020.104090

[8]

Nikolaidis "Facial feature extraction and pose determination" Pattern Recognit. (2000) 10.1016/s0031-3203(99)00176-4

[9]

Kulikajevas "Detection of sitting posture using hierarchical image composition and deep learning" PeerJ Comput. Sci. (2021) 10.7717/peerj-cs.442

[10]

Ryselis "Multiple Kinect based system to monitor and analyze key performance indicators of physical training" Hum.-Centric Comput. Inf. Sci. (2020) 10.1186/s13673-020-00256-4

[11]

(2020). An ANN-based gesture recognition algorithm for smart-home applications. KSII Trans. Internet Inf. Syst., 14, 1967–1983.

[12]

Abraham, L., Urru, A., Normani, N., Wilk, M., Walsh, M., and O’Flynn, B. (2018). Hand Tracking and Gesture Recognition Using Lensless Smart Sensors. Sensors, 18. 10.3390/s18092834

[13]

Ahmed, S., and Cho, S.H. (2020). Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier. Sensors, 20. 10.3390/s20020564

[14]

Alkemade "On the Efficiency of a VR Hand Gesture-Based Interface for 3D Object Manipulations in Conceptual Design" Int. J. Hum.–Comput. Interact. (2017) 10.1080/10447318.2017.1296074

[15]

Lee "Immersive Gesture Interfaces for Navigation of 3D Maps in HMD-Based Mobile Virtual Environments" Mob. Inf. Syst. (2018)

[16]

Lee, D.H., and Hong, K.S. (December, January 30). Game interface using hand gesture recognition. Proceedings of the 5th International Conference on Computer Sciences and Convergence Information Technology, Seoul, Republic of Korea.

[17]

Negin "PRAXIS: Towards automatic cognitive assessment using gesture recognition" Expert Syst. Appl. (2018) 10.1016/j.eswa.2018.03.063

[18]

Del Rio Guerra, M.S., Martin-Gutierrez, J., Acevedo, R., and Salinas, S. (2019). Hand Gestures in Virtual and Augmented 3D Environments for Down Syndrome Users. Appl. Sci., 9. 10.3390/app9132641

[19]

Kaczmarek, W., Panasiuk, J., Borys, S., and Banach, P. (2020). Industrial Robot Control by Means of Gestures and Voice Commands in Off-Line and On-Line Mode. Sensors, 20. 10.20944/preprints202010.0455.v1

[20]

Neto "Gesture-based human-robot interaction for human assistance in manufacturing" Int. J. Adv. Manuf. Technol. (2018) 10.1007/s00170-018-2788-x

[21]

Young "Designing Mid-Air Haptic Gesture Controlled User Interfaces for Cars" Proc. ACM Hum.-Comput. Interact. (2020) 10.1145/3397869

[22]

Yu "A novel hand gesture recognition method based on 2-channel sEMG" Technol. Health Care (2018) 10.3233/thc-174567

[23]

Object Detection Algorithm Based on Improved YOLOv3

Liquan Zhao, Shi-Feng Li

Electronics 10.3390/electronics9030537

[24]

Kulikajevas, A., Maskeliūnas, R., Damaševičius, R., and Ho, E.S.L. (2020). 3D Object Reconstruction from Imperfect Depth Data Using Extended YOLOv3 Network. Sensors, 20. 10.3390/s20072025

[25]

Neto, P., Pereira, D., Pires, J.N., and Moreira, A.P. (2013, January 6–10). Real-time and continuous hand gesture spotting: An approach based on artificial neural networks. Proceedings of the 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany. 10.1109/icra.2013.6630573

[26]

Mujahid, A., Awan, M.J., Yasin, A., Mohammed, M.A., Damaševičius, R., Maskeliūnas, R., and Abdulkareem, K.H. (2021). Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model. Appl. Sci., 11. 10.3390/app11094164

[27]

Strezoski, G., Stojanovski, D., Dimitrovski, I., and Madjarov, G. (2017). Hand Gesture Recognition Using Deep Convolutional Neural Networks. ICT Innovations 2016, Springer International Publishing. 10.1007/978-3-319-68855-8_5

[28]

Gao, X., Zhang, J., and Wei, Z. (2018, January 27–29). Deep learning for sequence pattern recognition. Proceedings of the 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC), Zhuhai, China. 10.1109/icnsc.2018.8361281

[29]

Elmezain, M. (2010). Hand Gesture Spotting and Recognition Using HMM and CRF in Color Image Sequences. [Ph.D. Thesis, Otto-von-Guericke-Universitaet]. 10.1109/icip.2009.5414322

[30]

Elmezain "A Robust Method for Hand Tracking Using Mean-shift Algorithm and Kalman Filter in Stereo Color Image Sequences. World Academy of Science, Engineering and Technology, Open Science Index 35" Int. J. Electron. Commun. Eng. (2009)

[31]

Elmezain "A Novel System for Automatic Hand Gesture Spotting and Recognition in Stereo Color Image Sequences" J. WSCG (2009)

[32]

Elmezain, M., Al-Hamadi, A., and Michaelis, B. (2010, January 23–26). A Robust Method for Hand Gesture Segmentation and Recognition Using Forward Spotting Scheme in Conditional Random Fields. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey. 10.1109/icpr.2010.938

[33]

Elmezain, M., Al-Hamadi, A., Sadek, S., and Michaelis, B. (2010, January 15–18). Robust methods for hand gesture spotting and recognition using Hidden Markov Models and Conditional Random Fields. Proceedings of the 10th IEEE International Symposium on Signal Processing and Information Technology, Luxor, Egypt. 10.1109/isspit.2010.5711749

Metrics

5

Citations

33

References

Details

Published: Dec 28, 2022
Vol/Issue: 10(1)
Pages: 1
License: View

Authors

M

Mahmoud Elmezain

Computer Science Department, Faculty of Science, Tanta University, Tanta 31527, Egypt; College of Computer Science and Engineering, Taibah University, Yanbu 966144, Saudi Arabia

M

Majed M. Alwateer

College of Computer Science and Engineering, Taibah University, Yanbu 966144, Saudi Arabia

R

Rasha El-Agamy

Computer Science Department, Faculty of Science, Tanta University, Tanta 31527, Egypt; College of Computer Science and Engineering, Taibah University, Yanbu 966144, Saudi Arabia

E

Elsayed Atlam

Computer Science Department, Faculty of Science, Tanta University, Tanta 31527, Egypt; College of Computer Science and Engineering, Taibah University, Yanbu 966144, Saudi Arabia

H

Hani M. Ibrahim

Mathematics & Computer Science Department, Faculty of Science, Menoufiya University, Menoufia 32511, Egypt

Cite This Article

Mahmoud Elmezain, Majed M. Alwateer, Rasha El-Agamy, et al. (2022). Forward Hand Gesture Spotting and Prediction Using HMM-DNN Model. Informatics, 10(1), 1. https://doi.org/10.3390/informatics10010001

Forward Hand Gesture Spotting and Prediction Using HMM-DNN Model

You May Also Like