Sign Language Recognition Using Residual Network Architectures for Alphabet And Diagraph Classification
DOI:
https://doi.org/10.33736/jcsi.7986.2025Keywords:
Alphabet Sign, Diagraph Sign, ResNet, Hearing-impaired, Sign Language Recognition, Support Vector MachineAbstract
Communication is crucial in human life, enabling the exchange of information through various methods beyond spoken language. Sign language translation is crucial for bridging communication gaps between hearing-impaired and hearing individuals, promoting effective interaction and understanding. This study presents a comprehensive model for identifying alphabet and digraph signs using feature extraction techniques from ResNet architectures, specifically ResNet18, ResNet50, and ResNet101. The system was designed to integrate both hand gestures and facial expressions, enhancing the accuracy of sign language recognition. Classification of sign language images into alphabet and diagraph categories was assessed using Support Vector Machine (SVM). The resulting classification accuracies were 61.7% for ResNet18, 64.5% for ResNet50, and 66.5% for ResNet101. The research results emphasize how deeper ResNet models are effective in improving recognition accuracy. This proposed model has significant implications for educational applications as it addresses attention-related challenges and aims to enhance student engagement in learning processes, thereby contributing to developing more inclusive educational environments.
References
Akansha Tyagi, S. B. (2022). Hybrid FiST_CNN Approach for Feature Extraction for Vision-Based Indian Sign Language Recognition. The International Arab Journal of Information Technology (IAJIT), 19(03), 403-411. https://doi.org/10.34028/iajit/19/3/15
Aksoy, B., Salman, O. K. M., & Ekrem, Ö. (2021). Detection of Turkish Sign Language Using Deep Learning and Image Processing Methods. Applied Artificial Intelligence, 35(12), 952–981. https://doi.org/10.1080/08839514.2021.1982184
Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M., Alrayes, T., Mathkour, H., & Mekhtiche, M. (2020). Deep Learning-Based Approach for Sign Language Gesture Recognition with Efficient Hand Gesture Representation. IEEE Access, 8, 192527–192542. https://doi.org/10.1109/ACCESS.2020.3032140
Asonye, E., Emma-Asonye, E., & Edward, M. (2018). Deaf in Nigeria: A Preliminary Survey of Isolated Deaf Communities. SAGE Open, 8, 215824401878653. https://doi.org/10.1177/2158244018786538
Blench, R., Warren, A., & Dendo, M. (2006). An unreported African Sign Language for the Deaf among the Bura in Northeast Nigeria.
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudrealt, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., Vogler, C., & Morris, M. (2019). Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective.
Chao, H., Fenhua, W., & Ran, Z. (2019). Sign Language Recognition Based on CBAM-ResNet. Proceedings of the 2019 International Conference on Artificial Intelligence and Advanced Manufacturing, 1–6. https://doi.org/10.1145/3358331.3358379
Chowdhury, A. R., Biswas, A., Hasan, S., Rahman, T. M., & Uddin, J. (2017). Bengali Sign language to text conversion using artificial neural network and support vector machine. 2017 3rd International Conference on Electrical Information and Communication Technology (EICT), 1–4. https://doi.org/10.1109/EICT.2017.8275248
Côté Allard, U., Fall, C. L., Drouin, A., Campeau-Lecours, A., Gosselin, C., Glette, K., Laviolette, F., & Gosselin, B. (2019). Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society, PP. https://doi.org/10.1109/TNSRE.2019.2896269
Daisie. (2023, June 21). Diagraphs Explained: Comprehensive Phonics Guide. Daisie Blog. https://blog.daisie.com/what-is-a-diagraph-a-comprehensive-guide-to-understanding-and-using-diagraphs-in-phonics/
Gupta, A. K., & Singh, S. (2024). Hand Gesture Recognition System Based on Indian Sign Language Using SVM and CNN. International Journal of Image and Graphics, 2650008. https://doi.org/10.1142/S0219467826500087
Haria, A., Subramanian, A., Asokkumar, N., Poddar, S., & Nayak, J. (2017). Hand Gesture Recognition for Human Computer Interaction. Procedia Computer Science, 115, 367–374. https://doi.org/10.1016/j.procs.2017.09.092
Hasanah, S. A., Pravitasari, A. A., Abdullah, A. S., Yulita, I. N., & Asnawi, M. H. (2023). A Deep Learning Review of ResNet Architecture for Lung Disease Identification in CXR Image. Applied Sciences, 13(24), Article 24. https://doi.org/10.3390/app132413111
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Presented at the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA. doi:10.1109/cvpr.2016.90
Irhebhude, M. E., Kolawole, A.O., & Abubakar, H. (2023). DIAGRAPH SIGN LANGUAGE RECOGNITION USING RESIDUAL NETWORK AND SUPPORT VECTOR MACHINE. International Conference on Communication and E-Systems for Economic Stability | CeSES’ 2023. Retrieved May 9, 2024
Irhebhude, M., Kolawole, A., & Goshit, N. (2023). Perspective on Dark-Skinned Emotion Recognition Using Deep-Learned and Handcrafted Feature Techniques (pp. 1–24). https://doi.org/10.5772/intechopen.109739
Jain, V., Jain, A., Chauhan, A., Kotla, S. S., & Gautam, A. (2021). American Sign Language recognition using Support Vector Machine and Convolutional Neural Network. International Journal of Information Technology, 13, 1193–1200. https://doi.org/10.1007/s41870-021-00617-x
Jiang, X., & Zhu, Z. (2019). Chinese Sign Language Identification via Wavelet Entropy and Support Vector Machine. 726–736. https://doi.org/10.1007/978-3-030-35231-8_53
Kashef, R. (2021). A boosted SVM classifier trained by incremental learning and decremental unlearning approach. Expert Systems with Applications, 167, 114154. https://doi.org/10.1016/j.eswa.2020.114154
Kothadiya, D., Bhatt, C., Sapariya, K., Patel, K., Gil-González, A.-B., & Corchado, J. M. (2022). Deepsign: Sign Language Detection and Recognition Using Deep Learning. Electronics, 11(11), 1780. https://doi.org/10.3390/electronics11111780
Liao, Y., Xiong, P., Min, W., Min, W., & Lu, J. (2019). Dynamic Sign Language Recognition Based on Video Sequence with BLSTM-3D Residual Networks. IEEE Access, 7, 38044–38054. https://doi.org/10.1109/ACCESS.2019.2904749
Lin, K., Zhao, Y., Gao, X., Zhang, M., Zhao, C., Peng, L., Zhang, Q., & Zhou, T. (2022). Applying a deep residual network coupling with transfer learning for recyclable waste sorting. Environmental Science and Pollution Research, 29(60), 91081–91095. https://doi.org/10.1007/s11356-022-22167-w
Ma, R., Zhang, Z., & Chen, E. (2021). Human Motion Gesture Recognition Based on Computer Vision. Complexity, 2021, 1–11. https://doi.org/10.1155/2021/6679746
Morgan, R. Z. (2002). Maganar Hannu: Language of the Hands: A Descriptive Analysis of Hausa Sign Language (review). Sign Language Studies, 2(3), 335–341. https://doi.org/10.1353/sls.2002.0011
Olabanji, A., & Ponnle, A. (2021). Development of A Computer Aided Real-Time Interpretation System for Indigenous Sign Language in Nigeria Using Convolutional Neural Network. European Journal of Electrical Engineering and Computer Science, 5, 68–74. https://doi.org/10.24018/ejece.2021.5.3.332
Oyedotun, O. K., Ismaeil, K. A., & Aouada, D. (2021). Training very deep neural networks: Rethinking the role of skip connections. Neurocomputing, 441, 105–117. https://doi.org/10.1016/j.neucom.2021.02.004
Sahoo, J., Ari, S., & Patra, S. (2021). A user independent hand gesture recognition system using deep CNN feature fusion and machine learning technique (pp. 189–207). https://doi.org/10.1016/B978-0-12-822133-4.00011-6
Sharma, S., & Singh, S. (2020). Vision-based sign language recognition system: A Comprehensive Review. 2020 International Conference on Inventive Computation Technologies (ICICT), 140–144. https://doi.org/10.1109/ICICT48043.2020.9112409
Shi, X., Jiao, X., Meng, C., & Bian, Z. (2022). 3D Sign language recognition based on multi-path hybrid residual neural network. 2022 14th International Conference on Machine Learning and Computing (ICMLC). https://doi.org/10.1145/3529836.3529943
Sreemathy, R., Turuk, M., Chaudhary, S., Lavate, K., Ushire, A., & Khurana, S. (2023). Continuous word level sign language recognition using an expert system based on machine learning. International Journal of Cognitive Computing in Engineering, 4, 170–178. https://doi.org/10.1016/j.ijcce.2023.04.002
Venugopalan, A., & Reghunadhan, R. (2023). Applying Hybrid Deep Neural Network for the Recognition of Sign Language Words Used by the Deaf COVID-19 Patients. Arabian Journal for Science and Engineering, 48(2), 1349–1362. https://doi.org/10.1007/s13369-022-06843-0
Wang, J., Wang, X., Li, X., & Yi, J. (2023). A Hybrid Particle Swarm Optimization Algorithm with Dynamic Adjustment of Inertia Weight Based on a New Feature Selection Method to Optimize SVM Parameters. Entropy, 25(3), 531. https://doi.org/10.3390/e25030531
Wen, F., Zhang, Z., He, T., & Lee, C. (2021). AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nature Communications, 12(1), 5378. https://doi.org/10.1038/s41467-021-25637-w
Xu, Y., Yang, W., Wu, X., Wang, Y., & Zhang, J. (2022). ResNet Model Automatically Extracts and Identifies FT-NIR Features for Geographical Traceability of Polygonatum kingianum. Foods, 11(22), 3568. https://doi.org/10.3390/foods11223568
Zhang, W., Quan, H., Gandhi, O., Rajagopal, R., Tan, C.-W., & Srinivasan, D. (2020). Improving Probabilistic Load Forecasting Using Quantile Regression NN with Skip Connections. IEEE Transactions on Smart Grid, 11(6), 5442–5450. https://doi.org/10.1109/TSG.2020.2995777
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Computing and Social Informatics
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright Transfer Statement for Journal
1) In signing this statement, the author(s) grant UNIMAS Publisher an exclusive license to publish their original research papers. The author(s) also grant UNIMAS Publisher permission to reproduce, recreate, translate, extract or summarise, and to distribute and display in any forms, formats, and media. The author(s) can reuse their papers in their future printed work without first requiring permission from UNIMAS Publisher, provided that the author(s) acknowledge and reference publication in the Journal.
2) For open access articles, the author(s) agree that their articles published under UNIMAS Publisher are distributed under the terms of the CC-BY-NC-SA (Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License) which permits unrestricted use, distribution, and reproduction in any medium, for non-commercial purposes, provided the original work of the author(s) is properly cited.
3) For subscription articles, the author(s) agree that UNIMAS Publisher holds copyright, or an exclusive license to publish. Readers or users may view, download, print, and copy the content, for academic purposes, subject to the following conditions of use: (a) any reuse of materials is subject to permission from UNIMAS Publisher; (b) archived materials may only be used for academic research; (c) archived materials may not be used for commercial purposes, which include but not limited to monetary compensation by means of sale, resale, license, transfer of copyright, loan, etc.; and (d) archived materials may not be re-published in any part, either in print or online.
4) The author(s) is/are responsible to ensure his or her or their submitted work is original and does not infringe any existing copyright, trademark, patent, statutory right, or propriety right of others. Corresponding author(s) has (have) obtained permission from all co-authors prior to submission to the journal. Upon submission of the manuscript, the author(s) agree that no similar work has been or will be submitted or published elsewhere in any language. If submitted manuscript includes materials from others, the authors have obtained the permission from the copyright owners.
5) In signing this statement, the author(s) declare(s) that the researches in which they have conducted are in compliance with the current laws of the respective country and UNIMAS Journal Publication Ethics Policy. Any experimentation or research involving human or the use of animal samples must obtain approval from Human or Animal Ethics Committee in their respective institutions. The author(s) agree and understand that UNIMAS Publisher is not responsible for any compensational claims or failure caused by the author(s) in fulfilling the above-mentioned requirements. The author(s) must accept the responsibility for releasing their materials upon request by Chief Editor or UNIMAS Publisher.
6) The author(s) should have participated sufficiently in the work and ensured the appropriateness of the content of the article. The author(s) should also agree that he or she has no commercial attachments (e.g. patent or license arrangement, equity interest, consultancies, etc.) that might pose any conflict of interest with the submitted manuscript. The author(s) also agree to make any relevant materials and data available upon request by the editor or UNIMAS Publisher.
