Teaching Chinese sign language with a smartphone

Abstract

Background

There is a large group of deaf-mutes worldwide, and sign language is a major communication tool in this community. It is necessary for deaf-mutes to be able to communicate with others who are capable of hearing, and hearing people also need to understand sign language, which produces a great demand for sign language tuition. Even though there have already been a large number of books written about sign language, it is inefficient to learn sign language through reading alone, and the same can be said on watching videos. To solve this problem, we developed a smartphone-based interactive Chinese sign language teaching system that facilitates sign language learning.

Methods

The system provides a learner with some learning modes and captures the learner's actions using the front camera of the smartphone. At present, the system provides a vocabulary set with 1000 frequently used words, and the learner can evaluate his/her sign action by subjective or objective comparison. In the mode of word recognition, the users can play any word within the vocabulary, and the system will return the top three retrieved candidates; thus, it can remind the learners what the sign is.

Results

This system provides interactive learning to enable a user to efficiently learn sign language. The system adopts an algorithm based on point cloud recognition to evaluate a user's sign and costs about 700ms of inference time for each sample, which meets the real-time requirements.

Conclusion

This interactive learning system decreases the communication barriers between deaf-mutes and hearing people.

Keyword

Computer assisted learning ; Sign language evaluation ; Smartphone platform

Cite this article

Yanxiao ZHANG, Yuecong MIN, Xilin CHEN. Teaching Chinese sign language with a smartphone. Virtual Reality & Intelligent Hardware, 2021, 3(3): 248-260 DOI:10.1016/j.vrih.2021.05.004

References

1. Gong C, Bin C, Lei Z. Study on the prevention and strategy of disability in China. Procedia-Social and Behavioral Sciences, 2010, 2(5): 6906–6913 DOI:10.1016/j.sbspro.2010.05.041

2. Lu W. Research and suggestions on sign language teaching in Chinese universities. Journal of Changchun University, 2014

3. Leo M, Medioni G, Trivedi M, Kanade T, Farinella G M. Computer vision for assistive technologies. Computer Vision and Image Understanding, 2017, 154: 1–15 DOI:10.1016/j.cviu.2016.09.001

4. Lee Y H, Medioni G. RGB-D camera based wearable navigation system for the visually impaired. Computer Vision and Image Understanding, 2016, 149: 3–20 DOI:10.1016/j.cviu.2016.03.019

5. Kayukawa S, Higuchi K, Guerreiro J, Morishima S, Sato Y, Kitani K, Asakawa C. BBeep: a sonic collision avoidance system for blind travellers and nearby pedestrians. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow Scotland Uk, New York, NY, USA, ACM, 2019, 1–12 DOI:10.1145/3290605.3300282

6. Zeng J H, Sun Y R, Wang F. A natural hand gesture system for intelligent human-computer interaction and medical assistance. In: 2012 Third Global Congress on Intelligent Systems. Wuhan, China, IEEE, 2012, 382–385 DOI:10.1109/gcis.2012.60

7. Parmar P, Morris B T. Measuring the quality of exercises. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando, FL, USA, IEEE, 2016, 2241–2244 DOI:10.1109/embc.2016.7591175

8. Feng W G, Liu R, Zhu M. Fall detection for elderly person care in a vision-based home surveillance environment using a monocular camera. Signal, Image and Video Processing, 2014, 8(6): 1129–1138 DOI:10.1007/s11760-014-0645-4

9. Martinel N, Piciarelli C, Micheloni C. A supervised extreme learning committee for food recognition. Computer Vision and Image Understanding, 2016, 148: 67–86 DOI:10.1016/j.cviu.2016.01.012

10. Zhang S. Design research in Kinect based sign-language teaching system. East China Normal University. 2014

11. Juan-les-Pins, France, York New, Press ACM, 2002, 137–145 DOI:10.1145/641007.641035

12. Phan H D, Ellis K, Dorin A. MIC, an interactive sign language teaching system. In: Proceedings of the 30th Australian Conference on Computer-Human Interaction. Melbourne Australia, New York, NY, USA, ACM, 2018, 544–547 DOI:10.1145/3292147.3292237

13. IEEE transactions on pattern analysis and machine intelligence-cover. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(5): 0_1–673 DOI:10.1109/tpami.2004.1273910

14. Jiang X W, Satapathy S C, Yang L X, Wang S H, Zhang Y D. A survey on artificial intelligence in Chinese sign language recognition. Arabian Journal for Science and Engineering, 2020, 45(12): 9859–9894 DOI:10.1007/s13369-020-04758-2

15. Pigou L, Dieleman S, Kindermans P J, Schrauwen B. Sign language recognition using convolutional neural networks. European Conference on Computer Vision. Springer, Cham, 2014, 572–578 DOI:10.1007/978-3-319-16178-5_40

16. Köpüklü O, Köse N, Rigoll G. Motion fused frames: data level fusion strategy for hand gesture recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Salt Lake City, UT, USA, IEEE, 2018, 2184–21848 DOI:10.1109/cvprw.2018.00284

17. IEEE transactions on pattern analysis and machine intelligence information for authors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(6): C3 DOI:10.1109/tpami.2015.2427753

18. Li Y N, Miao Q G, Tian K, Fan Y Y, Xu X, Li R, Song J F. Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model. In: 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, Mexico, IEEE, 2016, 25–30 DOI:10.1109/icpr.2016.7899602

19. Li Y N, Miao Q G, Tian K, Fan Y Y, Xu X, Li R, Song J F. Large-scale gesture recognition with a fusion of RGB-D data based on saliency theory and C3D model. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(10): 2956–2964 DOI:10.1109/tcsvt.2017.2749509

20. Huang J, Zhou W G, Li H Q, Li W P. Attention-based 3D-CNNs for large-vocabulary sign language recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(9): 2822–2832 DOI:10.1109/tcsvt.2018.2870740

21. Cate H, Dalvi F, Hussain Z. Sign language recognition using temporal classification. 2017

22. Chai X J, Liu Z P, Yin F, Liu Z, Chen X L. Two streams recurrent neural networks for large-scale continuous gesture recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, Mexico, IEEE, 2016, 31–36 DOI:10.1109/icpr.2016.7899603

23. Bantupalli K, Xie Y. American sign language recognition using deep learning and computer vision. In: 2018 IEEE International Conference on Big Data (Big Data). Seattle, WA, USA, IEEE, 2018, 4896–4899 DOI:10.1109/bigdata.2018.8622141

24. Liao Y Q, Xiong P W, Min W D, Min W Q, Lu J H. Dynamic sign language recognition based on video sequence with BLSTM-3D residual networks. IEEE Access, 2019, 7: 38044–38054 DOI:10.1109/access.2019.2904749

25. General Administration of Quality Supervision, Inspection and Quarantine of the People's Republic of China. Basic Chinese Sign Language Gestures. China Standards Press, 2009

26. Min Y, Chai X, Zhao L, Chen X. FlickerNet: Adaptive 3D gesture recognition from sparse point clouds. British Machine Vision Conference. 2019, 105

27. Sun Y, Zheng L, Yang Y, Tian Q, Wang S. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). Proceedings of the European Conference on Computer Vision. 2018, 480–496 DOI:10.1007/978-3-030-01225-0_30

28. SunY F, Zheng L, Yang Y, Tian Q, Wang S J. Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). Computer Vision-ECCV 2018, 2018, 480–496 DOI:10.1007/978-3-030-01225-0_30

29. de Smedt Q, Wannous H, -P Vandeborre J, Guerry J, le Saux B, Filliat D. 3D hand gesture recognition using a depth and skeletal dataset: SHREC'17 track. 3Dor '17: Proceedings of the Workshop on 3D Object Retrieval. 2017, 33–38 DOI:10.2312/3dor.20171049

30. Min Y C, Zhang Y X, Chai X J, Chen X L. An efficient PointLSTM for point clouds based gesture recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA, IEEE, 2020, 5760–5769 DOI:10.1109/cvpr42600.2020.00580

31. De Smedt Q, Wannous H, Vandeborre J P. Skeleton-based dynamic hand gesture recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Las Vegas, NV, USA, IEEE, 2016, 1206–1214 DOI:10.1109/cvprw.2016.153

32. Hou J X, Wang G J, Chen X H, Xue J H, Zhu R, Yang H Z. Spatial-temporal attention res-TCN for skeleton-based dynamic hand gesture recognition. Computer Vision-ECCV 2018 Workshops, 2019 DOI:10.1007/978-3-030-11024-6_18

33. Peng W, Hong X P, Chen H Y, Zhao G Y. Learning graph convolutional network for skeleton-based human action recognition by neural searching. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(3): 2669–2676 DOI:10.1609/aaai.v34i03.5652

34. Chen Y X, Zhao L, Peng X, Yuan J B, Metaxas D N. Construct dynamic graphs for hand gesture recognition via spatial-temporal attention. 2019

35. Luo Y. Love Sign Language (Version 1.3.5), 2020

36. Changsha Keenbow Information Technology Co., Ltd. 2020, Keenbow Sign Language (Version 1.4)

Media

Teaching Chinese sign language with a sma...