Affine transformation of virtual 3D object using 2D localization of fingertips

Abstract

Background

Interactions with virtual 3D objects in the virtual reality (VR) environment using the gesture of fingers captured in a wearable 2D camera have emerging applications in real-life.

Method

This paper presents an approach of a two-stage convolutional neural network, one for the detection of hand and another for the fingertips. One purpose of VR environments is to transform a virtual 3D object with affine parameters by using the gesture of thumb and index fingers.

Results

To evaluate the performance of the proposed system, one existing, and another developed egocentric fingertip databases are employed so that learning involves large variations that are common in real-life. Experimental results show that the proposed fingertip detection system outperforms the existing systems in terms of the precision of detection.

Conclusion

The interaction performance of the proposed system in the VR environment is higher than that of the existing systems in terms of estimation error and correlation between the ground truth and estimated affine parameters.

Keyword

Affine transformation ; Detection of fingertips ; Detection of hand ; Human-computer interaction ; Virtual reality

Cite this article

Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN. Affine transformation of virtual 3D object using 2D localization of fingertips. Virtual Reality & Intelligent Hardware, 2020, 2(6): 534-555 DOI:10.1016/j.vrih.2020.10.001

References

1. Burdea G C, Coiffet P. Virtual reality technology. John Wiley & Sons, 2003

2. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Computer Graphics and Applications, 2001, 21(6): 34–47 DOI:10.1109/38.963459

3. Milgram P, Kishino F. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 1994, 12: 1321–1329

4. Canessa A, Chessa M, Gibaldi A, Sabatini S P, Solari F. Calibrated depth and color cameras for accurate 3D interaction in a stereoscopic augmented reality environment. Journal of Visual Communication and Image Representation, 2014, 25(1): 227–237 DOI:10.1016/j.jvcir.2013.02.011

5. Bernardes J. Comparing a mouse and a free hand gesture interaction technique for 3D object manipulation. In: Lecture Notes in Computer Science. Cham: Springer International Publishing, 2020, 19–37 DOI:10.1007/978-3-030-49062-1_2

6. Feng Z Y, Xu S J, Zhang X, Jin L W, Ye Z C, Yang W X. Real-time fingertip tracking and detection using Kinect depth sensor for a new writing-in-the air system. In: Proceedings of the 4th International Conference on Internet Multimedia Computing and Service-ICIMCS'12. Wuhan, China, NewYork, Press ACM, 2012, 70–74 DOI:10.1145/2382336.2382356

7. Han J G, Shao L, Xu D, Shotton J. Enhanced computer vision with microsoft kinect sensor: a review. IEEE Transactions on Cybernetics, 2013, 43(5): 1318–1334 DOI:10.1109/tcyb.2013.2265378

8. Nai W Z, Liu Y, Rempel D, Wang Y T. Fast hand posture classification using depth features extracted from random line segments. Pattern Recognition, 2017, 65: 1–10 DOI:10.1016/j.patcog.2016.11.022

9. Kang S K, Nam M Y, Rhee P K. Color based hand and finger detection technology for user interaction. In: 2008 International Conference on Convergence and Hybrid Information Technology. Daejeon, South Korea, IEEE, 2008, 229–236

10. Gurav R M, Kadbe P K. Real time finger tracking and contour detection for gesture recognition using OpenCV. 2015

11. Bhuyan M K, MacDorman K F, Kar M K, Neog D R, Lovell B C, Gadde P. Hand pose recognition from monocular images by geometrical and texture analysis. Journal of Visual Languages & Computing, 2015, 28: 39–55 DOI:10.1016/j.jvlc.2014.12.001

12. Stergiopoulou E, Papamarkos N. Hand gesture recognition using a neural network shape fitting technique. Engineering Applications of Artificial Intelligence, 2009, 22(8): 1141–1158 DOI:10.1016/j.engappai.2009.03.008

13. RaySarkar A, Sanyal G, Majumder S. Hand gesture recognition systems: a survey. International Journal of Computer Applications, 2013, 71(15): 25–37 DOI:10.5120/12435-9123

14. Huang Y, Liu X, Jin L, Zhang X. Deepfinger: A cascade convolutional neuron network approach to finger key point detection in egocentric vision with mobile camera. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics. Kowloon, China, IEEE, 2015, 2944–2949

15. Liu X R, Huang Y C, Zhang X, Jin L W. Fingertip in the eye: an attention-based method for real-time hand tracking and fingertip detection in egocentric videos. In: Communications in Computer and Information Science. Singapore: Springer Singapore, 2016, 145–154 DOI:10.1007/978-981-10-3002-4_12

16. Wu W, Li C, Cheng Z, Zhang X, Jin L. Yolse: Egocentric fingertip detection from single RGB images. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Venice, Italy, IEEE, 2017, 623–630

17. Jain V, Hebbalaguppe R. AirPen: a touchless fingertip based gestural interface for smartphones and head-mounted devices. 2019

18. Huang Y, Liu X, Zhang X, Jin L. A pointing gesture based egocentric interaction system: Dataset, approach and application. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. Las Vegas, NV, USA, IEEE, 2016, 16–23

19. Poupyrev I, Billinghurst M, Weghorst S, Ichikawa T. The go-go interaction technique: non-linear mapping for direct manipulation in VR. In: Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology. Seattle, Washington, USA, New York, ACM Press, 1996, 79–80 DOI:10.1145/237091.237102

20. Bowman D A, Hodges L F. An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In: Proceedings of the symposium on Interactive 3D graphics. Rhode Island, USA, 1997 DOI:10.1145/253284.253301

21. Tomozoe Y, Machida T, Kiyokawa K, Takemura H. Unified gesture-based interaction techniques for object manipulation and navigation in a large-scale virtual environment. In: Proceedings of the IEEE Virtual Reality 2004. Chicago, IL, USA, IEEE, 2004, 259–260

22. Kiyokawa K, Takemura H. A tunnel window and its variations: Seamless teleportation techniques in a virtual environment. In: Proceedings of the HCI International. Citeseer, Las Vegas, Nevada, USA, 2005

23. Lee T, Hollerer T. Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In: 2007 11th IEEE International Symposium on Wearable Computers. Boston, MA, USA, IEEE, 2007, 83–90 DOI:10.1109/iswc.2007.4373785

24. Rani S S, Dhrisya K, Ahalyadas M. Hand gesture control of virtual object in augmented reality. In: Proceedings of the International Conference Advances in Computing, Communications and Informatics. Udupi, India, IEEE, 2017, 1500–1505

25. Bai H, Gao L, El-Sana J, Billinghurst M. Markerless 3D gesture-based interaction for handheld augmented reality interfaces. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Adelaide, SA, Australia, IEEE, 2013, 1–6

26. Song P, Yu H, Winkler S. Vision-based 3D finger interactions for mixed reality games with physics simulation. In: Proceedings of The 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry. Singapore, New York, ACM Press, 2008, 7 DOI:10.1145/1477862.1477871

27. Ong S K, Wang Z B. Augmented assembly technologies based on 3D bare-hand interaction. CIRP Annals, 2011, 60(1): 1–4 DOI:10.1016/j.cirp.2011.03.001

28. Le H Q, Kim J I. An augmented reality application with hand gestures for learning 3D geometry. In: 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). Jeju, South Korea, IEEE, 2017, 34–41 DOI:10.1109/BIGCOMP.2017.7881712

29. Weichel C, Lau M, Kim D, Villar N, Gellersen H W. MixFab: a mixed-reality environment for personal fabrication. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. Toronto, Ontario, Canada, New York, ACM Press, 2014, 3855–3864 DOI:10.1145/2556288.2557090

30. Lee J Y, Rhee G W, Seo D W. Hand gesture-based tangible interactions for manipulating virtual objects in a mixed reality environment. The International Journal of Advanced Manufacturing Technology, 2010, 51(9/10/11/12): 1069–1082 DOI:10.1007/s00170-010-2671-x

31. Alam M M, Rahman S M. Detection and tracking of fingertips for geometric transformation of objects in virtual environment. In: 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA). Abu Dhabi, United Arab Emirates, IEEE, 2019, 1–8 DOI:10.1109/AICCSA47632.2019.9035256

32. Wu M Y, Ting P W, Tang Y H, Chou E T, Fu L C. Hand pose estimation in object-interaction based on deep learning for virtual reality applications. Journal of Visual Communication and Image Representation, 2020, 70: 102802 DOI:10.1016/j.jvcir.2020.102802

33. Redmon J, Farhadi A. YOLO9000: better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017, 7263–7271

34. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014

35. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA, IEEE, 2016, 2818–2826

36. Chollet F. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA, IEEE, 2017, 1251–1258

37. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L C. Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT, USA, IEEE, 2018, 4510–4520

38. Kingma D P, Ba J. Adam: a method for stochastic optimization. 2014

39. Anton H, Rorres C. Elementary linear algebra: applications version. John Wiley & Sons, 2013

1. Chongyang SUN, Weizhi NAI, Xiaoying SUN, Tactile sensitivity in ultrasonic haptics: Do different parts of hand and different rendering methods have an impact on perceptual threshold? Virtual Reality & Intelligent Hardware 2019, 1(3): 265-275

2. Egemen ERTUGRUL, Ping LI, Bin SHENG, On attaining user-friendly hand gesture interfaces to control existing GUIs Virtual Reality & Intelligent Hardware 2020, 2(2): 153-161

3. Yuanyuan SHI, Yunan LI, Xiaolong FU, Kaibin MIAO, Qiguang MIAO, Review of dynamic gesture recognition Virtual Reality & Intelligent Hardware 2021, 3(3): 183-206

4. Wanlu ZHENG, Wenming ZHENG, Yuan ZONG, Multi-scale discrepancy adversarial network for cross-corpus speech emotion recognition Virtual Reality & Intelligent Hardware 2021, 3(1): 65-75

5. Xuezhi YAN, Qiushuang WU, Xiaoying SUN, Electrostatic tactile representation in multimedia mobile terminal Virtual Reality & Intelligent Hardware 2019, 1(2): 201-218

6. Yang LI, Dong WU, Jin HUANG, Feng TIAN, Hong'an WANG, Guozhong DAI, Influence of multi-modality on moving target selection in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(3): 303-315

7. Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI, Gesture interaction in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 84-112

8. Athirah SYAMIMI, Yiwei GONG, Ryan LIEW, VR industrial applications―A singapore perspective Virtual Reality & Intelligent Hardware 2020, 2(5): 409-420

9. Jie GUO, Dongdong WENG, Yue LIU, Qiyong CHEN, Yongtian WANG, Analysis of teenagers' preferences and concerns regarding HMDs in education Virtual Reality & Intelligent Hardware 2021, 3(5): 369-382

10. Susu HUANG, Daqing QI, Jiabin YUAN, Huawei TU, Review of studies on target acquisition in virtual reality based on the crossing paradigm Virtual Reality & Intelligent Hardware 2019, 1(3): 251-264

11. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

12. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

13. Yuan CHANG, Guo-Ping WANG, A review on image-based rendering Virtual Reality & Intelligent Hardware 2019, 1(1): 39-54

14. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

15. Xu PENG, Zhenyu GAO, Yitong DING, Dongfeng ZHAO, Xiaoyu CHI, Study of ghost image suppression in polarized catadioptric virtual reality optical systems Virtual Reality & Intelligent Hardware 2020, 2(1): 70-78

16. Zhiming HU, Sheng LI, Meng GAI, Temporal continuity of visual attention for future gaze prediction in immersive virtual reality Virtual Reality & Intelligent Hardware 2020, 2(2): 142-152

17. Lihui HUANG, Siti Faatihah Binte Mohd TAIB, Ryan Aung BA, Zhe An GOH, Mengshan XU, Virtual reality research and development in NTU Virtual Reality & Intelligent Hardware 2020, 2(5): 394-408

18. Stéphanie PHILIPPE, Alexis D. SOUCHET, Petros LAMERAS, Panagiotis PETRIDIS, Julien CAPORAL, Gildas COLDEBOEUF, Hadrien DUZAN, Multimodal teaching, learning and training in virtual reality: a review and case study Virtual Reality & Intelligent Hardware 2020, 2(5): 421-442

19. Jia Ming LEE, Xinxing XIA, Clemen OW, Felix CHUA, Yunqing GUAN, VEGO: A novel design towards customizable and adjustable head-mounted display for VR Virtual Reality & Intelligent Hardware 2020, 2(5): 443-453

20. Jingcheng QIAN, Yancong MA, Zhigeng PAN, Xubo YANG, Effects of Virtual-real fusion on immersion, presence, and learning performance in laboratory education Virtual Reality & Intelligent Hardware 2020, 2(6): 569-584

21. Hengwei XU, Siru LI, Wenpeng SONG, Jiajun SUN, Xinli WU, Xiaoqi WANG, Wenzhen YANG, Zhigeng PAN, Abdennour EI RHALIBI, Thermal perception method of virtual chemistry experiments Virtual Reality & Intelligent Hardware 2020, 2(4): 305-315

22. TJ MATTHEWS, Feng TIAN, Tom DOLBY, Interaction design for paediatric emergency VR training Virtual Reality & Intelligent Hardware 2020, 2(4): 330-344

23. Hongxin ZHANG, Jin ZHANG, Xue YIN, Kan ZHOU, Zhigeng PAN, Abdennour EI RHALIBI, Cloud-to-end rendering and storage management for virtual reality in experimental education Virtual Reality & Intelligent Hardware 2020, 2(4): 368-380

24. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

25. Haoyu WANG, Jianhuang WU, A virtual reality based surgical skills training simulator for catheter ablation with real-time and robust interaction Virtual Reality & Intelligent Hardware 2021, 3(4): 302-314

26. Na ZHANG, Liwen TAN, Fengying LI, Bing HAN, Yifa XU, Development and application of digital assistive teaching system for anatomy Virtual Reality & Intelligent Hardware 2021, 3(4): 315-335

27. Daniel VANKOV, David JANKOVSZKY, Effects of using headset-delivered virtual reality in road safety research: A systematic review of empirical studies Virtual Reality & Intelligent Hardware 2021, 3(5): 351-368

28. Dangxiao WANG, Yuan GUO, Shiyi LIU, Yuru ZHANG, Weiliang XU, Jing XIAO, Haptic display for virtual reality: progress and challenges Virtual Reality & Intelligent Hardware 2019, 1(2): 136-162

29. Aiguo SONG, Liyue FU, Multi-dimensional force sensor for haptic interaction: a review Virtual Reality & Intelligent Hardware 2019, 1(2): 121-135

30. Wenmin ZHU, Xiumin FAN, Yanxin ZHANG, Applications and research trends of digital human models in the manufacturing industry Virtual Reality & Intelligent Hardware 2019, 1(6): 558-579

31. Yuan WEI, Dongdong GUAN, Qiuchen WANG, Xiangxian LI, Yulong BIAN, Pu QIN, Yanning XU, Chenglei YANG, Virtual fire drill system supporting co-located collaboration Virtual Reality & Intelligent Hardware 2019, 1(3): 290-302