Development of augmented reality serious games with a vibrotactile feedback jacket

Abstract

Background

In the past few years, augmented reality (AR) has rapidly advanced and has been applied in different fields. One of the successful AR applications is the immersive and interactive serious games, which can be used for education and learning purposes.

Methods

In this project, a prototype of an AR serious game is developed and demonstrated. Gamers utilize a head-mounted device and a vibrotactile feedback jacket to explore and interact with the AR serious game. Fourteen vibration actuators are embedded in the vibrotactile feedback jacket to generate immersive AR experience. These vibration actuators are triggered in accordance with the designed game scripts. Various vibration patterns and intensity levels are synthesized in different game scenes. This article presents the details of the entire software development of the AR serious game, including game scripts, game scenes with AR effects design, signal processing flow, behavior design, and communication configuration. Graphics computations are processed using the graphics processing unit in the system.

Results /Conclusions

The performance of the AR serious game prototype is evaluated and analyzed. The computation loads and resource utilization of normal game scenes and heavy computation scenes are compared. With 14 vibration actuators placed at different body positions, various vibration patterns and intensity levels can be generated by the vibrotactile feedback jacket, providing different real-world feedback. The prototype of this AR serious game can be valuable in building large-scale AR or virtual reality educational and entertainment games. Possible future improvements of the proposed prototype are also discussed in this article.

Keyword

Augmented reality ; AR serious games ; Vibrotactile feedback jacket ; Game scenes

Cite this article

Lingfei ZHU, Qi CAO, Yiyu CAI. Development of augmented reality serious games with a vibrotactile feedback jacket. Virtual Reality & Intelligent Hardware, 2020, 2(5): 454-470 DOI:10.1016/j.vrih.2020.05.005

References

1. Cai Y. 3D Immersive Interactive Learning. Singapore: Springer Verlag, 2013 DOI:10.1007/978-981-4021-90-6

2. Xie Y, Zhang Y Z, Cai Y Y. Virtual reality engine disassembly simulation with natural hand-based interaction. In: VR, Simulations and Serious Games for Education. Singapore: Springer Singapore, 2018, 121–128 DOI:10.1007/978-981-13-2844-2_11

3. Orozco M, Silva J, El A, Petriu E. The role of haptics in games. In: Haptics Rendering and Applications. InTech, 2012, 217–234 DOI:10.5772/32809

4. Barajas A O, Al Osman H, Shirmohammadi S. A Serious Game for children with Autism Spectrum Disorder as a tool for play therapy. In: 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH). Perth, WA, Australia, IEEE, 2017, 1–7 DOI:10.1109/segah.2017.7939266

5. DiPietro J, Kelemen A, Liang Y L, Sik-Lanyi C. Computer- and robot-assisted therapies to aid social and intellectual functioning of children with autism spectrum disorder. Medicina, 2019, 55(8): 440 DOI:10.3390/medicina55080440

6. Tsikinas S, Xinogalos S. Studying the effects of computer serious games on people with intellectual disabilities or autism spectrum disorder: a systematic literature review. Journal of Computer Assisted Learning, 2019, 35(1): 61–73 DOI:10.1111/jcal.12311

7. Lu A, Chan S, Cai Y Y, Huang L H, Nay Z T, Goei S L. Learning through VR gaming with virtual pink dolphins for children with ASD. Interactive Learning Environments, 2018, 26(6): 718–729 DOI:10.1080/10494820.2017.1399149

8. Cai Y Y, Chiew R, Nay Z T, Indhumathi C, Huang L H. Design and development of VR learning environments for children with ASD. Interactive Learning Environments, 2017, 25(8): 1098–1109 DOI:10.1080/10494820.2017.1282877

9. Tang J S Y, Falkmer M, Chen N T M, Bӧlte S, Girdler S. Designing a serious game for youth with ASD: perspectives from end-users and professionals. Journal of Autism and Developmental Disorders, 2019, 49(3): 978–995 DOI:10.1007/s10803-018-3801-9

10. Borresen A, Wolfe C, Lin C, Tian Y, Raghuraman S, Nahrstedt K, Prabhakaran B, Annaswamy T M. Usability of an immersive augmented reality based telerehabilitation system with haptics (ARTESH) for synchronous remote musculoskeletal examination. International Journal of Telerehabilitation, 2019, 11(1): 23–32 DOI:10.5195/ijt.2019.6275

11. Culbertson H, Kuchenbecker K J. Ungrounded haptic augmented reality system for displaying roughness and friction. IEEE/ASME Transactions on Mechatronics, 2017, 22(4): 1839–1849 DOI:10.1109/tmech.2017.2700467

12. Maisto M, Pacchierotti C, Chinello F, Salvietti G, de Luca A, Prattichizzo D. Evaluation of wearable haptic systems for the fingers in augmented reality applications. IEEE Transactions on Haptics, 2017, 10(4): 511–522 DOI:10.1109/toh.2017.2691328

13. Pridhvi Krishna M V, Mehta S, Verma S, Rane S. Mixed reality in smart computing education system. In: 2018 International Conference on Smart Systems and Inventive Technology (ICSSIT). Tirunelveli, India, IEEE, 2018, 72–75 DOI:10.1109/icssit.2018.8748813

14. Bhargava A, Bertrand J W, Gramopadhye A K, Madathil K C, Babu S V. Evaluating multiple levels of an interaction fidelity continuum on performance and learning in near-field training simulations. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(4): 1418–1427 DOI:10.1109/tvcg.2018.2794639

15. Johnson C I, Bailey S K T, van Buskirk W L. Designing effective feedback messages in serious games and simulations: A research review. In: Instructional Techniques to Facilitate Learning and Motivation of Serious Games. Cham: Springer International Publishing, 2016, 119–140 DOI:10.1007/978-3-319-39298-1_7

16. Menelas B A, Benaoudia R. Use of haptics to promote learning outcomes in serious games. Multimodal Technologies and Interaction, 2017, 1(4): 31 DOI:10.3390/mti1040031

17. Vaquero-Melchor D, Bernardos A M. Enhancing interaction with augmented reality through mid-air haptic feedback: architecture design and user feedback. Applied Sciences, 2019, 9(23): 5123 DOI:10.3390/app9235123

18. Lee J, Kim Y, Kim G J. Effects of visual feedback on out-of-body illusory tactile sensation when interacting with augmented virtual objects. IEEE Transactions on Human-Machine Systems, 2017, 47(1): 101–112 DOI:10.1109/thms.2016.2599492

19. de Jesus Oliveira V A, Brayda L, Nedel L, Maciel A. Designing a vibrotactile head-mounted display for spatial awareness in 3D spaces. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(4): 1409–1417 DOI:10.1109/tvcg.2017.2657238

20. Garciavalle G, Ferre M, Brenosa J, Vargas D. Evaluation of presence in virtual environments: haptic vest and user's haptic skills. IEEE Access, 2018, 6: 7224–7233 DOI:10.1109/access.2017.2782254

21. Rantala J, Majaranta P, Kangas J, Isokoski P, Akkil D, Špakov O, Raisamo R. Gaze interaction with vibrotactile feedback: review and design guidelines. Human-Computer Interaction, 2020, 35(1): 1–39 DOI:10.1080/07370024.2017.1306444

22. Shim Y A, Lee J, Lee G. Exploring multimodal watch-back tactile display using wind and vibration. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, New York, ACM Press, 2018, 1–12 DOI:10.1145/3173574.3173706

23. Kaul O B, Rohs M. HapticHead: 3D guidance and target acquisition through a vibrotactile grid. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. California, USA, New York, ACM Press, 2016, 2533–2539 DOI:10.1145/2851581.2892355

24. Young E M, Memar A H, Agarwal P, Colonnese N. Bellowband: a pneumatic wristband for delivering local pressure and vibration. In: 2019 IEEE World Haptics Conference (WHC). Tokyo, Japan, IEEE, 2019, 55–60 DOI:10.1109/whc.2019.8816075

25. Wolf D, Rietzler M, Hnatek L, Rukzio E. Face/on: multi-modal haptic feedback for head-mounted displays in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(11): 3169–3177 DOI:10.1109/tvcg.2019.2932215

26. Louison C, Ferlay F, Mestre D R. Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). Los Angeles, CA, USA, IEEE, 2017, 99–102 DOI:10.1109/3dui.2017.7893324

27. Delazio A, Nakagaki K, Klatzky R L, Hudson S E, Lehman J F, Sample A P. Force jacket: pneumatically-actuated jacket for embodied haptic experiences. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, New York, ACM Press, 2018 DOI:10.1145/3173574.3173894

28. Kishishita Y, Das S, Ramirez A V, Thakur C, Tadayon R, Kurita Y. Muscleblazer: force-feedback suit for immersive experience. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka, Japan, IEEE, 2019, 1813–1818 DOI:10.1109/vr.2019.8797962

29. Rognon C, Koehler M, Duriez C, Floreano D, Okamura A M. Soft haptic device to render the sensation of flying like a drone. IEEE Robotics and Automation Letters, 2019, 4(3): 2524-2531 DOI:10.1109/LRA.2019.2907432

30. Ouyang Q Q, Wu J, Wu M. Vibrotactile display of flight attitude with combination of multiple coding parameters. Applied Sciences, 2017, 7(12): 1291 DOI:10.3390/app7121291

31. Günther S, Makhija M, Müller F, Schön D, Mühlhäuser M, Funk M. PneumAct: pneumatic kinesthetic actuation of body joints in virtual reality environments. In: Proceedings of the 2019 on Designing Interactive Systems Conference. San Diego, CA, USA, ACM, 2019, 227–240 DOI:10.1145/3322276.3322302

32. ImmerzInc, product of KOR-FX [2020-02-20]. http://www.korfx.com/products

33. bHapticsInc, product of TACTOT, haptic vest for torso [2020-02-20]. https://www.bhaptics.com/tactsuit/

34. USAInc Woojer, product of Vest Pro [2020-02-20]. https://www.woojer.com/vest/

35. Ltd VRElectronics, product of TESLASUIT [2020-02-20]. https://teslasuit.io/the-suit/

36. Zhang Z. A Flexible New Technique for Camera Calibration (Technical report). Microsoft Research. MSR-TR-98-71, 1998

37. Reaction Time Statistics (2020) [2020-02-20]. http://www.humanbenchmark.com/tests/reactiontime/statistics

38. Tang Z, Lin Y, Lee K, Hwang J, Chuang J. ESTHER: joint camera self-calibration and automatic radial distortion correction from tracking of walking humans. IEEE Access, 2019, 7: 10754–10766 DOI:10.1109/access.2019.2891224

39. Zhang Z D, Matsushita Y, Ma Y. Camera calibration with lens distortion from low-rank textures. In: CVPR 2011. Providence, RI, USA, IEEE, 2011, 2321–2328 DOI:10.1109/cvpr.2011.5995548

40. Tehrani M A, Beeler T, Grundhöfer A. A practical method for fully automatic intrinsic camera calibration using directionally encoded light. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA, IEEE, 2017, 125–133 DOI:10.1109/cvpr.2017.21

41. Kakani V, Kim H, Lee J, Ryu C, Kumbham M. Automatic distortion rectification of wide-angle images using outlier refinement for streamlining vision tasks. Sensors, 2020, 20(3): 894 DOI:10.3390/s20030894

1. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

2. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

3. Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG, Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems Virtual Reality & Intelligent Hardware 2021, 3(4): 274-286

4. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

5. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

6. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

7. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

8. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

9. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

10. Pengfei HAN, Gang ZHAO, A review of edge-based 3D tracking of rigid objects Virtual Reality & Intelligent Hardware 2019, 1(6): 580-596

11. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

12. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

13. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

14. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460