Cloud-to-end rendering and storage management for virtual reality in experimental education

Abstract

Background

Real-time 3D rendering and interaction is important for virtual reality (VR) experimental education. Unfortunately, standard end-computing methods prohibitively escalate computational costs. Thus, reducing or distributing these requirements needs urgent attention, especially in light of the COVID-19 pandemic.

Methods

In this study, we design a cloud-to-end rendering and storage system for VR experimental education comprising two models: background and interactive. The cloud server renders items in the background and sends the results to an end terminal in a video stream. Interactive models are then lightweight-rendered and blended at the end terminal. An improved 3D warping and hole-filling algorithm is also proposed to improve image quality when the user’s viewpoint changes.

Results

We build three scenes to test image quality and network latency. The results show that our system can render 3D experimental education scenes with higher image quality and lower latency than any other cloud rendering systems.

Conclusions

Our study is the first to use cloud and lightweight rendering for VR experimental education. The results demonstrate that our system provides good rendering experience without exceeding computation costs.

Keyword

Clout-to-end render ; Cloud storage ; Virtual reality ; Experimental education

Cite this article

Hongxin ZHANG, Jin ZHANG, Xue YIN, Kan ZHOU, Zhigeng PAN, Abdennour EI RHALIBI. Cloud-to-end rendering and storage management for virtual reality in experimental education. Virtual Reality & Intelligent Hardware, 2020, 2(4): 368-380 DOI:10.1016/j.vrih.2020.07.001

References

1. Yang J, Huang X, Liu X. An analysis of education inequality in China. International Journal of Educational Development, 2014, 37: 2–10 DOI:10.1016/j.ijedudev.2014.03.002

2. Burdea G, Coiffet P. Virtual Reality Technology, New York, Wiley, 1994

3. Richter T, Boehringer D, Jeschke S. LiLa: A European project on networked experiments. In: Automation, Communication and Cybernetics in Science and Engineering 2009/2010. Berlin, Heidelberg, Springer Berlin Heidelberg, 2010, 307–317 DOI:10.1007/978-3-642-16208-4_27

4. Koretsky M D, Amatore D, Barnes C, Kimura S. Enhancement of student learning in experimental design using a virtual laboratory. IEEE Transactions on Education, 2008, 51(1): 76–85 DOI:10.1109/te.2007.906894

5. Hashemipour M, Manesh H F, Bal M. A modular virtual reality system for engineering laboratory education. Computer Applications in Engineering Education, 2011, 19(2): 305–314 DOI:10.1002/cae.20312

6. Jara C A, Candelas F A, Puente S T, Torres F. Hands-on experiences of undergraduate students in Automatics and Robotics using a virtual and remote laboratory. Computers & Education, 2011, 57(4): 2451–2461 DOI:10.1016/j.compedu.2011.07.003

7. Torres F, Candelas F A, Puente S T, Pomares J, Gil P, Ortiz F G. Experiences with virtual environment and remote laboratory for teaching and learning robotics at the university of Alicante. International Journal of Engineering Education, 2006, 22(4): 766–776

8. Eldrdgem H G. A scalable graphics system for clusters. In: Proc. of ACM SIGRAPH 2001. Press ACM, LosAngeles, 2001

9. Lei X H, Jiang X H, Wang C H. Design and implementation of streaming media processing software based on RTMP. 2012 5th International Congress on Image and Signal Processing, 2012, 192‒196 DOI:10.1109/cisp.2012.6469981

10. Humphreys G, Houston M, Ng R, Frank R, Ahern S, Kirchner P D, Klosowski J T. Chromium: a stream-processing framework for interactive rendering on clusters. 2002, 21(3): 693–702

11. OnLive. http://www.onlive.com, 2012

12. Gaikai. http://www.gaikai.com, 2012

13. Ubitus. http://www.ubitus.com/, 2013

14. Ciinow. http://www.ciinow.com/, 2014

15. Huang C Y, Hsu C H, Chang Y C, Chen K T. GamingAnywhere: an open cloud gaming system. In: Proceedings of the 4th ACM Multimedia Systems Conference on-MMSys'13. Oslo, Norway, NewYork, Press ACM, 2013, 36–47 DOI:10.1145/2483977.2483981

16. Hoppe H. Progressive meshes. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques-SIGGRAPH '96. New York, ACM Press, 1996, 99–108 DOI:10.1145/237170.237216

17. Hoppe H, DeRose T, Duchamp T, McDonald J, Stuetzle W. Mesh optimization. In: Proceedings of the 20th annual conference on Computer graphics and interactive techniques-SIGGRAPH '93. New York, ACM Press, 1993,19–26 DOI:10.1145/166117.166119

18. Ma J P, Luo X N, Chen B, Chen H H. A geometry simplification method for mobile 3D graphics. Journal of Computer Research and Development, 2008, 45(8): 1395–1401

19. Jr. L M. An image-based approach to three-dimensional computer graphics. Thesis (Ph.D.)-University of North Carolina at Chapel Hill, 1997

20. Shi S, Nahrstedt K, Campbell R. A real-time remote rendering system for interactive mobile graphics. ACM Transactions on Multimedia Computing, Communications, and Applications, 2012, 8(3s): 1–20 DOI:10.1145/2348816.2348825

21. Lamport L. The part-time parliament. ACM Transactions on Computer Systems (TOCS), 1998, 16(2): 133–169 DOI:10.1145/279227.279229

22. Ongaro D, Ousterhout J. In search of an understandable consensus algorithm. In: 2014 {USENIX} Annual Technical Conference ({USENIX}{ATC} 14). 2014, 305‒319

23. Hunt P, Konar M, Junqueira F P, Reed B. ZooKeeper: wait-free coordination for internet-scale systems. In: USENIX Annual Technical Conference. 2010

24. Dawson-Howe K M, Vernon D. Simple pinhole camera calibration. International Journal of Imaging Systems and Technology, 1994, 5(1): 1–6 DOI:10.1002/ima.1850050102

25. Wang Y, Ostermann J, Zhang Y Q. Video processing and communications. 2002

26. Wang Z, Lu L G, Bovik A C. Video quality assessment based on structural distortion measurement. Signal Processing: Image Communication, 2004, 19(2): 121–132 DOI:10.1016/s0923-5965(03)00076-6

1. Yang LI, Dong WU, Jin HUANG, Feng TIAN, Hong'an WANG, Guozhong DAI, Influence of multi-modality on moving target selection in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(3): 303-315

2. Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI, Gesture interaction in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 84-112

3. Athirah SYAMIMI, Yiwei GONG, Ryan LIEW, VR industrial applications―A singapore perspective Virtual Reality & Intelligent Hardware 2020, 2(5): 409-420

4. Jie GUO, Dongdong WENG, Yue LIU, Qiyong CHEN, Yongtian WANG, Analysis of teenagers' preferences and concerns regarding HMDs in education Virtual Reality & Intelligent Hardware 2021, 3(5): 369-382

5. Yijun LI, Miao WANG, Derong JIN, Frank STEINICKE, Qinping ZHAO, Effects of virtual environment and self-representations on perception and physical performance in redirected jumping Virtual Reality & Intelligent Hardware 2021, 3(6): 451-469

6. Susu HUANG, Daqing QI, Jiabin YUAN, Huawei TU, Review of studies on target acquisition in virtual reality based on the crossing paradigm Virtual Reality & Intelligent Hardware 2019, 1(3): 251-264

7. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

8. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

9. Yuan CHANG, Guo-Ping WANG, A review on image-based rendering Virtual Reality & Intelligent Hardware 2019, 1(1): 39-54

10. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

11. Xu PENG, Zhenyu GAO, Yitong DING, Dongfeng ZHAO, Xiaoyu CHI, Study of ghost image suppression in polarized catadioptric virtual reality optical systems Virtual Reality & Intelligent Hardware 2020, 2(1): 70-78

12. Zhiming HU, Sheng LI, Meng GAI, Temporal continuity of visual attention for future gaze prediction in immersive virtual reality Virtual Reality & Intelligent Hardware 2020, 2(2): 142-152

13. Lihui HUANG, Siti Faatihah Binte Mohd TAIB, Ryan Aung BA, Zhe An GOH, Mengshan XU, Virtual reality research and development in NTU Virtual Reality & Intelligent Hardware 2020, 2(5): 394-408

14. Stéphanie PHILIPPE, Alexis D. SOUCHET, Petros LAMERAS, Panagiotis PETRIDIS, Julien CAPORAL, Gildas COLDEBOEUF, Hadrien DUZAN, Multimodal teaching, learning and training in virtual reality: a review and case study Virtual Reality & Intelligent Hardware 2020, 2(5): 421-442

15. Jia Ming LEE, Xinxing XIA, Clemen OW, Felix CHUA, Yunqing GUAN, VEGO: A novel design towards customizable and adjustable head-mounted display for VR Virtual Reality & Intelligent Hardware 2020, 2(5): 443-453

16. Jingcheng QIAN, Yancong MA, Zhigeng PAN, Xubo YANG, Effects of Virtual-real fusion on immersion, presence, and learning performance in laboratory education Virtual Reality & Intelligent Hardware 2020, 2(6): 569-584

17. Hengwei XU, Siru LI, Wenpeng SONG, Jiajun SUN, Xinli WU, Xiaoqi WANG, Wenzhen YANG, Zhigeng PAN, Abdennour EI RHALIBI, Thermal perception method of virtual chemistry experiments Virtual Reality & Intelligent Hardware 2020, 2(4): 305-315

18. TJ MATTHEWS, Feng TIAN, Tom DOLBY, Interaction design for paediatric emergency VR training Virtual Reality & Intelligent Hardware 2020, 2(4): 330-344

19. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

20. Haoyu WANG, Jianhuang WU, A virtual reality based surgical skills training simulator for catheter ablation with real-time and robust interaction Virtual Reality & Intelligent Hardware 2021, 3(4): 302-314

21. Na ZHANG, Liwen TAN, Fengying LI, Bing HAN, Yifa XU, Development and application of digital assistive teaching system for anatomy Virtual Reality & Intelligent Hardware 2021, 3(4): 315-335

22. Daniel VANKOV, David JANKOVSZKY, Effects of using headset-delivered virtual reality in road safety research: A systematic review of empirical studies Virtual Reality & Intelligent Hardware 2021, 3(5): 351-368

23. Xiaolong LIU, Lili WANG, Redirected jumping in virtual scenes with alleys Virtual Reality & Intelligent Hardware 2021, 3(6): 470-483

24. Liming WANG, Xianwei CHEN, Tianyang DONG, Jing FAN, Virtual climbing: An immersive upslope walking system using passive haptics Virtual Reality & Intelligent Hardware 2021, 3(6): 435-450

25. Muhammad IRFAN, Muhammad MUNSIF, Deepdive: a learning-based approach for virtual camera in immersive contents Virtual Reality & Intelligent Hardware 2022, 4(3): 247-262

26. Dangxiao WANG, Yuan GUO, Shiyi LIU, Yuru ZHANG, Weiliang XU, Jing XIAO, Haptic display for virtual reality: progress and challenges Virtual Reality & Intelligent Hardware 2019, 1(2): 136-162

27. Aiguo SONG, Liyue FU, Multi-dimensional force sensor for haptic interaction: a review Virtual Reality & Intelligent Hardware 2019, 1(2): 121-135

28. Wenmin ZHU, Xiumin FAN, Yanxin ZHANG, Applications and research trends of digital human models in the manufacturing industry Virtual Reality & Intelligent Hardware 2019, 1(6): 558-579

29. Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN, Affine transformation of virtual 3D object using 2D localization of fingertips Virtual Reality & Intelligent Hardware 2020, 2(6): 534-555

30. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

31. Yuan WEI, Dongdong GUAN, Qiuchen WANG, Xiangxian LI, Yulong BIAN, Pu QIN, Yanning XU, Chenglei YANG, Virtual fire drill system supporting co-located collaboration Virtual Reality & Intelligent Hardware 2019, 1(3): 290-302

32. Xiaoxiong FAN, Yun CAI, Yufei YANG, Tianxing XU, Yike Li, Songhai ZHANG, Fanglue ZHANG, Detection of scene-irrelevant head movements via eye-head coordination information Virtual Reality & Intelligent Hardware 2021, 3(6): 501-514

33. Haochen HU, Yue LIU, Kang YUE, Yongtian WANG, Navigation in virtual and real environment using brain computer interface:a progress report Virtual Reality & Intelligent Hardware 2022, 4(2): 89-114

34. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246