Temporal continuity of visual attention for future gaze prediction in immersive virtual reality

Abstract

Background

Eye tracking technology is receiving increased attention in the field of virtual reality. Specifically, future gaze prediction is crucial in pre-computation for many applications such as gaze-contingent rendering, advertisement placement, and content-based design. To explore future gaze prediction, it is necessary to analyze the temporal continuity of visual attention in immersive virtual reality.

Methods

In this paper, the concept of temporal continuity of visual attention is presented. Subsequently, an autocorrelation function method is proposed to evaluate the temporal continuity. Thereafter, the temporal continuity is analyzed in both free-viewing and task-oriented conditions.

Results

Specifically, in free-viewing conditions, the analysis of a free-viewing gaze dataset indicates that the temporal continuity performs well only within a short time interval. A task-oriented game scene condition was created and conducted to collect users’ gaze data. An analysis of the collected gaze data finds the temporal continuity has a similar performance with that of the free-viewing conditions. Temporal continuity can be applied to future gaze prediction and if it is good, users’ current gaze positions can be directly utilized to predict their gaze positions in the future.

Conclusions

The current gaze’s future prediction performances are further evaluated in both free-viewing and task-oriented conditions and discover that the current gaze can be efficiently applied to the task of short-term future gaze prediction. The task of long-term gaze prediction still remains to be explored.

Keyword

Temporal continuity ; Visual attention ; Autocorrelation analysis ; Gaze prediction ; Virtual reality

Cite this article

Zhiming HU, Sheng LI, Meng GAI. Temporal continuity of visual attention for future gaze prediction in immersive virtual reality. Virtual Reality & Intelligent Hardware, 2020, 2(2): 142-152 DOI:10.1016/j.vrih.2020.01.002

References

1. Duchowski A T. Gaze-based interaction: a 30 year retrospective. Computers & Graphics, 2018, 73, 59–69 DOI:10.1016/j.cag.2018.04.002

2. Mardanbegi D, Mayer B, Pfeuffer K, Jalaliniya S, Gellersen H, Perzl A. EyeSeeThrough: unifying tool selection and application in virtual environments. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka, Japan, IEEE, 2019, 474–483 DOI:10.1109/vr.2019.8797988

3. Guenter B, Finch M, Drucker S, Tan D, Snyder J. Foveated 3D graphics. ACM Transactions on Graphics, 2012, 31(6): 164 DOI:10.1145/2366145.2366183

4. Patney A, Salvi M, Kim J, Kaplanyan A, Wyman C, Benty N, Luebke D, Lefohn A. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics, 2016, 35(6): 1–12 DOI:10.1145/2980179.2980246

5. Alghofaili R, Solah M S, Huang H K, Sawahata Y, Pomplun M, Yu L F. Optimizing visual element placement via visual attention analysis. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka, Japan, IEEE, 2019, 464–473 DOI:10.1109/vr.2019.8797816

6. Hu Z M, Zhang C Y, Li S, Wang G P, Manocha D. SGaze: a data-driven eye-head coordination model for realtime gaze prediction. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(5): 2002–2010 DOI:10.1109/tvcg.2019.2899187

7. Berton F, Olivier A H, Bruneau J, Hoyet L, Pettre J. Studying gaze behaviour during collision avoidance with a virtual walker: influence of the virtual reality setup. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Osaka, Japan, IEEE, 2019, 717–725 DOI:10.1109/vr.2019.8798204

8. Chen J, Mi L T, Chen C P, Liu H W, Jiang J H, Zhang W B. Design of foveated contact lens display for augmented reality. Optics Express, 2019, 27(26): 38204–38219 DOI:10.1364/oe.381200

9. Zhou L, Chen C P, Wu Y S, Zhang Z L, Wang K Y, Yu B, Li Y. See-through near-eye displays enabling vision correction. Optics Express, 2017, 25(3): 2130–2142 DOI:10.1364/oe.25.002130

10. Itti L. Models of bottom-up and top-down visual attention. California Institute of Technology. 2000

11. Connor C E, Egeth H E, Yantis S. Visual attention: bottom-up versus top-down. Current Biology, 2004, 14(19): R850–R852 DOI:10.1016/j.cub.2004.09.041

12. Pinto Y, van der Leij A R, Sligte I G, Lamme V A F, Scholte H S. Bottom-up and top-down attention are independent. Journal of Vision, 2013, 13(3): 16 DOI:10.1167/13.3.16

13. Rottach K G, von Maydell R D, Das V E, Zivotofsky A Z, Discenna A O, Gordon J L, Landis D M D, Leigh R J. Evidence for independent feedback control of horizontal and vertical saccades from Niemann-Pick type C disease. Vision Research, 1997, 37(24): 3627–3638 DOI:10.1016/s0042-6989(96)00066-1

14. Sitzmann V, Serrano A, Pavel A, Agrawala M, Gutierrez D, Masia B, Wetzstein G. Saliency in VR: how do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics, 2018, 24(4): 1633–1642 DOI:10.1109/tvcg.2018.2793599

15. Henderson J. Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 2003, 7(11): 498–504 DOI:10.1016/j.tics.2003.09.006

16. Henderson J M, Nuthmann A, Luke S G. Eye movement control during scene viewing: Immediate effects of scene luminance on fixation durations. Journal of Experimental Psychology: Human Perception and Performance, 2013, 39(2): 318–322 DOI:10.1037/a0031224

17. Henderson J M, Olejarczyk J, Luke S G, Schmidt J. Eye movement control during scene viewing: Immediate degradation and enhancement effects of spatial frequency filtering. Visual Cognition, 2014, 22(3/4): 486–502 DOI:10.1080/13506285.2014.897662

18. Cheng M M, Zhang G X, Mitra N J, Huang X L, Hu S M. Global contrast based salient region detection. In: CVPR 2011. Colorado Springs, CO, USA, IEEE, 2011: 409–416 DOI:10.1109/cvpr.2011.5995344

19. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254–1259 DOI:10.1109/34.730558

20. Borji A, Sihite D N, Itti L. Probabilistic learning of task-specific visual attention. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI, IEEE, 2012, 470–477 DOI:10.1109/cvpr.2012.6247710

21. Harel J, Koch C, Perona P. Graph-based visual saliency. In: Advances in neural information processing systems. 2007, 545–552 DOI: 10.7551/mitpress/7503.003.0073

22. Marcella C, Lorenzo B, Giuseppe S, Rita C. Predicting human eye fixations via an LSTM-based saliency attentive model. IEEE Transactions on Image Processing, 2018, 27(10): 5142–5154 DOI:10.1109/tip.2018.2851672

23. Koulieris G A, Drettakis G, Cunningham D, Mania K. Gaze prediction using machine learning for dynamic stereo manipulation in games. In: 2016 IEEE Virtual Reality (VR). Greenville, SC, USA. IEEE, 2016, 113–120 DOI:10.1109/vr.2016.7504694

24. Arabadzhiyska E, Tursun O T, Myszkowski K, Seidel H P, Didyk P. Saccade landing position prediction for gaze-contingent rendering. ACM Transactions on Graphics, 2017, 36(4): 1–12 DOI:10.1145/3072959.3073642

25. Box G E, Jenkins G M, Reinsel G C. Time series analysis: forecasting and control. John Wiley & Sons, 2015

26. Lachenbruch P A, Cohen J. Statistical power analysis for the behavioral sciences (2nd Ed.). Journal of the American Statistical Association, 1989, 84(408): 1096 DOI:10.2307/2290095

27. Rumsey D J. Statistics II for dummies. John Wiley & Sons, 2009

1. Yang LI, Dong WU, Jin HUANG, Feng TIAN, Hong'an WANG, Guozhong DAI, Influence of multi-modality on moving target selection in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(3): 303-315

2. Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI, Gesture interaction in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 84-112

3. Athirah SYAMIMI, Yiwei GONG, Ryan LIEW, VR industrial applications―A singapore perspective Virtual Reality & Intelligent Hardware 2020, 2(5): 409-420

4. Jie GUO, Dongdong WENG, Yue LIU, Qiyong CHEN, Yongtian WANG, Analysis of teenagers' preferences and concerns regarding HMDs in education Virtual Reality & Intelligent Hardware 2021, 3(5): 369-382

5. Yijun LI, Miao WANG, Derong JIN, Frank STEINICKE, Qinping ZHAO, Effects of virtual environment and self-representations on perception and physical performance in redirected jumping Virtual Reality & Intelligent Hardware 2021, 3(6): 451-469

6. Susu HUANG, Daqing QI, Jiabin YUAN, Huawei TU, Review of studies on target acquisition in virtual reality based on the crossing paradigm Virtual Reality & Intelligent Hardware 2019, 1(3): 251-264

7. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

8. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

9. Yuan CHANG, Guo-Ping WANG, A review on image-based rendering Virtual Reality & Intelligent Hardware 2019, 1(1): 39-54

10. Shiguang QIU, Shuntao LIU, Deshuai KONG, Qichang HE, Three-dimensional virtual-real mapping of aircraft autom-atic spray operation and online simulation monitoring Virtual Reality & Intelligent Hardware 2019, 1(6): 611-621

11. Xu PENG, Zhenyu GAO, Yitong DING, Dongfeng ZHAO, Xiaoyu CHI, Study of ghost image suppression in polarized catadioptric virtual reality optical systems Virtual Reality & Intelligent Hardware 2020, 2(1): 70-78

12. Lihui HUANG, Siti Faatihah Binte Mohd TAIB, Ryan Aung BA, Zhe An GOH, Mengshan XU, Virtual reality research and development in NTU Virtual Reality & Intelligent Hardware 2020, 2(5): 394-408

13. Stéphanie PHILIPPE, Alexis D. SOUCHET, Petros LAMERAS, Panagiotis PETRIDIS, Julien CAPORAL, Gildas COLDEBOEUF, Hadrien DUZAN, Multimodal teaching, learning and training in virtual reality: a review and case study Virtual Reality & Intelligent Hardware 2020, 2(5): 421-442

14. Jia Ming LEE, Xinxing XIA, Clemen OW, Felix CHUA, Yunqing GUAN, VEGO: A novel design towards customizable and adjustable head-mounted display for VR Virtual Reality & Intelligent Hardware 2020, 2(5): 443-453

15. Jingcheng QIAN, Yancong MA, Zhigeng PAN, Xubo YANG, Effects of Virtual-real fusion on immersion, presence, and learning performance in laboratory education Virtual Reality & Intelligent Hardware 2020, 2(6): 569-584

16. Hengwei XU, Siru LI, Wenpeng SONG, Jiajun SUN, Xinli WU, Xiaoqi WANG, Wenzhen YANG, Zhigeng PAN, Abdennour EI RHALIBI, Thermal perception method of virtual chemistry experiments Virtual Reality & Intelligent Hardware 2020, 2(4): 305-315

17. TJ MATTHEWS, Feng TIAN, Tom DOLBY, Interaction design for paediatric emergency VR training Virtual Reality & Intelligent Hardware 2020, 2(4): 330-344

18. Hongxin ZHANG, Jin ZHANG, Xue YIN, Kan ZHOU, Zhigeng PAN, Abdennour EI RHALIBI, Cloud-to-end rendering and storage management for virtual reality in experimental education Virtual Reality & Intelligent Hardware 2020, 2(4): 368-380

19. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

20. Haoyu WANG, Jianhuang WU, A virtual reality based surgical skills training simulator for catheter ablation with real-time and robust interaction Virtual Reality & Intelligent Hardware 2021, 3(4): 302-314

21. Na ZHANG, Liwen TAN, Fengying LI, Bing HAN, Yifa XU, Development and application of digital assistive teaching system for anatomy Virtual Reality & Intelligent Hardware 2021, 3(4): 315-335

22. Daniel VANKOV, David JANKOVSZKY, Effects of using headset-delivered virtual reality in road safety research: A systematic review of empirical studies Virtual Reality & Intelligent Hardware 2021, 3(5): 351-368

23. Xiaolong LIU, Lili WANG, Redirected jumping in virtual scenes with alleys Virtual Reality & Intelligent Hardware 2021, 3(6): 470-483

24. Liming WANG, Xianwei CHEN, Tianyang DONG, Jing FAN, Virtual climbing: An immersive upslope walking system using passive haptics Virtual Reality & Intelligent Hardware 2021, 3(6): 435-450

25. Muhammad IRFAN, Muhammad MUNSIF, Deepdive: a learning-based approach for virtual camera in immersive contents Virtual Reality & Intelligent Hardware 2022, 4(3): 247-262

26. Dangxiao WANG, Yuan GUO, Shiyi LIU, Yuru ZHANG, Weiliang XU, Jing XIAO, Haptic display for virtual reality: progress and challenges Virtual Reality & Intelligent Hardware 2019, 1(2): 136-162

27. Aiguo SONG, Liyue FU, Multi-dimensional force sensor for haptic interaction: a review Virtual Reality & Intelligent Hardware 2019, 1(2): 121-135

28. Wenmin ZHU, Xiumin FAN, Yanxin ZHANG, Applications and research trends of digital human models in the manufacturing industry Virtual Reality & Intelligent Hardware 2019, 1(6): 558-579

29. Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN, Affine transformation of virtual 3D object using 2D localization of fingertips Virtual Reality & Intelligent Hardware 2020, 2(6): 534-555

30. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

31. Yuan WEI, Dongdong GUAN, Qiuchen WANG, Xiangxian LI, Yulong BIAN, Pu QIN, Yanning XU, Chenglei YANG, Virtual fire drill system supporting co-located collaboration Virtual Reality & Intelligent Hardware 2019, 1(3): 290-302

32. Xiaoxiong FAN, Yun CAI, Yufei YANG, Tianxing XU, Yike Li, Songhai ZHANG, Fanglue ZHANG, Detection of scene-irrelevant head movements via eye-head coordination information Virtual Reality & Intelligent Hardware 2021, 3(6): 501-514

33. Haochen HU, Yue LIU, Kang YUE, Yongtian WANG, Navigation in virtual and real environment using brain computer interface:a progress report Virtual Reality & Intelligent Hardware 2022, 4(2): 89-114

34. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246