Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2022, 4(2): 115-131

Published Date:2022-4-20 DOI: 10.1016/j.vrih.2021.12.002

Design and evaluation of window management operations in AR headset+smartphone interface

Abstract

Background
The combination of an augmented reality (AR) headset and a smartphone can simultaneously provide a wider display and a precise touch input; it can redefine the way we use applications today. However, users are deprived of such benefits owing to the independence of the two devices. There is a lack of intuitive and direct interactions between them.
Methods
In this study, we conduct a formative investigation to understand the window management requirements and interaction preferences of using an AR headset and a smartphone simultaneously and report the insights we gained. In addition, we introduce an example vocabulary of window management operations in the AR headset and smartphone interface.
Results
This allows users to manipulate windows in a virtual space and shift windows between devices efficiently and seamlessly.

Keyword

Augmented reality ; Smartphone ; Window management ; User study ; Interaction

Cite this article

Jie REN, Chun YU, Yueting WENG, Chengchi ZHOU, Yuanchun SHI. Design and evaluation of window management operations in AR headset+smartphone interface. Virtual Reality & Intelligent Hardware, 2022, 4(2): 115-131 DOI:10.1016/j.vrih.2021.12.002

References

1. Fitzmaurice G W. Situated information spaces and spatially aware palmtop computers. Communications of the ACM, 1993, 36(7): 39–49 DOI:10.1145/159544.159566

2. Wallis C. The multitasking generation. Time, 2006, 167(13): 48–55

3. Ens B M, Finnegan R, Irani P P. The personal cockpit: a spatial interface for effective task switching on head-worn displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2014, 3171–3180 DOI:10.1145/2556288.2557058

4. Ens B, Hincapié-Ramos J D, Irani P. Ethereal planes: a design framework for 2D information space in 3D mixed reality environments. In: Proceedings of the 2nd ACM symposium on Spatial user interaction. 2014, 2–12 DOI:10.1145/2659766.2659769

5. Grubert J, Heinisch M, Quigley A, Schmalstieg D. MultiFi: multi fidelity interaction with displays on and around the body. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2015, 3933–3942 DOI:10.1145/2702123.2702331

6. Millette A, McGuffin M J. DualCAD: integrating augmented reality with a desktop GUI and smartphone interaction. In: 2016 IEEE International Symposium on Mixed and Augmented Reality. Merida, Mexico, IEEE, 2016, 21–26 DOI:10.1109/ismar-adjunct.2016.0030

7. Grandi J G, Debarba H G, Nedel L, Maciel A. Design and evaluation of a handheld-based 3D user interface for collaborative object manipulation. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017, 5881–5891 DOI:10.1145/3025453.3025935

8. Wang J, Lindeman R. Coordinated 3D interaction in tablet- and HMD-based hybrid virtual environments. In: Proceedings of the 2nd ACM symposium on Spatial user interaction. 2014, 70–79 DOI:10.1145/2659766.2659777

9. Kaufmann H, Schmalstieg D. Mathematics and geometry education with collaborative augmented reality. Computers & Graphics, 2003, 27(3): 339–345 DOI:10.1016/s0097-8493(03)00028-1

10. Bowman D A, Wineman J, Hodges L F, Allison D. Designing animal habitats within an immersive VE. IEEE Computer Graphics and Applications, 1998, 18(5): 9–13 DOI:10.1109/38.708555

11. Szalavári Z, Gervautz M. The personal interaction panel-a two-handed interface for augmented reality. Computer Graphics Forum, 1997, 16(3): C335–C346 DOI:10.1111/1467-8659.00137

12. Billinghurst M, Bowskill J, Jessop M, Morphett J. A wearable spatial conferencing space. In: Digest of Papers. Second International Symposium on Wearable Computers. Pittsburgh, PA, USA, IEEE, 1998, 76–83 DOI:10.1109/iswc.1998.729532

13. Feiner S, MacIntyre B, Haupt M, Solomon E. Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the 6th annual ACM symposium on User interface software and technology. Atlanta, Georgia, USA, New York, ACM Press, 1993, 145–155 DOI:10.1145/168642.168657

14. Billinghurst M, Bowskill J, Dyer N, Morphett J. An evaluation of wearable information spaces. In: Proceedings of IEEE 1998 Virtual Reality Annual International Symposium. Atlanta, GA, USA, IEEE, 1998, 20–27 DOI:10.1109/vrais.1998.658418

15. Lee J H, An S G, Kim Y, Bae S H. Projective windows: bringing windows in space to the fingertip. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC Canada, New York, NY, USA, ACM, 2018 DOI:10.1145/3173574.3173792

16. Pierce J S, Forsberg A S, Conway M J, Hong S, Zeleznik R C, Mine M R. Image plane interaction techniques in 3D immersive environments. In: Proceedings of the 1997 symposium on Interactive 3D graphics. Providence, Rhode Island, USA, New York, ACM Press, 1997, 39 DOI:10.1145/253284.253303

17. Valkov D, Steinicke F, Bruder G, Hinrichs K. 2D touching of 3D stereoscopic objects. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011, 1353–1362 DOI:10.1145/1978942.1979142

18. Henderson S J, Feiner S. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In: 2009 8th IEEE International Symposium on Mixed and Augmented Reality. Orlando, FL, USA, IEEE, 2009, 135–144 DOI:10.1109/ismar.2009.5336486

19. Al-Sada M, Ishizawa F, Tsurukawa J, Nakajima T. Input Forager: a user-driven interaction adaptation approach for head worn displays. In: Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia. 2016, 115–122 DOI:10.1145/3012709.3012719

20. Arora R, Kazi R H, Grossman T, Fitzmaurice G, Singh K. SymbiosisSketch: combining 2D & 3D sketching for designing detailed 3D objects in situ. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018, 1–15 DOI:10.1145/3173574.3173759

21. Wang J, Lindeman R W. Object impersonation: towards effective interaction in tablet- and HMD-based hybrid virtual environments. 2015 IEEE Virtual Reality (VR), 2015, 111–118 DOI:10.1109/vr.2015.7223332

22. Yee K P. Peephole displays: pen interaction on spatially aware handheld computers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2003, 1–8 DOI:10.1145/642611.642613

23. Chen X, Marquardt N, Tang A, Boring S, Greenberg S. Extending a mobile device's interaction space through body-centric interaction. In: Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services. 2012, 151–160 DOI:10.1145/2371574.2371599

24. Chan L W, Kao H S, Chen M Y, Lee M S, Hsu J, Hung Y P. Touching the void: direct-touch interaction for intangible displays. In: Proceedings of the 28th international conference on Human factors in computing systems. Atlanta, Georgia, USA, New York, ACM Press, 2010, 2625–2634 DOI:10.1145/1753326.1753725

25. Cornelio Martinez P I, de Pirro S, Vi C T, Subramanian S. Agency in mid-air interfaces. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver Colorado USA, New York, NY, USA, ACM, 2017, 2426–2439 DOI:10.1145/3025453.3025457

Related

1. Yanxiao ZHANG, Yuecong MIN, Xilin CHEN, Teaching Chinese sign language with a smartphone Virtual Reality & Intelligent Hardware 2021, 3(3): 248-260

2. Mengting XIAO, Zhiquan FENG, Xiaohui YANG, Tao XU, Qingbei GUO, Multimodal interaction design and application in augmented reality for chemical experiment Virtual Reality & Intelligent Hardware 2020, 2(4): 291-304

3. Mingxuan CHEN, Ping ZHANG, Zebo WU, Xiaodan CHEN, A multichannel human-swarm robot interaction system in augmented reality Virtual Reality & Intelligent Hardware 2020, 2(6): 518-533

4. Yukang YAN, Xin YI, Chun YU, Yuanchun SHI, Gesture-based target acquisition in virtual and augmented reality Virtual Reality & Intelligent Hardware 2019, 1(3): 276-289

5. Ahmed L ALYOUSIFY, Ramadhan J MSTAFA, AR-assisted children book for smart teaching and learning of Turkish alphabets Virtual Reality & Intelligent Hardware 2022, 4(3): 263-277

6. Yonghang TAI, Junsheng SHI, Junjun PAN, Aimin HAO, Victor CHANG, Augmented reality-based visual-haptic modeling for thoracoscopic surgery training systems Virtual Reality & Intelligent Hardware 2021, 3(4): 274-286

7. Yuan GAO, Le XIE, A review on the application of augmented reality in craniomaxillofacial surgery Virtual Reality & Intelligent Hardware 2019, 1(1): 113-120

8. Jinyu LI, Bangbang YANG, Danpeng CHEN, Nan WANG, Guofeng ZHANG, Hujun BAO, Survey and evaluation of monocular visual-inertial SLAM algorithms for augmented reality Virtual Reality & Intelligent Hardware 2019, 1(4): 386-410

9. Xiaomei ZHAO, Fulin TANG, Yihong WU, Real-time human segmentation by BowtieNet and a SLAM-based human AR system Virtual Reality & Intelligent Hardware 2019, 1(5): 511-524

10. Chan QIU, Shien ZHOU, Zhenyu LIU, Qi GAO, Jianrong TAN, Digital assembly technology based on augmented reality and digital twins: a review Virtual Reality & Intelligent Hardware 2019, 1(6): 597-610

11. Wang LI, Junfeng WANG, Sichen JIAO, Meng WANG, Shiqi LI, Research on the visual elements of augmented reality assembly processes Virtual Reality & Intelligent Hardware 2019, 1(6): 622-634

12. Pengfei HAN, Gang ZHAO, A review of edge-based 3D tracking of rigid objects Virtual Reality & Intelligent Hardware 2019, 1(6): 580-596

13. Shenze WANG, Kaikai DU, Ningfang SONG, Dongfeng ZHAO, Di FENG, Zhengqian TU, Study on the adaptability of augmented reality smartglasses for astigmatism based on holographic waveguide grating Virtual Reality & Intelligent Hardware 2020, 2(1): 79-85

14. Lingfei ZHU, Qi CAO, Yiyu CAI, Development of augmented reality serious games with a vibrotactile feedback jacket Virtual Reality & Intelligent Hardware 2020, 2(5): 454-470

15. Xiang ZHOU, Liyu TANG, Ding LIN, Wei HAN, Virtual & augmented reality for biological microscope in experiment education Virtual Reality & Intelligent Hardware 2020, 2(4): 316-329

16. Jiaxin LIU, Hongxin ZHANG, Chuankang LI, COMTIS: Customizable touchless interaction system for large screen visualization Virtual Reality & Intelligent Hardware 2020, 2(2): 162-174

17. Yun-Han LEE, Tao ZHAN, Shin-Tson WU, Prospects and challenges in augmented reality displays Virtual Reality & Intelligent Hardware 2019, 1(1): 10-20

18. Zike YAN, Hongbin ZHA, Flow-based SLAM: From geometry computation to learning Virtual Reality & Intelligent Hardware 2019, 1(5): 435-460

19. Mohib ULLAH, Sareer Ul AMIN, Muhammad MUNSIF, Muhammad Mudassar YAMIN, Utkurbek SAFAEV, Habib KHAN, Salman KHAN, Habib ULLAH, Serious games in science education: a systematic literature review Virtual Reality & Intelligent Hardware 2022, 4(3): 189-209

20. Rafik HAMZA, Minh-Son DAO, Privacy-preserving deep learning techniques for wearable sensor-based big data applications Virtual Reality & Intelligent Hardware 2022, 4(3): 210-222

21. Yang LI, Jin HUANG, Feng TIAN, Hong-An WANG, Guo-Zhong DAI, Gesture interaction in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 84-112

22. Rachel HUANG, Carisa HARRIS-ADAMSON, Dan ODELL, David REMPEL, Design of finger gestures for locomotion in virtual reality Virtual Reality & Intelligent Hardware 2019, 1(1): 1-9

23. Jun-Hao YIN, Chin-Boon CHNG, Pooi-Mun WONG, Nicholas HO, Matthew CHUA, Chee-Kong CHUI, VR and AR in human performance researchAn NUS experience Virtual Reality & Intelligent Hardware 2020, 2(5): 381-393

24. Xiaoxiong FAN, Yun CAI, Yufei YANG, Tianxing XU, Yike Li, Songhai ZHANG, Fanglue ZHANG, Detection of scene-irrelevant head movements via eye-head coordination information Virtual Reality & Intelligent Hardware 2021, 3(6): 501-514

25. Hayat ULLAH, Sitara AFZAL, Imran Ullah KHAN, Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey Virtual Reality & Intelligent Hardware 2022, 4(3): 223-246

26. Yang JIAO, Jiangtao GONG, Yingqing XU, A study of multimodal interaction design based on touch and audition for the visually impaired Virtual Reality & Intelligent Hardware 2019, 1(2): 233-239

27. Chongyang SUN, Weizhi NAI, Xiaoying SUN, Tactile sensitivity in ultrasonic haptics: Do different parts of hand and different rendering methods have an impact on perceptual threshold? Virtual Reality & Intelligent Hardware 2019, 1(3): 265-275

28. Egemen ERTUGRUL, Ping LI, Bin SHENG, On attaining user-friendly hand gesture interfaces to control existing GUIs Virtual Reality & Intelligent Hardware 2020, 2(2): 153-161

29. TJ MATTHEWS, Feng TIAN, Tom DOLBY, Interaction design for paediatric emergency VR training Virtual Reality & Intelligent Hardware 2020, 2(4): 330-344

30. Rui HAN, Zhiquan FENG, Jinglan TIAN, Xue FAN, Xiaohui YANG, Qingbei GUO, An intelligent navigation experimental system based on multi-mode fusion Virtual Reality & Intelligent Hardware 2020, 2(4): 345-353

31. Carlos ARCE-LOPERA, María José ARIAS, Gustavo CORRALES, Training birdsong recognition using virtual reality Virtual Reality & Intelligent Hardware 2021, 3(5): 397-406

32. Wei LIU, Designing generation Y interactions: The case of YPhone Virtual Reality & Intelligent Hardware 2022, 4(2): 132-152

33. Shiwei CHENG, Qiufeng PING, Jialing WANG, Yijian CHEN, EyeGaze: Hybrid eye tracking approach for handheld mobile devices Virtual Reality & Intelligent Hardware 2022, 4(2): 173-188

34. Aiguo SONG, Liyue FU, Multi-dimensional force sensor for haptic interaction: a review Virtual Reality & Intelligent Hardware 2019, 1(2): 121-135

35. Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN, Affine transformation of virtual 3D object using 2D localization of fingertips Virtual Reality & Intelligent Hardware 2020, 2(6): 534-555

36. Yuanyuan SHI, Yunan LI, Xiaolong FU, Kaibin MIAO, Qiguang MIAO, Review of dynamic gesture recognition Virtual Reality & Intelligent Hardware 2021, 3(3): 183-206

37. Wanlu ZHENG, Wenming ZHENG, Yuan ZONG, Multi-scale discrepancy adversarial network for cross-corpus speech emotion recognition Virtual Reality & Intelligent Hardware 2021, 3(1): 65-75

38. Vivian GÓMEZ, Kelly PEÑARANDA, Pablo FIGUEROA, Lessons learned from requirements gathering for virtual reality simulators Virtual Reality & Intelligent Hardware 2021, 3(5): 407-422

39. Xuezhi YAN, Qiushuang WU, Xiaoying SUN, Electrostatic tactile representation in multimedia mobile terminal Virtual Reality & Intelligent Hardware 2019, 1(2): 201-218