A study of multimodal interaction design based on touch and audition for the visually impaired

Yang JIAO , Jiangtao GONG , Yingqing XU

Department of Information Art and Design, Tsinghua University, Beijing 100084, China

Abstract

Background

It studies multimodal interaction design based on touch and audition for the visually impaired and related assistive devices.

Methods

With the consideration of traditional multimodal hypothesis, cognition and multimodal interaction design, this study proposes a multimodal interaction modal based on touch and auditions for the visually impaired, illustrates information flows from tactile and auditory interaction interfaces, and conducts a blind user experiment under multimodal scenario to analyze the ability of user perception and cognition.

Results

Under the multimodal interaction scenario, the visually impaired users achieved better user experience and higher recognition efficiency.

Conclusions

Discusses the design principles of blind assistive devices and their interactions by this interaction modal and user experience, guiding the explorations of blind assistive devices design. The visually impaired can enjoy the internet like us.

Keyword

The visually impaired ; Sense of touch & hearing ; Multiple channel interaction ; Assistive device

Cite this article

Yang JIAO, Jiangtao GONG, Yingqing XU. A study of multimodal interaction design based on touch and audition for the visually impaired. Virtual Reality & Intelligent Hardware, 2019, 1(2): 233-239 DOI:10.3724/SP.J.2096-5796.2019.0002

References

1. Salisbury K. Haptics: the technology of touch. HPCwire special to HPCwire, 1995

2. 董士海, 戴国忠, 王坚. 人机沟通有方: 多通道用户界面的发展. 中国计算机用户, 1998(29): 25−27

3. Dong S, Chen M. The model, method and instances of multimodal user interface. Acta Scientiarum Naturalium Universitatis Pekinensis, 1998, 34(2): 231–239
董士海, 陈敏. 多通道用户界面的模型, 方法及实例. 北京大学学报(自然科学版), 1998, 34(2): 231−239

4. 李宏汀, 陈柏鸿, 葛列众. 触觉交互研究的回顾与展望. 人类工效学, 2008, 14(3): 51−53, 17

5. Deborah Kent Stein, The Optacon : Past, Present, and Future [2019-01-29]. https: //nfb. org/images/nfb/publications/bm/bm98/bm980506. htm

6. Wu D Y, Gan J, Wu D Y, Gan J. Assistant haptic interaction technology for blind Internet user. Journal of Engineering Design, 2010, 17(2): 128−133, 155 DOI:10.3785/j.issn.1006-754X.2010.02.010
吴丹玥, 干静. 辅助盲人访问Internet的力/触觉交互技术. 工程设计学报, 2010, 17(2): 128−133, 155

7. Apple: Siri. http: //www. apple. com/ios/siri/

8. VoiceOver for OSX. A feature that speaks for itself [2019-01-29]. http://www.apple.com/accessibility/osx/voiceover/

9. Google Talkbac: Android Apps on Google Play [2019-01-29]. https://play.google.com/store/apps/details?id=com. google. android. marvin. talkback

10. Allport D A, Antonis B, Reynolds P. On the division of attention: A disproof of the single channel hypothesis. Quarterly Journal of Experimental Psychology, 1972, 24(2): 225–235 DOI:10.1080/00335557243000102

11. Kokjer K. The information capacity of the human fingertip. IEEE Transactions on Systems, Man, and Cybernetics, 1987, 17(1): 100–102 DOI:10.1109/tsmc.1987.289337

12. Goldstein E B. Blackwell Handbook of Sensation and Perception. Malden, MA, USA: Blackwell Publishing Ltd., 2005 DOI:10.1002/9780470753477

13. O'Modhrain S, Giudice N A, Gardner J A, Legge G E. Designing media for visually-impaired users of refreshable touch displays: Possibilities and pitfalls. IEEE Transactions on Haptics, 2015, 8(3): 248−257 DOI:10.1109/toh.2015.2466231

14. Pawluk D T V, Adams R J, Kitada R. Designing haptic assistive technology for individuals who are blind or visually impaired. IEEE Transactions on Haptics, 2015, 8(3): 258−278 DOI:10.1109/toh.2015.2471300

15. Vidal-Verdu F, Hafez M. Graphical tactile displays for visually-impaired people. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2007, 15(1): 119−130 DOI:10.1109/tnsre.2007.891375

16. Lumbreras M, Sanchez J. Interactive 3D Sound Hyperstories for Blind Children. In: the SIGCHI conference on Human Factors in Computing Systems. ACM, 1999, 318−325

17. Microsoft: Cortana [2019-01-29]. https://dev.windows.com/en-us/cortana

18. JIAO Yang. Research on tactile display and user experience by haptic perception for the visually impaired. Beijing: Tsinghua University, 2017
焦阳. 面向盲人触觉认知的触觉显示与体验研究. 北京: 清华大学, 2017

19. Bach-Y-rita P, Collins C C, Saunders F A, White B, Scadden L. Vision substitution by tactile image projection. Nature, 1969, 221(5184): 963–964 DOI:10.1038/221963a0

20. Schweikhardt W, Klöpfer K. Rechnerunterstützte aufbereitung vonbildshirmtext-grafiken in eine tastbare darstellung. Stuttgart: Stuttgart University Press, 1984

21. Jiao Y, Gong J T, Xu Y Q. Graille: design research of graphical tactile display for the visually impaired. 2016(1): 94−96 DOI:10.16272/j.cnki.cn11-1392/j.2016.01.024
焦阳, 龚江涛, 徐迎庆. 盲人触觉图像显示器Graile设计研究. 装饰, 2016(1): 94−96

22. Bristol Braille: Bristol Braille Technology: Canute 360 [2019-01-29]. http://www.bristolbraille.co.uk/

23. Völkel T, Weber G, Baumann U. Tactile Graphics Revised: The Novel BrailleDis 9000 Pin-Matrix Device with Multitouch Input. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008: 835−842 DOI:10.1007/978-3-540-70540-6_124

24. Metec-AG: Display 7200, a tactile graphical display by Metec-AG [2019-01-29]. https://www.metec-ag.de/en/produkte-graphik-display.php

Full Text

62 PDF

5292 HTML

EXPORT

EndNote ProCite Reference Manager BibTex RefWorks

SHARE ON

ARTICLES BY AUTHORS

VRIH

BAIDU SCHOLAR

WANFANG DATA

CNKI