Electrostatic tactile representation in multimedia mobile terminal

Abstract

Electrostatic tactile representation technology can enhance the authenticity and immersion of human-computer interaction by perception of tactile features such as the shape and texture of visual objects in touch screen in naked finger. Focusing on the application in multimedia terminal of electrostatic force tactile representation technology, this paper summarizes the typical devices of electrostatic tactile representation, tactile rendering model-driven and data-driven algorithm, driving signal loading method, tactile representation effect evaluation method and so on. The author's view on the development status and future prospects of this technology is presented as follows: (1) Electrostatic tactile representation technology is an optimization scheme for implementing the low power and bare finger tactile representation function on the surface of the multimedia terminal; (2) The rendering dynamic range of electrostatic tactile representation technology is still insufficient, and the rendering effect of rough texture is better, but there is no effective algorithm for the rendering of fine texture. Multiple fusion technology may be one of the solutions; (3) From the perspective of evaluation of tactile representation effect, which shows that there is still considerable room for improvement both in theoretical models and applied algorithms; (4) Electrostatic tactile representation technology is essentially an applied science. Mutual promotion and benign promotion of theoretical research and commercial application is the only way for its progress. Therefore, a more mature prototype of principle is urgently needed to be popularized and applied in commerce.

Keyword

Electrostatic tactile ; Rendering algorithm ; Model-driven ; Data-driven ; Surface tactile representation ; Human-computer interaction

Cite this article

Xuezhi YAN, Qiushuang WU, Xiaoying SUN. Electrostatic tactile representation in multimedia mobile terminal. Virtual Reality & Intelligent Hardware, 2019, 1(2): 201-218 DOI:10.3724/SP.J.2096-5796.2019.0003

References

1. Jiang S Q, Min W Q, Wang S H. Survey and prospect of intelligent interaction-oriented image recognition techniques. Journal of Computer Research and Development, 2016, 53: 113−122 DOI:10.7544/issn1000-1239.2016.20150689
蒋树强, 闵巍庆, 王树徽. 面向智能交互的图像识别技术综述与展望. 计算机研究与发展, 2016, 53: 113−122

2. Zhang Q G, Yan J Z, Wang P. 3D user interface configuration based on semantic virtual connector. Journal of Bejing University of Technology, 2012, 38: 1062−1067
张全贵, 闫健卓, 王普. 基于语义虚拟接口的三维用户界面组态. 北京工业大学学报, 2012, 38: 1062−1067

3. Ji B H, Yan X G, Wen W B. Interactive method and experiment in 3D human motion data abstraction. Journal of Beijing University of Aeronautics and Astronautics, 2000, 26(1): 91–94 DOI:10.3969/j.issn.1001-5965.2000.01.025
季白桦, 袁修干, 温文彪. 三维人体运动数据提取的人机交互方法及实验. 北京航空航天大学学报, 2000, 26(1): 91–94

4. Campbell P. Editorial on special issue on big data: Community cleverness required. Nature, 2008, 455: 1

5. Mallinckrodt E, Hughes A L, Sleator Jr W. Perception by the skin of electrically induced vibrations. Science, 1953, 118(3062): 277

6. Strong R M, Troxel D E. An electrotactile display. IEEE Transactions on Man-Machine Systems, 1970, 11: 72−79

7. Beebe D J, Hymel C M, Kaczmarek K A, Tyler M E. A polyimide-on-silicon electrostatic fingertip tactile display. In: Proceedings of 17th International Conference of the Engineering in Medicine and Biology Society. Montreal, Quebec, Canada: IEEE, 1995: 1545−1546

8. Tang H, Beebe DJ. A microfabricated electrostatic haptic display for persons with visual impairments. IEEE Transactions on Rehabilitation Engineering, 1998, 6(3): 241−248 DOI:DOI:10.1109/86.712216

9. Robles-De-La-Torre G, Hayward V. Force can overcome object geometry in the perception of shape through active touch. Nature, 2001, 412(6845): 445−448

10. Yamamoto A, Ishii T, Higuchi T. Electrostatic tactile display for presenting surface roughness sensation. IEEE International Conference on Industrial Technology, 2003, 2: 680−684 DOI:10.1109/ICIT.2003.1290736

11. Yamamoto A, Nagasawa S, Yamamoto H, Higuchi T. Electrostatic tactile display with thin film slider and its application to tactile telepresentation systems. IEEE Transactions on Visualization and Computer Graphics, 2006, 12(2): 168–177 DOI:10.1109/tvcg.2006.28

12. Linjama J, Mäkinen V. E-sense screen: Novel haptic display with capacitive electrosensory interface. In: 4th Workshop for Haptic and Audio Interaction Design. HAID, 2009

13. Bau O, Poupyrev I, Israr A, Harrison C. TeslaTouch: electrovibration for touch surfaces. In: Proceedings of the 23nd annual ACM symposium on User interface software and technology. New York, USA: ACM, 2010: 283−292

14. Meyer D J, Peshkin M A, Colgate J E. Fingertip friction modulation due to electrostatic attraction. In: World Haptics Conference (WHC). Daejeon, South Korea, 2013: 43−48 DOI:10.1109/WHC.2013.6548382

15. Makinen V, Suvanto P, Linjama J. Method and apparatus for sensory stimulation. US Patent: 7924144, 2011-4-12

16. Wu S W, Sun X Y, Wang Q L, Chen J. Tactile modeling and rendering image-textures based on electrovibration. The Visual Computer, 2017, 33(5): 637–646 DOI:10.1007/s00371-016-1214-3

17. Wang T T, Sun X Y. Electrostatic tactile rendering of image based on shape from shading. In: International Conference on Audio, Language and Image Processing. Shanghai, China: 2014, 775−779 DOI:10.1109/ICALIP.2014.7009900

18. Wu S W, Chen J, Sun X Y. Electrostatic force tactile rendering method for video perception. Journal of Computer Application, 2016, 36(4): 1137−1140 DOI:10.11772/j.issn.1001-9081.2016.04.1137
吴赛文, 陈建, 孙晓颖. 面向视频感知的静电力触觉渲染方法. 计算机应用, 2016, 36(4): 1137−1140

19. Giraud F, Amberg M, Lemaire-Semail B. Merging two tactile stimulation principles: Electrovibration and squeeze film effect. In: World Haptics Conference (WHC). Daejeon, South Korea, 2013: 199−203 DOI:10.1109/WHC.2013.6548408

20. Vezzoli E, Messaoud W B, Amberg M, Giraud F, Lemaire-Semail B, Bueno M A. Physical and perceptual independence of ultrasonic vibration and electrovibration for friction modulation. IEEE Transactions on Haptics, 2015, 8(2): 235–239 DOI:10.1109/toh.2015.2430353

21. Ito K, Okamoto S, Elfekey H, Kajimto H, Yamada Y. A texture display using vibrotactile and electrostatic friction stimuli surpasses one based on either type of stimulus. IEEE International Conference on Systems. Man, and Cybernetics (SMC). Banff, AB, Canada: 2017, 2343−2348 DOI:DOI:10.1109/SMC.2017.8122972

22. Lécuyer A, Burkhardt J M, Etienne L. Feeling bumps and holes without a haptic interface: the perception of pseudo-haptic textures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vienna, Austria: ACM, 2004, 239−246

23. Xu C, Israr A, Poupyrev I, Bau O, Harrison C. Tactile display for the visually impaired using TeslaTouch. In: CHI'11 Extended Abstracts on Human Factors in Computing Systems. Vancouver, BC, Canada: ACM, 2011: 317−322

24. Park J, Doxon A J, Provancher W R, Johnson D E, Tan H Z. Haptic edge sharpness perception with a contact location display. IEEE Transactions on Haptics, 2012, 5(4): 323–331 DOI:10.1109/toh.2012.14

25. Kocsis M B, Cholewiak S A, Traylor R M, Adelstein B D, Daniel Hirleman E, Tan H Z. Discrimination of real and virtual surfaces with sinusoidal and triangular gratings using the fingertip and stylus. IEEE Transactions on Haptics, 2013, 6(2): 181–192 DOI:10.1109/toh.2012.31

26. Perez A G, Lobo D, Chinello F, Cirio G, Malvezzi M, Martín J S, Prattichizzo D, Otaduy M A. Soft finger tactile rendering for wearable haptics. World Haptics Conference (WHC). Evanston, IL, USA: IEEE, 2015, 327−332 DOI:DOI:10.1109/WHC.2015.7177733

27. Ito K, Okamoto S, Elfekey H, Yamada Y. High-Quality Texture Display: The Use of Vibrotactile and Variable-Friction Stimuli in Conjunction. Lecture Notes in Electrical Engineering. Singapore: Springer Singapore, 2017: 125−130 DOI:10.1007/978-981-10-4157-0_22

28. Osgouei R H, Kim J R, Choi S. Improving 3D shape recognition withelectrostatic friction display. IEEE Transactions on Haptics, 2017, 10(4): 533–544 DOI:10.1109/toh.2017.2710314

29. Minsky M, Ming O Y, Steele O, F P Jr Brooks, Behensky M. Feeling and seeing: Issues in force display. ACM SIGGRAPH Computer Graphics, 1990, 24(2): 235–241 DOI:10.1145/91394.91451

30. Vasudevan H, Manivannan M. Tangible images: runtime generation of haptic textures from images. In: Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. Reno, NE, USA, 2008: 357−360 DOI:10.1109/HAPTICS.2008.4479971

31. Li J L, Song A G, Zhang X R. Texture force/tactile rendering method for color images. Journal of computer aided design and graphics, 2011, 23: 719−724
李佳璐, 宋爱国, 张小瑞. 彩色图像的纹理力/触觉渲染方法. 计算机辅助设计与图形学学报, 2011, 23: 719−724

32. Saga S, Deguchi K. Lateral-force-based 2. 5-dimensional tactile display for touch screen. In: Haptics Symposium (HAPTICS). Vancouver, BC, Canada: IEEE2012: 15−22 DOI:10.1109/HAPTIC.2012.6183764

33. Kim S C, Israr A, Poupyrev I. Tactile rendering of 3D features on touch surfaces. In: Proceedings of the 26th annual ACM symposium on User interface software and technology. St. Andrews, Scotland, United Kingdom: ACM, 2013: 531−538

34. Tian L, Song A G, Wang W. Research on the image haptic display method based on PDE. Chinese Journal of Scientific Instrument, 2013, 34(10): 2316–2321 DOI:10.19650/j.cnki.cjsi.2013.10.023
田磊, 宋爱国, 王蔚. 基于PDE方法的图像力触觉再现方法研究. 仪器仪表学报, 2013, 34(10): 2316–2321

35. Meyer D J, Peshkin M A, Colgate J E. Modeling and synthesis of tactile texture with spatial spectrograms for display on variable friction surfaces. World Haptics Conference (WHC). Evanston, IL, USA: IEEE, 2015, 125−130 DOI:10.1109/WHC.2015.7177702

36. Wang T T, Chen J, Sun X Y. Electrostatic tactile rendering of local texture characteristics of image. Journal of Image and Graphics, 2016, 21: 1383−1391
王婷婷, 陈建, 孙晓颖. 图像局部纹理特性的静电力触觉渲染. 中国图象图形学报, 2016, 21: 1383−1391

37. Kim J R, Shin S. Touch3D: touchscreen interaction on multiscopic 3D with electrovibration haptics. In: ACM SIGGRAPH 2017 Posters. Los Angeles, California: ACM, 2017: 1−2

38. Vasudevan H, Manivannan M. Recordable haptic textures. In: International Workshop on Haptic Audio Visual Environments and their Applications (HAVE 2006). Ottawa, Ont, Canada: IEEE, 2006: 130−133 DOI:10.1109/HAVE.2006.283779

39. Andrews S, Lang J. Interactive scanning of haptic textures and surface compliance. Sixth International Conference on 3-d Digital Imaging and Modeling (3DIM 2007). Montreal, QC, Canada: 2007, 99−106 DOI:10.1109/3DIM.2007.30

40. Romano J M, Kuchenbecker K J. Creating realistic virtual textures from contact acceleration data. IEEE Transactions on Haptics, 2012, 5(2): 109–119 DOI:10.1109/toh.2011.38

41. Culbertson H, Unwin J, Kuchenbecker K J. Modeling and rendering realistic textures from unconstrained tool-surface interactions. IEEE Transactions on Haptics, 2014, 7(3): 381–393 DOI:10.1109/toh.2014.2316797

42. Abdulali A, Hassan W, Jeon S. Sample selection of multi-trial data for data-driven haptic texture modeling. World Haptics Conference (WHC). Munich, Germany: IEEE, 2017, 66−71 DOI:10.1109/WHC.2017.7989878

43. Ilkhani G, Aziziaghdam M, Samur E. Data-driven texture rendering on an electrostatic tactile display. International Journal of Human–Computer Interaction, 2017, 33(9): 756–770 DOI:10.1080/10447318.2017.1286766

44. Osgouei R H, Shin S, Kim J R, Choi S. An inverse neural network model for data-driven texture rendering on electrovibration display. Haptics Symposium (HAPTICS). San Francisco, CA, USA, IEEE, 2018: 270−277 DOI:10.1109/HAPTICS.2018.8357187

45. Sianov A, Harders M. Exploring feature-based learning for data-driven haptic rendering. IEEE Transactions on Haptics, 2018, 11(3): 388–399 DOI:10.1109/toh.2018.2817483

46. Jiao J, Zhang Y R, Wang D X, Visell Y, Cao D K, Guo X W, Sun X Y. Data-driven rendering of fabric textures on electrostatic tactile displays. Haptics Symposium (HAPTICS). San Francisco, CA, USA: IEEE, 2018, 169−174 DOI:10.1109/HAPTICS.2018.8357171

47. Tezuka M, Kitamura N, Tanaka K, Miki N. Presentation of various tactile sensations using micro-needle electrotactile display. PLoS One, 2016, 11(2): e0148410 DOI:10.1371/journal.pone.0148410

48. Lim J M, Jeong H T. Force and displacement analysis of a haptic touchscreen. International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, IEEE, 2015, 589−591 DOI:DOI:10.1109/ICCE.2015.7066539

49. Wijekoon D, Cecchinato M E, Hoggan E, Linjama J. Electrostatic Modulated Friction as Tactile Feedback: Intensity Perception. Haptics: Perception, Devices, Mobility, and Communication. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012: 613−624 DOI:10.1007/978-3-642-31401-8_54

50. Accot J, Zhai S. Beyond fitts' law: models for trajectory-based HCI tasks. In: Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM, 1997, 97(1): 295−302

51. Yamamoto T, Yamamoto Y. Electrical properties of the epidermal stratum corneum. Medical & Biological Engineering, 1976, 14(2): 151−158 DOI:10.1007/bf02478741

52. Shultz C D, Peshkin M A, Colgate J E. Surface haptics via electroadhesion: Expanding electrovibration with Johnsen and Rahbek. World Haptics Conference (WHC). Evanston, IL, USA: IEEE, 2015: 57−62 DOI:10.1109/WHC.2015.7177691

53. Vezzoli E, Amberg M, Giraud F, Lemaire-Semail B. Electrovibration Modeling Analysis. In: Haptics: Neuroscience, Devices, Modeling, and Applications. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014: 369−376

54. Kim H, Kang J, Kim K D, Lim K M, Ryu J. Method for providing electrovibration with uniform intensity. IEEE Transactions on Haptics, 2015, 8(4): 492–496 DOI:10.1109/toh.2015.2476810

55. Long H. Measurement modeling and evaluation for effect of electrostatic tactile display. Dissertation for Master Degree. Changchun: Jilin University, 2017
龙慧. 静电力触觉再现效果测量建模与评价. 长春: 吉林大学, 2017

56. S J Jr Bolanowski , Gescheider G A, Verrillo R T, Checkosky C M. Four channels mediate the mechanical aspects of touch. The Journal of the Acoustical Society of America, 1988, 84(5): 1680−1694 DOI:10.1121/1.397184

57. Accot J, Zhai S. Scale effects in steering law tasks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Seattle, Washington, USA: ACM, 2001: 1−8

58. Kaczmarek K A, Nammi K, Agarwal A K, Tyler M E, Haase S J, Beebe D J. Polarity effect in electrovibration for tactile display. IEEE Transactions on Biomedical Engineering, 2006, 53(10): 2047−2054 DOI:10.1109/tbme.2006.881804

2. Chongyang SUN, Weizhi NAI, Xiaoying SUN, Tactile sensitivity in ultrasonic haptics: Do different parts of hand and different rendering methods have an impact on perceptual threshold? Virtual Reality & Intelligent Hardware 2019, 1(3): 265-275

3. Egemen ERTUGRUL, Ping LI, Bin SHENG, On attaining user-friendly hand gesture interfaces to control existing GUIs Virtual Reality & Intelligent Hardware 2020, 2(2): 153-161

4. Shiwei CHENG, Qiufeng PING, Jialing WANG, Yijian CHEN, EyeGaze: Hybrid eye tracking approach for handheld mobile devices Virtual Reality & Intelligent Hardware 2022, 4(2): 173-188

5. Mohammad Mahmudul ALAM, S. M. Mahbubur RAHMAN, Affine transformation of virtual 3D object using 2D localization of fingertips Virtual Reality & Intelligent Hardware 2020, 2(6): 534-555

6. Yuanyuan SHI, Yunan LI, Xiaolong FU, Kaibin MIAO, Qiguang MIAO, Review of dynamic gesture recognition Virtual Reality & Intelligent Hardware 2021, 3(3): 183-206

7. Wanlu ZHENG, Wenming ZHENG, Yuan ZONG, Multi-scale discrepancy adversarial network for cross-corpus speech emotion recognition Virtual Reality & Intelligent Hardware 2021, 3(1): 65-75

8. Xiaoxiong FAN, Yun CAI, Yufei YANG, Tianxing XU, Yike Li, Songhai ZHANG, Fanglue ZHANG, Detection of scene-irrelevant head movements via eye-head coordination information Virtual Reality & Intelligent Hardware 2021, 3(6): 501-514