Home About the Journal Latest Work Current Issue Archive Special Issues Editorial Board
<< Previous Next >>

2019, 1(2): 185-200

Published Date:2019-4-20 DOI: 10.3724/SP.J.2096-5796.2018.0012

Psychophysics of wearable haptic/tactile perception in a multisensory context


Multisensory lab based in Peking University, has carried out basic studies in multisensory space and time processing, intersensory binding and haptic/tactile perception. We exploited a typical paradigm of multisensory illusion-temporal ventriloquist effect and applied it in a wide range of multisensory interactions (mainly focused on temporal processing). In this work, we summarized how the tactile stimuli were exploited to compose tactile cues and as tactile apparent motion to interface with other sensory stimuli (visual and auditory stimuli) to examine the underlying perceptual organization in a multisensory context. Moreover, we introduced two examples of wearable haptic/tactile perception in our lab, by using two customized tactile devices and discussed the potential applications in this field.


Multisensory ; Ventriloquism effect ; Wearable haptics ; Perceptual organization

Cite this article

Xiao LEI, Tingwei ZHANG, Kun CHEN, Jue ZHANG, Yue TIAN, Fang FANG, Lihan CHEN. Psychophysics of wearable haptic/tactile perception in a multisensory context. Virtual Reality & Intelligent Hardware, 2019, 1(2): 185-200 DOI:10.3724/SP.J.2096-5796.2018.0012


1. Shams L, Beierholm U R. Causal inference in perception. Trends in Cognitive Sciences, 2010, 14(9): 425–432 DOI:10.1016/j.tics.2010.07.001

2. Kramer A F, Jacobson A. Perceptual organization and focused attention: The role of objects and proximity in visual processing. Perception & Psychophysics, 1991, 50(3): 267–284 DOI:10.3758/bf03206750

3. Wagemans J, Elder J H, Kubovy M, Palmer S E, Peterson M A, Singh M, von der Heydt R. A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bulletin, 2012, 138(6): 1172–1217 DOI:10.1037/a0029333

4. King A J, Calvert G A. Multisensory integration: perceptual grouping by eye and ear. Current Biology, 2001, 11(8): R322–R325 DOI:10.1016/s0960-9822(01)00175-0

5. Gepshtein S, Kubovy M. Stability and change in perception: Spatial organization in temporal context. Experimental Brain Research, 2005, 160(4): 487–495 DOI:10.1007/s00221-004-2038-3

6. Sanabria D, Soto-Faraco S, Chan J S, Spence C. When does visual perceptual grouping affect multisensory integration? Cognitive, Affective, & Behavioral Neuroscience, 2004, 4(2): 218–229 DOI:10.3758/cabn.4.2.218

7. Sanabria D, Soto-Faraco S, Chan J, Spence C. Intramodal perceptual grouping modulates multisensory integration: Evidence from the crossmodal dynamic capture task. Neuroscience Letters, 2005, 377(1): 59–64 DOI:10.1016/j.neulet.2004.11.069

8. Fendrich R, Corballis P M. The temporal cross-capture of audition and vision. Perception & Psychophysics, 2001, 63(4): 719–725 DOI:10.3758/bf03194432

9. Vroomen J, de Gelder B. Temporal ventriloquism: sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception and Performance, 2004, 30(3): 513–518 DOI:10.1037/0096-1523.30.3.513

10. Freeman E, Driver J. Direction of visual apparent motion driven solely by timing of a static sound. Current Biology, 2008, 18(16): 1262–1266 DOI:10.1016/j.cub.2008.07.066

11. Slutsky D A, Recanzone G H. Temporal and spatial dependency of the ventriloquism effect. Neuroreport, 2001, 12(1): 7–10 DOI:10.1097/00001756-200101220-00009

12. Alais D, Burr D. The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 2004, 14(3): 257–262 DOI:10.1016/j.cub.2004.01.029

13. Roseboom W, Kawabe T, Nishida S. Direction of visual apparent motion driven by perceptual organization of cross-modal signals. Journal of Vision, 2013, 13(1): 6 DOI:10.1167/13.1.6

14. Chen L H, Vroomen J. Intersensory binding across space and time: A tutorial review. Attention, Perception, & Psychophysics, 2013, 75(5): 790–811 DOI:10.3758/s13414-013-0475-4

15. Chen L H, Zhou X L. Capture of intermodal Visual/Tactile apparent motion by moving and static sound. Seeing and Perceiving, 2011, 24(4): 369–389 DOI:10.1163/187847511x584434

16. Chen L H, Zhou X L, Müller H J, Shi Z H. What you see depends on what you hear: Temporal averaging and crossmodal integration. . Journal of Experimental Psychology: General, 2018, 147(12): 1851–1864 DOI:10.1037/xge0000487

17. Jiang Y S, Chen L H. Mutual influences of intermodal Visual/Tactile apparent motion and auditory motion with uncrossed and crossed arms. Multisensory Research, 2013, 26(1/2): 19–51 DOI:10.1163/22134808-00002409

18. Shi Z H, Chen L H, Müller H J. Auditory temporal modulation of the visual Ternus effect: The influence of time interval. Experimental Brain Research, 2010, 203(4): 723–735 DOI:10.1007/s00221-010-2286-3

19. Chen L H, Shi Z H, Müller H J. Influences of intra-and crossmodal grouping on visual and tactile Ternus apparent motion. Brain Research, 2010, 1354: 152–162 DOI:10.1016/j.brainres.2010.07.064

20. Yiltiz H, Chen L H. Corrigendum: Tactile input and empathy modulate the perception of ambiguous biological motion. Frontiers in Psychology, 2015, 6: 161 DOI:10.3389/fpsyg.2015.00161

21. Shi Z H, Zou H, Rank M, Chen L H, Hirche S, Muller H J. Effects of packet loss and latency on the temporal discrimination of visual-haptic events. IEEE Transactions on Haptics, 2010, 3(1): 28–36 DOI:10.1109/toh.2009.45

22. Lyons G, Sanabria D, Vatakis A, Spence C. The modulation of crossmodal integration by unimodal perceptual grouping: A visuotactile apparent motion study. Experimental Brain Research, 2006, 174(3): 510–516 DOI:10.1007/s00221-006-0485-8

23. Chen L H, Shi Z H, Müller H J. Interaction of perceptual grouping and crossmodal temporal capture in tactile apparent-motion. PLoS One, 2011, 6(2): e17130 DOI:10.1371/journal.pone.0017130

24. Wang Q, Hayward V. Tactile synthesis and perceptual inverse problems seen from the viewpoint of contact mechanics. ACM Transactions on Applied Perception, 2008, 5(2): 1–19 DOI:10.1145/1279920.1279921

25. Hogan N, Kay B A, Fasse E D, Mussa-Ivaldi F A. Haptic illusions: experiments on human manipulation and perception of “virtual objects”. Cold Spring Harbor Symposia on Quantitative Biology, 1990, 55: 925–931 DOI:10.1101/sqb.1990.055.01.086

26. Wang D X, Zhang Y R, Hou J X, Wang Y, Lv P, Chen Y G, Zhao H. Idental: A haptic-based dental simulator and its preliminary user evaluation. IEEE Transactions on Haptics, 2012, 5(4): 332–343 DOI:10.1109/toh.2011.59

27. Katzakis N, Tong J, Ariza O, Chen L, Klinker G, R B, Röder B. , Steinicke F. Stylo and handifact: modulating haptic perception through visualizations for posture training in augmented reality. In: Proceedings of the 5th Symposium on Spatial User Interaction. Brighton, United Kingdom, ACM, 2017: 58-67 DOI:10.1145/3131277.3132181

28. Von Békésy G. Neural funneling along the skin and between the inner and outer hair cells of the cochlea. The Journal of the Acoustical Society of America, 1959, 31(9): 1236–1249 DOI:10.1121/1.1907851

29. Von Békésy G. Funneling in the nervous system and its role in loudness and sensation intensity on the skin. The Journal of the Acoustical Society of America, 1958, 30(5): 399–412 DOI:10.1121/1.1909626

30. Barghout A, Cha J, Saddik A E, Kammerl J, Steinbach E. Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion. In: 2009 IEEE International Workshop on Haptic Audio visual Environments and Games, 2009, 19–23 DOI:10.1109/HAVE.2009.5356122

31. Geldard F A. Saltation in somesthesis. Psychological Bulletin, 1982, 92(1): 136–175 DOI:10.1037//0033-2909.92.1.136

32. Geldard F A, Sherrick C E. The cutaneous “rabbit”: A perceptual illusion. Science, 1972, 178(4057): 178–179 DOI:10.1126/science.178.4057.178

33. Tong J, Ngo V, Goldreich D. Tactile length contraction as Bayesian inference. Journal of Neurophysiology, 2016, 116(2): 369–379 DOI:10.1152/jn.00029.2016

34. Blankenburg F, Ruff C C, Deichmann R, Rees G, Driver J. The cutaneous rabbit illusion affects human primary sensory cortex somatotopically. PLoS Biology, 2006, 4(3): e69 DOI:10.1371/journal.pbio.0040069

35. Nijhawan R. Motion extrapolation in catching. Nature, 1994, 370(6487): 256–257 DOI:10.1038/370256b0

36. Sheth B R, Nijhawan R, Shimojo S. Changing objects lead briefly flashed ones. Nature Neuroscience, 2000, 3(5): 489–495 DOI:10.1038/74865

37. Chen L. Tactile flash lag effect: Taps with changing intensities lead briefly flashed taps. In: 2013 World Haptics Conference (WHC). Daejeon, South Korea, 2013, 253–258 DOI:10.1109/WHC.2013.6548417

38. Gallace A, Tan H Z, Spence C. Tactile change detection. In: First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems World Haptics Conference. 2005, Pisa, Italy, 2005: 12–16 DOI:10.1109/WHC.2005.122

39. Gallace A, Tan H Z, Spence C. Numerosity judgments for tactile stimuli distributed over the body surface. Perception, 2006, 35(2): 247–266 DOI:10.1068/p5380

40. Medina J, McCloskey M, Coslett H B, Rapp B. Somatotopic representation of location: Evidence from the Simon effect. Journal of Experimental Psychology: Human Perception and Performance, 2014, 40(6): 2131–2142 DOI:10.1037/a0037975

41. Salzer Y, Aisenberg D, Oron-Gilad T, Henik A. In touch with the Simon effect *The first two authors contributed equally. Experimental Psychology, 2014, 61(3): 165–179 DOI:10.1027/1618-3169/a000236

42. Zheng W T, Chen L H. The roles of attentional shifts and attentional reengagement in resolving the spatial compatibility effect in tactile simon-like tasks. Scientific Reports, 2018, 8: 8760 DOI:10.1038/s41598-018-27114-9

43. Tian Y, Chen L H. Cross-modal attention modulates tactile subitizing but not tactile numerosity estimation. Attention, Perception, & Psychophysics, 2018, 80(5): 1229–1239 DOI:10.3758/s13414-018-1507-x

44. Bortone I, Leonardis D, Mastronicola N, Crecchi A, Bonfiglio L, Procopio C, Solazzi M, Frisoli A. Wearable haptics and immersive virtual reality rehabilitation training in children with neuromotor impairments. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2018, 26(7): 1469–1478 DOI:10.1109/tnsre.2018.2846814

45. Prattichizzo D, Chinello F, Pacchierotti C, Malvezzi M. Towards wearability in fingertip haptics: A 3-DoF wearable device for cutaneous force feedback. IEEE Transactions on Haptics, 2013, 6(4): 506–516 DOI:10.1109/toh.2013.53

46. Ceballos R, Ionascu B, Park W, Eid M. Implicit emotion communication. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018, 14(1): 1–18 DOI:10.1145/3152128

47. Schecter S, Lin W, Gopal A, Fan R, Rashba E. Haptics and the heart: Force and tactile feedback system for cardiovascular interventions. Cardiovascular Revascularization Medicine, 2018, 19(6): 36–40 DOI:10.1016/j.carrev.2018.05.017

48. Demain S, Metcalf C D, Merrett G V, Zheng D Y, Cunningham S. A narrative review on haptic devices: Relating the physiology and psychophysical properties of the hand to devices for rehabilitation in central nervous system disorders. Disability and Rehabilitation: Assistive Technology, 2013, 8(3): 181–189 DOI:10.3109/17483107.2012.697532

49. Navarro E, González P, López-Jaquero V, Montero F, Molina J P, Romero-Ayuso D. Adaptive, multisensorial, physiological and social: The next generation of telerehabilitation systems. Frontiers in Neuroinformatics, 2018, 12: 43 DOI:10.3389/fninf.2018.00043

50. Hale K S, Stanney K M. Haptic rendering-beyond visual computing-deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Computer Graphics and Applications, 2004, 24(2): 33–39 DOI:10.1109/mcg.2004.1274059

51. Sato Y, Ueoka R: Investigating Haptic Perception of and Physiological Responses to Air Vortex Rings on a User's Cheek. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Denver, Colorado, USA, ACM, 2017: 3083–3094 DOI:10.1145/3025453.3025501

52. Taelman J, Adriaensen T, Horst C v d, Linz T, Spaepen A. Textile integrated contactless EMG sensing for stress analysis. In: 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2007, 3966–3969 DOI:10.1109/IEMBS.2007.4353202

53. Wang L, Loh K J, Chiang W H, Manna K. Micro-patterned graphene-based sensing skins for human physiological monitoring. Nanotechnology, 2018, 29(10): 105503 DOI:10.1088/1361-6528/aaa709

54. Xu H, Xiang J X, Lu Y F, Zhang M K, Li J J, Gao B B, Zhao Y J, Gu Z Z. Multifunctional wearable sensing devices based on functionalized graphene films for simultaneous monitoring of physiological signals and volatile organic compound biomarkers. ACS Applied Materials & Interfaces, 2018, 10(14): 11785–11793 DOI:10.1021/acsami.8b00073

55. Pan J P, Tompkins W J. A real-time QRS detection algorithm. IEEE Transactions on Biomedical Engineering, 1985, BME-32(3): 230–236 DOI:10.1109/tbme.1985.325532

56. Brainard D H. The psychophysics toolbox. Spatial Vision, 1997, 10(4): 433–436 DOI:10.1163/156856897x00357

57. Kleiner M, Brainard D, Pell D. What's new in Psychtoolbox-3? PERCEPTION. 2007, 36: 1–16 DOI:10.1068/v070821

58. Pelli D G. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision, 1997, 10(4): 437–442 DOI:10.1163/156856897x00366

59. Treutwein B, Strasburger H. Fitting the psychometric function. Perception & Psychophysics, 1999, 61(1): 87–106 DOI:10.3758/bf03211951

60. Wichmann F A, Hill N J. The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & Psychophysics, 2001, 63(8): 1293–1313 DOI:10.3758/bf03194544

61. Gallace A, Auvray M, Tan H Z, Spence C. When visual transients impair tactile change detection: A novel case of crossmodal change blindness? Neuroscience Letters, 2006, 398(3): 280–285 DOI:10.1016/j.neulet.2006.01.009

62. Gallace A, Tan H Z, Haggard P, Spence C. Short term memory for tactile stimuli. Brain Research, 2008, 1190: 132–142 DOI:10.1016/j.brainres.2007.11.014

63. Gallace A, Tan H Z, Spence C. Multisensory numerosity judgments for visual and tactile stimuli. Perception & Psychophysics, 2007, 69(4): 487–501 DOI:10.3758/bf03193906

64. Gallace A, Tan H Z, Spence C. Can tactile stimuli be subitised? an unresolved controversy within the literature on numerosity judgments. Perception, 2008, 37(5): 782–800 DOI:10.1068/p5767

65. Assumpção L, Shi Z H, Zang X L, Müller H J, Geyer T. Contextual cueing: implicit memory of tactile context facilitates tactile search. Attention, Perception, & Psychophysics, 2015, 77(4): 1212–1222 DOI:10.3758/s13414-015-0848-y

66. Assumpção L, Shi Z H, Zang X L, Müller H J, Geyer T. Contextual cueing of tactile search is coded in an anatomical reference frame. . Journal of Experimental Psychology: Human Perception and Performance, 2018, 44(4): 566–577 DOI:10.1037/xhp0000478

67. Cohen Z Z, Naparstek S, Henik A. Tactile enumeration of small quantities using one hand. Acta Psychologica, 2014, 150: 26–34 DOI:10.1016/j.actpsy.2014.03.011

68. Cohen Z Z, Henik A. Effects of numerosity range on tactile and visual enumeration. Perception, 2016, 45(1): 83–98 DOI:10.1177/0301006615614662

69. Spence C. Multisensory attention and tactile information-processing. Behavioural Brain Research, 2002, 135(1/2): 57–64 DOI:10.1016/s0166-4328(02)00155-9

70. Jorna P G. Spectral analysis of heart rate and psychological state: A review of its validity as a workload index. Biological Psychology, 1992, 34: 237–257 DOI:10.1016/0301-0511(92)90017-O

71. Brookings J B, Wilson G F, Swain C R. Psychophysiological responses to changes in workload during simulated air traffic control. Biological Psychology, 1996, 42(3): 361–377 DOI:10.1016/0301-0511(95)05167-8

72. Recarte M A, Nunes L M. Mental workload while driving: Effects on visual search, discrimination, and decision making. Journal of Experimental Psychology: Applied, 2003, 9(2): 119–137 DOI:10.1037/1076-898x.9.2.119


1. Dangxiao WANG, Yuan GUO, Shiyi LIU, Yuru ZHANG, Weiliang XU, Jing XIAO, Haptic display for virtual reality: progress and challenges Virtual Reality & Intelligent Hardware 2019, 1(2): 136-162