Cite this article: Yang LI, Dong WU, Jin HUANG, Feng TIAN, Hong'an WANG, Guozhong DAI. Influence of multi-modality on moving target selection in virtual reality. Virtual Reality & Intelligent Hardware, 2019, 1(3): 303-315 DOI:10.3724/SP.J.2096-5796.2019.0013

Influence of multi-modality on moving target selection in virtual reality

Yang LI ¹ , Dong WU ^3,⁴ , Jin HUANG ¹ , Feng TIAN ^1,² , Hong'an WANG ^1,² , Guozhong DAI ¹

1. Beijing Key Laboratory of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China

2. State Key Laboratory of Computer Science, Chinese Academy of Sciences, Beijing 100190, China

3. Institute of Remote Sensing and Digital Earth，Chinese Academy of Sciences, Beijing 100190, China

4. University Chinese Academy Sciences, Beijing 100190, China

Abstract

Background

Owing to recent advances in virtual reality (VR) technologies, effective user interaction with dynamic content in 3D scenes has become a research hotspot. Moving target selection is a basic interactive task in which the user performance research in tasks is significant to user interface design in VR. Different from the existing static target selection studies, the moving target selection in VR is affected by the change in target speed, angle and size, and lack of research on some key factors.

Methods

This study designs an experimental scenario in which the users play badminton under the condition of VR. By adding seven kinds of modal clues such as vision, audio, haptics, and their combinations, five kinds of moving speed and four kinds of serving angles, and the effect of these factors on the performance and subjective feelings in moving target selection in VR, is studied.

Results

The results show that the moving speed of the shuttlecock has a significant impact on the user performance. The angle of service has a significant impact on hitting rate, but has no significant impact on the hitting distance. The acquisition of the user performance by the moving target is mainly influenced by vision under the combined modalities; adding additional modalities can improve user performance. Although the hitting distance of the target is increased in the trimodal condition, the hitting rate decreases.

Conclusion

This study analyses the results of user performance and subjective perception, and then provides suggestions on the combination of modality clues in different scenarios.

Keywords： Multimodal ; Moving target selection ; Virtual reality

Content

1 Introduction

Moving target selection is a basic task that selects the moving targets in a planar or a 3D space. In a kind of moving target selection task, the users need to place the pointer in the target space, without selecting the target, in a fixed time window. The user performance is greatly affected by the moving speed and the size of the moving target^[1]. In a second kind, the users need to complete the target selection in a fixed time window that is not limited by the spatial location of the target. Choosing the target too early or too late can lead to failure^[2]. Another moving target selection task requires spatial accuracy in addition to timing accuracy, rhythm, and consistency. Actual dynamic interaction tasks are more complex, and are often affected by many factors such as time, space, modality cues, and environment. Because of the universality of the task, moving target selection has been widely used in games, music, teaching, and other fields. We studied the moving target selection task in specific time and space areas.

There are two major categories used to increase the accuracy of selection in moving target selection^[3]: target enhancement^[4], and cursor enhancement^[5,6]. The ultimate goal is to improve interaction accuracy by increasing the optionality of space and time. According to the theory of cue perception fusion, multimodal cues have a great influence on user motion perception^[7]. It is generally believed that the superposition of multimodal cues (such as vision, auditory, and haptic) that convey consistent information can enhance user perception accuracy and improve the performance. However, in a virtual reality (VR) environment, different modality cues have different expressive abilities to moving objects. The question arises whether user performance in multi-perception modality is better than that in the single perception modality. The comparison between the bimodal and the trimodal conditions also needs to be studied. It is of great significance to study the effect of complementary or redundant information of a different perceptual modality on the interaction efficiency.

Virtual reality scenes have some existing issues like diplopia, vertigo, and lack of perceptual cues, which are different from real scenes. Users typically have inefficient interaction due to the lack of some cues. Traditional VR systems usually employ a combination of visual and auditory modality to improve the accuracy of target selection. With the popularity of data gloves, handheld mobile devices, and other tools, users have started studying haptic information to enrich interactive information, enhance user experience, and improve interaction efficiency^[8]. Previously, researchers focused on the effects of different modalities on static 3D target selection. In this study, we examined the effects of different interaction modalities on user performance and subjective rating of 3D moving target selection in a VR environment.

The main contributions are as follows: first, obtaining a set of empirical data of multimodal conditions effecting the moving target selection in VR; second, analyzing the effects of modality combination, target speed, and target angle on user performance and subjective rating of the moving target selection; third, summarizing and obtaining the effects of a set of multimodal conditions on the moving target selection in a VR environment. Guidelines for the interface design are provided for future interaction design in this field.

2 Related work

2.1　Moving target selection

While tracking the trajectory of a moving target, moving target selection needs to plan the time of target selection, which requires high coordination ability of user motion-perception. Therefore, the level of perception and the motion ability will affect the completion time and the error rate of tasks^[9]. Jagacinski et al. demonstrated that the target moving speed has a significant impact on the completion time of the target selection task. The faster the target moving speed, the more difficult it is to select a user, and the time consumed is longer^[10]. Zhai et al. believe that the initial size of the target does not affect the target selection time, while the size of the target will affect the time required for the target selection when the user clicks on it^[11]. Huang et al. have shown that the endpoint distribution for a 1D moving target selection is greatly influenced by the moving speed and width of the target, however, not by the initial distance between the pointing device and the target^[12]. Lee et al. studied the selection of the moving targets in a fixed time window. The error rate of the selection is related to two factors: the length of the waiting time for the moving targets, and the width of the time window in which the targets can be selected^[2]. To explain the influence of visual factors on the selection results, and enhance the practical use of prediction results, Lee et al. further studied the influence of temporal structure (repeatability, rhythm, etc.) and visual perceptible clues (target moving speed, color, etc.) on the prediction accuracy based on the previous work^[13].

To reduce the impact of speed and size on the completion time and the error rate of the moving target selection in different scenarios, a variety of methods to increase its interaction efficiency are proposed, including reducing the distance from the target and increasing the size of the target. For example, Area Cursor proposed by Kabbash et al. aims to increase the contact range between the cursor and the target and reduce the moving distance between them by increasing the area of the cursor^[14]. Bubble Cursor invented by Grossman et al. makes the cursor contain only one selected target in the selected range through the adjacent attributes of the target, and improves the completion time of the user selection by dynamically changing the selection range of the cursor^[15]. Hasan et al. added the selected area similar to the comet tail to the target according to the moving trajectory and the speed of the target. The size of the tail is related to the moving speed of the target, to increase the selected area of the target^[16].

2.2　Multimodal feedback

Multimodal human-computer interaction is an interdisciplinary field of computer vision, psychology, and many other areas. It is widely used in the field of human-computer interaction. In real scenes, users predominantly interact with the outside environment only according to the information obtained by visual cues, which increases the burden on the visual system and reduces the information acquisition ability of human haptic and auditory sensory systems^[17,18]. Users are affected by different perception modalities under different interaction conditions, and different information resources are obtained by using multi-perception modalities. For example, when the user judges the size, the shape, and the location, the results are more susceptible to visual cues when providing the user with visual-haptic modality cues^[19]. However, for some application scenarios with weak visual modality cues, their judgment is more susceptible to the influence of haptic modality characteristics^[20]. Choosing an appropriate multimodal feedback can increase the robustness of the system and reduce the error rate of interaction^[21,22]. Andy and Stephen studied the effects of multi-modality on the small target selection in GUI, including non-speech audio, tactile, and viscous pseudo-haptic feedbacks, and their combinations on static target acquisition tasks, to guide designers in choosing different modalities to increase the experience of target selection^[23].

3D target selection in VR as a basic task is a hot research topic with the development of the 3D user interface. The accuracy of data acquisition by tracking devices and the perception information provided by the modality feedback limit the actual interaction efficiency. Therefore, various methods are proposed to improve user performance and the subjective feelings of target selection. To reduce the limitation of tracking device accuracy, scholars have studied the relationship between gesture pointing motion and cognitive function^[24,25]，Schmidt presents a pointing-based probabilistic selection algorithm to reduce tracking uncertainty^[26]. Alternatively, the lack of modality information or the difference between the modality information and the real-world cues will reduce user operation performance^[27]. Visual cues are often used to prompt users when selecting targets in a VR environment, for example if the target is selected, the target color changes. Although this method reduces the redundancy of information, it does not improve user interaction performance, and may even reduce user selection efficiency^[28]. Auditory and haptic modalities are often used as auxiliary perception modalities to increase the user’s interaction efficiency and subjective experience. Menelas et al. assisted users to locate and select the targets occluded in virtual environments with haptic or auditory cues^[29]. Cabreir et al. studied the effects of multi-modal cues on the interaction efficiency of static target selection using mid-air gestures by elderly users^[30]. Ariza et al. analyzed the effects of discrete and continuous cues of multimodal feedback on static target selection^[31]. Faeth et al. added auditory, visual, and haptic feedbacks to virtual buttons, and studied the effects of different modalities and their combinations on the time and the error rates of virtual button tasks^[32]. In the real world or the VR environment, although there are many research results on the influence of multi-modality on static target selection, few have studied the influence of modalities on the result of moving target selection. Mould et al. studied the influence of visual cues on the completion time and the error rate of target selection under different occlusion conditions in a 2D graphical user interface^[33]. The target was occluded by other objects whose number and moving speed varied in many ways, including no visual feedback, visual feedback only for selected objects, and visual feedback for all objects. In this study, we focus on the 3D user interface in VR. We study the influence of visual modal cues on moving target selection, in addition to the influence of other modalities on user performance and subjective feelings, which can be used to guide the interactive design of the moving target selection in a virtual environment.

3 Experiment

3.1　Apparatus and users

This study uses the Unity3D engine to draw the experimental environment, Dell computer with Intel Core i5, HTC Vive head mounted display, and its accessories (headphones and handles).

Data were collected from 16 experimental users, 8 males, aged 21 to 24 years, with an average age of 22.6 years. All of them were right-handed, colorless, with normal or corrected visual acuity, healthy, without motor or cognitive impairment, and had no virtual device operation experience. They were all postgraduate students and not paid for their participation.

3.2　Design

Moving target selection has a wide range of applications in VR. This study describes a "badminton playing" scene by adding modality cues to the moving target; studies the effects of visual, auditory, haptic modalities, and their combinations on user selection results. We asked the pitcher to throw the shuttlecock at different angles and speeds to the player, with the speeds unchanged in the flight process, using different types of modality clues to indicate the location of the target. The player should quickly swing to hit the shuttlecock in the optional range, where there is only one choice opportunity in each trial. The shuttlecock will disappear after the user made the swinging action or the shuttlecock will have flown to the user’s center position. If the user hits the target, there will be a "peng" auditory before the shuttlecock disappears. The new shuttlecock will be sent out from the starting position after the shuttlecock disappears for 1 s. The shuttlecock can disappear in two conditions; one is the user hitting the shuttlecock in an optional area, and the other is the shuttlecock reaching the user’s center position.

The starting position of the shuttlecock and the user’s center position are 1.5m high. The horizontal distance between the shuttlecock and the user is 6m. The optional window is a semi-spherical area with the user as the center, 1.5m high with 1 m radius. There are seven modality clues blocks, five speeds, and four angles in each block.

(1) Five moving speeds: 2, 3, 4, 5, and 6 m/s;

(2) Four starting angle position: 15^°, 30^°, 45^°, and 60^°;

(3) Seven modality cues：Visual (V), Auditory (A), Haptic (H), Visual + Auditory (VA), Visual + Haptic(VH), Auditory+ Haptic (AH), Visual + Auditory+ Haptic (VAH).

In the experiment, the speed direction of the shuttlecock is the tangent direction of flight, and it maintains a constant speed from the starting point. The angle is between the starting flight direction and the horizontal direction. The specific flight speed and the angle are shown in Figure 1.

The modality cues are vision, auditory, haptic modalities, and their combinations. The specific stimulus cues of visual, auditory, and tactile modalities are as follows:

(1) Vision Cues: The shuttlecock is green outside the optional area. It will turn red entering the optional area, as shown in Figure 2;

(2) Auditory Cues: The closer the shuttlecock flying distance is to the user, the louder the volume. The volume varies linearly with the moving distance, and the sound changes when the shuttlecock enters the optional area. The hitting sound is “peng”, when the user hits the shuttlecock;

(3) Haptic Cues: The closer the shuttlecock flying distance is to the user, the greater the vibration amplitude of the handle. The vibration amplitude varies linearly with the moving distance. The vibration frequency outside the optional area is 100Hz, and the vibration frequency in the optional range is 200Hz. When the user hits the shuttlecock, the hitting sound “peng” occurs.

3.3　Task

Participants performed seven modality blocks of 20 trials each; the trials are the combination of five speeds and four angles, resulting in 140 trials per participant (7 blocks × 20 trials). The order of the blocks and the trials in each block were selected at random each time. We assessed three response variables: the hit rate during the task, the hit distance when the shuttlecock disappeared in each trial, and the user’s subjective rating of each modalities feedback. At the end of each modality block experiment, the user was asked "the impact of this cue on hitting the shuttlecock,” and the 7-point Likert scale was used to make the user’s subjective score osn seven different modalities. The greater the impact, the higher the score was. Figure 3 is the experimental environment of the user hitting the shuttlecock under a certain modality cue condition. The visual cue of a user hitting the shuttlecock under the VR head mounted display condition is shown in Figure 2. There is no specific picture of tactile cue and auditory cue.

4 Analysis

This section reports the 2240 results collected from seven different modality blocks containing different angles and speeds. Sixteen participants performed 20 trials for each block. The results contain the hit rate and the hit distance. Hit rate refers to whether the user hits the shuttlecock target when swinging the racket, and is expressed by the average hitting times. The higher the hitting rate, the better the interaction efficiency. Hitting distance refers to the distance between the location of the target and the center of the selection window when hitting the shuttlecock. The longer the hitting distance, the higher the efficiency of interactive tasks under different speed conditions, angles, and modality cues. Logistic regression is used to analyze the hit rate, and the hitting distance is analyzed by one-way repeated measures in ANOVA.

4.1　Speed

Figure 4 shows the hit rate results at five speeds. The hit rate is from 0.72 (speed 2m/s) to 0.64 (speed 6m/s), which decreases with the increase in speed. Logistic regression showed that the speed of the target had a significant effect on the hit rate at a significant level of 0.05 (p=0.023).

Figure 5 shows the average hitting distance corresponding to the five speeds. The trend of the average hitting distance is 3>2>4>5>6 (m/s). The hitting distance increases first and then decreases with the increase in speeds. The average hitting distance ranged from 0.643392 (speed 3m/s) to 0.562703 (speed 6m/s), and the average hitting distance of the five speeds was 0.60. One-way repeated measures ANOVA showed that different speeds had significant effects on the hitting distance at a significant level of 0.05 (F(4,0.403)=15.650, p<0.001).

4.2　Angle

Figure 6 shows the hit rate from four angles, ranging from 0.83 (angle 15°) to 0.49 (angle 60°). The hitting rate decreases with the increase in angle. Logistic regression showed that the starting angle position of the shuttlecock had a significant influence on the hit rate (p<0.001) at the significant level of 0.05.

Figure 7 shows the average hitting distance corresponding to the four angles. Overall, there is no significant change in the hitting distance. One-way repeated measures in ANOVA showed that the different angles had no significant effect on the hitting distance at a significant level of 0.05 (F(3,0.044)=1.660, P=0.174>0.05).

4.3　Multimodal cues

Figure 8 shows the hit rate under each modality cue. It ranged from 0.4 (H) to 0.92 (VH). The order of the hit rate from big to small is VH>VA>VAH>V>AH>A>H. Logistic regression showed that the seven modality clues had a significant effect on the hit rate at a significant level of 0.05 (p<0.001).

After adding the visual cues based on the auditory cues, the haptic cues, and the auditory + the haptic cues, the hit rate increased significantly (p<0.001). After adding the auditory cues based on the visual cues, the hit rate increased significantly (p=0.024). After adding the haptic cues based on the visual cues, the hit rate increased significantly (p=0.001). Adding both the auditory and the haptic cues simultaneously based on the visual cues, the hit rate increased significantly (p=0.033).

After adding the auditory cues to the combination of the visual and the haptic cues, the hit rate did not change significantly (p=0.183). After adding the haptic cues to the combination of the visual and the auditory cues, the hit rate did not change significantly (p=0.900).

Figure 9 shows the average hitting distance corresponding to each modality cue. Among different modality cues, the average hitting distance ranged from 0.51(A) to 0.66(VAH). The order of the average hitting distance from big to small is VAH>VH>VA>H>V>AH>A. One-way repeated measures in ANOVA showed that different modality cues had significant effects on the hitting distance at a significant level of 0.05 (F(6,0.688)=28.524, p<0.001).

We used Tukey HSD test to look for the difference between the different modality cues. The hitting distance with the visual cues significantly increased compared with the auditory cues (p=0.001); there was no significant difference between the visual cues and the haptic cues (p>0.05). The hitting distance with the haptic cues significantly increased compared with the auditory cues (p<0.05); the hitting distance with the visual cues significantly increased compared with the auditory and the haptic cues (p=0.001); and the hitting distance with the visual and the haptic cues significantly increased compared with the auditory cues (p=0.001). There was no significant difference between the visual + the auditory cues and the haptic cues (p>0.05).

Based on the visual cues, the hitting distance increased significantly by adding the auditory cues (p=0.018); the haptic cues or the auditory + the haptic cues increased significantly (p<0.001). There was no significant difference in the hitting distance while using the haptic cues and the auditory cues (p>0.05).

Adding the auditory cues on the basis of the haptic cues significantly reduced the hitting distance (p<0.001); adding the visual cues had no significant difference (p=0.058); the hitting distance increased significantly by adding the visual cues compared with adding the auditory cues, (p<0.001); adding the visual + the auditory cues significantly increased the hitting distance (p<0.001).

When the visual cues or the visual + the haptic cues were added on the basis of the auditory cues, the hitting distance increased significantly (p<0.05); when the haptic cues were added on the basis of the auditory cues, the hitting distance did not change significantly (p>0.05); adding the visual cues on the basis of the auditory cues, compared with adding the haptic cues, the hitting distance increased significantly (p<0.001).

Adding the haptic cues on the basis of the visual cues + the auditory cues, the hitting distance increased significantly (p=0.017); adding the auditory cues on the basis of the visual + the haptic cues, the hitting distance did not change significantly (p>0.05); adding the visual cues on the basis of the auditory + the haptic cues, the hitting distance increased significantly (p<0.001).

4.4　Subject rating

Figure 10 shows the subjective rating of different modality cues. Among different modality cues, the score ranged from 1.4(H) to 6.81(VAH). The order of the mean score from big to small is VAH>VH>VA>H>V>AH>A, SD is VA>V>AH>VH>VAH=A>H. VAH has the highest mean score, reaching 6.81, followed by VA and VH with a mean score 5.6 and 5.1, respectively. Visual modality has a score of 4.4. Above numerical results, show that users prefer the visual-related modality cues and adding additional modality cues based on vision modality will increase the subjective experience of the users. The users can receive more accurate location information and thus hit the target quickly. The lowest score is for the haptic cues, 1.4, followed by the auditory mode, with a score of 1.8. Users are full of uncertainties about the auditory or the tactile modality cues, which affects users' subjective sense. The lowest standard deviation is for the haptic cues, followed by the auditory and the VAH modality cues, that is, the user's most favorite modality cues are the VAH and the least favorite are the haptic modality cues. The auditory score have a small difference from the haptic modality score. In conclusion, except for the AH score of 3.2, users generally think that the trimodal conditions are better than the bimodal conditions, and the bimodal conditions are better than the single modality.

5 Discussion

Ignoring the influence of different modality cues, velocities, and angles, the average hit rate was 0.683, among which the changes of the modality cues (p<0.001), velocities (p=0.023) and angles (p<0.001) had significant effects on the hit rate. Ignoring the influence of different modality cues, velocities, and angles, the average hitting distance was 0.60 m, the changes in the modality cues (F(6, 0.688)=28.524, p<0.001) and the velocities (F(4,0.403)=15.650, p<0.001) had significant effects on the hitting distance, while the changes in the angles (F(3,0.044)=1.660, p=0.174) had no significant effects on the hitting distance.

5.1　Effect of speed on “hitting the shuttlecock” task

The analysis results showed that the speed of the shuttlecock has a significant impact on the hit rate and hit distance. The hit rate decreases with increasing speed; the hitting distance increases first, and then decreases, with the increase in speed, indicating that slower the shuttlecock, the better the task effect. When the speed is too slow, the hitting distance will also decrease, that is, the hitting target effect will be worse.

5.2　Effect of angle on "hitting the shuttlecock" task

The results showed that the angle of the shuttlecock has a significant effect on the hit rate, however has little effect on the hit distance. With the increasing of the starting angle position of the shuttlecock, the hitting rate decreases significantly, while the hitting distance does not change significantly. The hitting rate results indicate that bigger the angle, the more difficult it is for the users to determine the moving distance, which affects the accuracy of hitting the target. The change in the starting angle position affects the accuracy of hitting the shuttlecock, however it does not affect the reaction time.

5.3　Effect of modality cues type on the hit rate

The results showed that the type of modality cue has a significant impact on the hit rate. Bimodal conditions VH and VA have a superior hit rate compared with the other modality conditions (trimodal condition, single modality). However, there was no significant difference between the VAH and the bimodal conditions containing the visual cues in the experiment. Adding another cue based on the bimodal conditions combining the visual cues would not significantly improve the hit rate. VH or VA can improve the hit rate compared with a single visual modality. In general, the hitting rate associated with the visual modality cues is relatively high because it is more definite, therefore the visual modality hitting rate is higher than the auditory, the tactile and the AH. Adding auditory or haptic cues based on the visual cues will have a higher hitting rate as it provides additional sensory information. Adding auditory and haptic modality simultaneously has no significant impact on the hit rate compared with the addition of a single modality because it increases the user's cognitive load and then affects the user's judgment results, thereby reducing the hit accuracy. In the real world, users rarely select targets through auditory and haptic modality cues; hence, the hit rate is relatively low compared with visual cues.

5.4　Effect of modality cues type on hit distance

The results showed that the type of modality cues has a significant effect on the distance of hitting.

5.4.1 　 Comparison with single modality

Auditory cues significantly reduce the hitting distance compared with the visual or the haptic modality cues, that is, the hitting reaction time is longer. There was no significant difference between the visual and the haptic cues in this experiment. Therefore, visual cues and tactile cues can obtain larger hitting distance in single modality cues.

5.4.2 　 Comparison with bimodal conditions

There is no significant difference when compared with the bimodal conditions, VH and VA, by adding the tactile or the auditory modality cues on the basis of the visual cues; adding visual cues on the basis of the haptic cues, VH, has more distance than adding the auditory cues AH; adding visual cues on the basis of the auditory cues VA has more distance than adding haptic cues AH. The bimodal conditions, VA and VH, have larger hitting distance, which is significantly different from the AH.

5.4.3 　 Two cases of comparison with the bimodal condition and the single modality

In the first case, the single modality is included in the bimodal conditions. VA gets a larger hitting distance than the single modality cues V or A; VH gets a larger hitting distance than the single visual cues V, however there is no significant difference when compared with the single haptic cues H; AH has no significant difference compared with the single auditory cues A. The hitting distance decreases when compared with the single tactile cue and the hitting reaction time becomes longer. That is, VA, VH, and H have larger hitting distances.

In the second case, the single modality is not included in the bimodal conditions. AH has a smaller hitting distance compared with the single visual cues V; VH has a larger hitting distance compared with the single auditory cues A, and VA has no significant difference from the single haptic modality cues H. That is, V and VH have larger batting distances.

Therefore, in the comparison of single modality and bimodal conditions, VH, VA, and H have larger hitting distances.

5.4.4 　 Comparison between the trimodal conditions and the single modality

Trimodal conditions have larger hitting distance compared to the single modality cues.

5.4.5 　 Comparison between the trimodal conditions and the bimodal conditions

Trimodal conditions can obtain larger hitting distances compared with VA or AH; however, it has no significant impact on the hitting distance compared with the VH. Comparing the two, VAH and VH have larger hitting distances.

In conclusion, except for AH, more the modality cues, larger are the hitting distance. Effect of the trimodal conditions on the hitting distance is greater than the bimodal conditions, which is in turn greater than the single modality conditions. Visual modality cues were not significantly different from the haptic modality cues as they all have a larger hitting distance. It may be because, as the user shakes the handle, the vibration increases the user's sensory experience, and thus shortens the user's reaction time.

5.5　Subjective

Subjective scale results showed that the users believe that what you see is true, so the visual-related modal scores are relatively higher. In addition, the users generally believe that the more the modality cues, the more assistant information for the hitting position will be provided, thus improving the accuracy of hitting. Therefore, the trimodal condition score is the highest, while the bimodal conditions VA and VH were slightly lower. Besides observing the world with the eyes, users also communicate and locate the surrounding environment with sounds, however, rarely with the vibrations. Therefore, the users are more familiar with the method of sound location compared with the haptic modality. Therefore, they have a higher degree of trust in auditory modality than haptic modality, thus the scores associated with the haptic modality are lower than other modality conditions, e.g., the scores of VA are higher than those of VH, auditory modality are higher than haptic modality. The subjective scores have two characteristics, one is that the familiarity modality will enhance the users’ subjective perception, with familiarity ranked as visual modality > auditory modality > tactile modality. The other is that the users generally believed that multi-modal cues would increase the amount of information, except for AH which has lesser daily contact. It was generally believed that the impact on the interaction efficiency compliance with the case that trimodal conditions > bimodal conditions > single modality.

6 Conclusion

The data analysis results showed that faster the shuttlecock flies, the lower the hit rate, and if the shuttlecock is too fast or too slow, it will affect the user's judgment time and reduce the hit distance. Moreover, the bigger the angle, the lower the hit rate. That is to say, when a small shuttlecock flies into a larger bending degree, it is not easy for the user to hit the target, and the hit distance is not affected by the angle. Multi-modal conditions can provide more information compared with the single modality interaction, reduce the time to complete the tasks, and affect the hitting rate. The results showed that the hitting rate of the moving target selection related to the visual modality is relatively high, and the hitting rate of the VH and the VA are the highest, however there is no significant difference between them and the VAH.

The impact of other modalities on the hitting distance is similar to the user's subjective feeling except for the AH, which conforms to trimodal conditions>bimodal conditions>single modality conditions. Although the user's unfamiliarity with the haptic modality affects the subjective interactive feeling, the direct vibration of the handle stimulates the user's sensory senses, which makes the user to respond quickly to the batting, so that it has a larger hit distance. Although the multimodal conditions increase the information and enhance the subjective feelings, they also make the users have the illusion of judgment, increase the cognitive load, and then do not improve the accuracy of interaction. It is not that more the modality conditions, higher the user’s performance.

In practical applications, the modality conditions V+A or V+H can be selected according to the environment to enhance the interaction efficiency of users' choice of moving target selection in VR, and guide the multi-modal interface in a virtual environment. In the future, 3D audio headphones can be selected to enhance the spatial sense of auditory modality. The appropriate haptic feedback device providing different intensity levels can be selected or manufactured according to the task. In the research of moving target selection, we chose the appropriate modality conditions to increase the interaction efficiency. At the same time, we studied the influence of intensity levels and modality changes on the user interaction efficiency and subjective feelings.

Reference

Gunn T J, Irani P, Anderson J. An evaluation of techniques for selecting moving targets. In: Proceedings of the 27th international conference extended abstracts on Human factors in computing systems. Boston, MA, USA, ACM, 2009, 3329–3334 DOI:10.1145/1520340.1520481

Lee B, Oulasvirta A: Modelling Error Rates in Temporal Pointing. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose, California, USA, ACM, 2016, 1857–1868 DOI:10.1145/2858036.2858143

Ilich M V. Moving target selection in interactive video. Tanpakushitsu Kakusan Koso Protein Nucleic Acid Enzyme, 2009, 44(12 Suppl): 1682–1690

MacKenzie I S, Buxton W. Extending Fitts' law to two-dimensional tasks. In: Proceedings of the SIGCHI conference on Human factors in computing systems. Monterey, California, USA, ACM, 1992 DOI:10.1145/142750.142794

Tresilian J R. Hitting a moving target: Perception and action in the timing of rapid interceptions. Perception & Psychophysics 2005, 67(1): 129–149 DOI:10.3758/bf03195017

Su X J, Au O K C, Lau R W H. The implicit fan cursor: a velocity dependent area cursor. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014: 753–762 DOI:10.1145/2556288.2557095

Ernst M O, Banks M S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 2002, 415(6870): 429–433 DOI:10.1038/415429a

Vanacken L, Raymaekers C, Coninx K. Evaluating the influence of multimodal feedback on egocentric selection metaphors in virtual environments//Haptic and Audio Interaction Design. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006: 12–23 DOI:10.1007/11821731_2

Shadmehr R, Smith M A, Krakauer J W. Error correction, sensory prediction, and adaptation in motor control. Annual Review of Neuroscience, 2010, 33(1): 89–108 DOI:10.1146/annurev-neuro-060909-153135

10.

Jagacinski R J, Repperger D W, Ward S L, Moran M S. A test of fitts' law with moving targets. Human Factors: the Journal of the Human Factors and Ergonomics Society 1980, 22(2): 225–233 DOI:10.1177/001872088002200211

11.

Zhai S, Conversy S, Beaudouin-Lafon M, Guiard Y. Human on-line response to target expansion. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Ft. Lauderdale, Florida, USA, ACM, 2003, 177–184 DOI:10.1145/642611.642644

12.

Huang J, Tian F, Fan X M, Zhang X, Zhai S M. Understanding the uncertainty in 1D unidirectional moving target selection. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, ACM, 2018, 1–12 DOI:10.1145/3173574.3173811

13.

Lee B, Kim S, Oulasvirta A, Lee J I, Park E. Moving target selection. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal QC, Canada, ACM, 2018, 1–12 DOI:10.1145/3173574.3173804

14.

Kabbash P, Buxton W A S. The “prince” technique: Fitts' law and selection using area cursors. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press, 1995, 273–279

15.

Grossman T, Balakrishnan R. The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2005: 281–290 DOI:10.1145/1054972.1055012

16.

Hasan K, Grossman T, Irani P. Comet and target ghost: techniques for selecting moving targets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Vancouver, BC, Canada, ACM, 2011, 839–848

17.

Oviatt S, DeAngeli A, Kuhn K. Integration and synchronization of input modes during multimodal human-computer interaction. In: Referring Phenomena in a Multimedia Context and their Computational Treatment. Madrid, Spain, Association for Computational Linguistics, 1997, 1–13 DOI:10.3115/1621585.1621587

18.

Oviatt S. Multimodal interfaces//The Human-Computer Interaction Handbook. CRC Press, 2007, 413–432 DOI:10.1201/9781410615862.ch21

19.

Hay J C, Pick H L, Ikeda K. Visual capture produced by prism spectacles. Psychonomic Science 1965, 2(1–12): 215–216 DOI:10.3758/bf03343413

20.

Heller M A. Haptic dominance in form perception with blurred vision. Perception 1983, 12(5): 607–613 DOI:10.1068/p120607

21.

Jaimes A, Sebe N. Multimodal human–computer interaction: A survey. Computer Vision and Image Understanding 2007, 108(1/2): 116–134 DOI:10.1016/j.cviu.2006.10.019

22.

Oviatt S. Mutual disambiguation of recognition errors in a multimodel architecture. In: Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit. Pittsburgh, Pennsylvania, USA, ACM, 1999 DOI:10.1145/302979.303163

23.

Cockburn A, Brewster S. Multimodal feedback for the acquisition of small targets. Ergonomics 2005, 48(9): 1129–1150 DOI:10.1080/00140130500197260

24.

MacKenzie C L, Marteniuk R G, Dugas C, Liske D, Eickmeier B. Three-dimensional movement trajectories in fitts' task: implications for control. The Quarterly Journal of Experimental Psychology Section A 1987, 39(4): 629–647 DOI:10.1080/14640748708401806

25.

Whitney D, Westwood D A, Goodale M A. The influence of visual motion on fast reaching movements to a stationary object. Nature 2003, 423(6942): 869–873 DOI:10.1038/nature01693

26.

Schmidt G, Baillot Y, Brown D G, Tomlin E B, Swan J E. Toward Disambiguating Multiple Selections for Frustum-Based Pointing. In: 3D User Interfaces (3DUI'06). Alexandria, VA, USA, IEEE, 2006, 87–94 DOI:10.1109/tridui.2006.1618277

27.

Mine M R, BrooksJr. F P, Sequin C H. Moving objects in space: exploiting proprioception in virtual-environment interaction. In: Proceedings of the 24th annual conference on Computer graphics and interactive techniques. 1997, 19–26 DOI:10.1145/258734.258747

28.

Poupyrev I, Ichikawa T, Weghorst S, Billinghurst M. Egocentric object manipulation in virtual environments: empirical evaluation of interaction techniques. Computer Graphics Forum 1998, 17(3): 41–52 DOI:10.1111/1467-8659.00252

29.

Menelas B A J, Picinali L, Bourdot P, Katz B F G. Non-visual identification, localization, and selection of entities of interest in a 3D environment. Journal on Multimodal User Interfaces 2014, 8(3): 243–256 DOI:10.1007/s12193-014-0148-1

30.

Cabreira A T, Hwang F. Evaluating the effects of feedback type on older adults' performance in mid-air pointing and target selection. In: Proceedings of the Symposium on Spatial User Interaction. Berlin, Germany, ACM, 2018 DOI:10.1145/3267782.3267933

31.

Ariza O, Bruder G, Katzakis N, Steinicke F. Analysis of proximity-based multimodal feedback for 3D selection in immersive virtual environments. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. Tuebingen/Reutlingen, Germany. New York, USA, IEEE, 2018 DOI:10.1109/vr.2018.8446317

32.

Faeth A, Harding C. Emergent effects in multimodal feedback from virtual buttons. ACM Transactions on Computer-Human Interaction 2014, 21(1): 1–23 DOI:10.1145/2535923

33.

Mould D, Gutwin C. The effects of feedback on targeting with multiple moving targets. In: Proceedings of Graphics Interface. 2004, 25–32