Chinese

2019,  1 (1):   1 - 9   Published Date：2019-2-20

DOI: 10.3724/SP.J.2096-5796.2018.0007

Abstract

Background
Within a virtual environment (VE) the control of locomotion (e.g., self-travel) is critical for creating a realistic and functional experience. Usually the direction of locomotion, while using a head-mounted display (HMD), is determined by the direction the head is pointing and the forward or backward motion is controlled with a hand held controllers. However, hand held devices can be difficult to use while the eyes are covered with a HMD. Free hand gestures, that are tracked with a camera or a hand data glove, have an advantage of eliminating the need to look at the hand controller but the design of hand or finger gestures for this purpose has not been well developed.
Methods
This study used a depth-sensing camera to track fingertip location (curling and straightening the fingers), which was converted to forward or backward self-travel in the VE. Fingertip position was converted to self-travel velocity using a mapping function with three parameters: a region of zero velocity (dead zone) around the relaxed hand position, a linear relationship of fingertip position to velocity (slope or $\beta$) beginning at the edge of the dead zone, and an exponential relationship rather than a linear one mapping fingertip position to velocity (exponent). Using a HMD, participants moved forward along a virtual road and stopped at a target on the road by controlling self-travel velocity with finger flexion and extension. Each of the 3 mapping function parameters was tested at 3 levels. Outcomes measured included usability ratings, fatigue, nausea, and time to complete the tasks.
Results
Twenty subjects participated but five did not complete the study due to nausea. The size of the dead zone had little effect on performance or usability. Subjects preferred lower $\beta$ values which were associated with better subjective ratings of control and reduced time to complete the task, especially for large targets. Exponent values of 1.0 or greater were preferred and reduced the time to complete the task, especially for small targets.
Conclusions
Small finger movements can be used to control velocity of self-travel in VE. The functions used for converting fingertip position to movement velocity influence usability and performance.

Content

1 Introduction
As an emerging field, the immersive nature of virtual reality (VR) has the potential to dramatically change and enhance education, vocational training, design, travel, communications, gaming, and other activities. Identifying optimal methods for efficiently interacting with and navigating through a virtual environment (VE) is critical for the success of this technology. Locomotion, or the ability to move around the VE space using a head-mounted display (HMD), can be controlled in a variety of ways, such as physically walking to move through the VE, teleporting, or “scene-in-hand” methods[1]. Physical walking while wearing a HMD may presents safety issues since HMDs may be wired, the real space may have objects to collide with, and orientation (up-down) may be distorted. Teleporting, or virtually jumping to a new location in VE, can be disorienting and, at least initially, difficult for the user to control. The most common method for controlling self-travel is to point the head in the desired direction of motion and use a hand-held controller to move forward or backward in the VE. This can be confusing if the controller is also used to control the view or other tasks, such as object manipulation[2]. In addition, the controller cannot be seen with the HMD and therefore may be difficult to properly grasp and use. An alternative is to use free hand or finger gestures that are tracked with a camera or wireless data-glove to control self-travel. This method may be less fatiguing and safer, since there is no device to hold and the hands are free to provide the user with an extra level of security from falls or collisions in the real environment[3,4]. However, hand gestures may be misinterpreted by the technology leading to unintended actions. Advantages and disadvantages of different methods of input can be evaluated with usability studies while subjects perform tasks in VE.
Using hand or arm gestures for controlling locomotion in VE has been explored to a limited extent. Gestures designed based on natural language concepts can translate into understood commands thereby improving locomotive control while reducing cognitive load. However, in order to achieve good usability with low fatigue, the cognitive and physical demands of mapping of hand gestures to motion control should be thoroughly evaluated. For example, the design of the hand gesture may make it easy or difficult to control movement over very small or large distances[5]. Yet users are familiar with velocity-controlled movement in everyday travel, such as accelerating or decelerating a car with a foot pedal; thus velocity control using hand gestures can translate naturally from the real world to a virtual one.
A previous study by our group investigated different mapping functions for translating hand position to forward and backward movement in VE[6]. Forward motion in VE occurred when the hand and forearm were moved forward away from the body, and backward motion occurred when the hand was moved toward the body. How the position of the hand controlled velocity (e.g., shape of the curve mapping hand position to velocity) influenced productivity, error and user preference. However, large arm movements can be fatiguing. The purpose of this study was to investigate throughput (i.e., productivity and error) and usability when small finger motions are used to control velocity; small finger motions may be associated with less fatigue than large arm or whole-body motions. Several parameters were studied for mapping fingertip position to velocity while subjects moved forward in a VE toward targets of different sizes.
2 Methods
This was a within-subjects laboratory study, where participants wore a HMD and were immersed in a virtual environment and were required to repeatedly move forward along a road and stop on a target. The velocity of their movement on the road was controlled with different finger postures–curling (flexing) or extending the fingers. Participants were instructed to move as rapidly as possible to each target. Participants were 18 years of age or older and had no arm or shoulder injuries during the past month. The university Committee on Human Research approved the study and subjects signed an informed consent.
2.1 Equipment
Subjects were seated and wore a short sleeve shirt. They were fit with a VE headset (Oculus Rift, DK2, Menlo Park, CA) that was connected to a PC. A depth-sensing camera (Leap Motion, San Francisco, CA) was mounted on the front of the HMD and tilted down 13.4 degrees so that it viewed the hands from above in order to track fingertip position (Figure 1).
In VE, subjects viewed a road that stretched forward to infinity (Figure 2). Their position on the road was marked with a red bar. Ahead of them on the road was a target sphere―the near and far edges of the sphere were marked on the road by 2 white lines. Their task was to move themselves (red bar), as rapidly as possible, forward and to stop between the 2 white lines. After successfully stopping on the target, a new target appeared in the distance on the road. The graphics and task were designed in Unity and are described in a prior paper[6]. The participant could look around the environment by moving their head but they were only allowed to move forward or backward along the road.
Forward and backward movement in the VE was controlled by fingertip position. Initially, the researcher explained the setup and goals of the study and demonstrated the gestures for moving forward or backward. Subjects were instructed to position their right hand in front at the level of the solar plexus with the upper arm relaxed by the torso and the forearm held parallel to the ground. The palm of the hand faced to the left. The hand posture began from a resting (zero) position with the fingers gently curved. The 4 fingers were moved forward (extension) to indicate forward movement in VE or moved toward the palm toward the posture of a fist (flexion) to indicate backward movement in VE (Figure 3). The fingers were moved back to the resting position to stop movement in VE. Subjects practiced the task with each test condition until they were able to comfortably control movement.
2.3 Task conditions―target distance and size
For each trial there were 30 targets to land on. The distance between targets was 3m, 15m or 27m, as measured in Unity. For each trial, the distances between targets for the last 24 targets were randomized but balanced to ensure that each distance was included 8 times. For a trial, the target size was either 1m or 2 m.
2.4 Three gesture parameters tested―beta, dead zone, and exponent
Three parameters (beta coefficient, dead zone, and exponent), for translating fingertip position to forward or backward velocity, were evaluated, each at 3 levels. The depth camera was used to measure the distance from the distal end of the index finger of the participant’s right hand to the center of the palm (using Leap Motion APIs). The value in millimeters was scaled by a factor of 2.74 to reduce noise from finger tremor. A zero resting position was determined at the beginning of the experiment when the participant held their hand in a relaxed, gently curled, comfortable finger position.
The 3 dead zone dimensions tested were: 10mm, 25mm, and 40mm (dzw). If the fingertip position was within the dead zone of the zero position (zc), there was no movement (Figure 4).
When the fingertips were extended forward beyond the dead zone, the velocity of subject movement (red bar) increased linearly with fingertip distance. When the fingertips moved in the other direction, toward the palm, and moved beyond the dead zone, the subject (red bar) moved backward. A control transfer function (Equation 1) mapped fingertip position linearly to velocity.
$velocity={\left(\beta ×\gamma \right)}^{}$
Where $\beta$ =slope coefficient=velocity/fingertip position; $\gamma$ =distance of the fingertip from the boundary of the dead zone such that $\gamma$ =(position×2.74)−(zc+dcw) for forward motion and $\gamma$ =(zcdzw)−(position×2.74) for backward motion; zc is the zero-center distance value, and dzw is the dead zone width. The three $\beta \text{'}s$ tested were 12m/s, 21m/s, and 30m/s per millimeter.
The exponent parameters ( $\alpha$ ) tested were 0.5, 1.0, and 1.7, and followed equation 2.
$velocity={\left(\beta ×\gamma \right)}^{a}$
When varying the levels of one parameter, the other parameters were fixed at their middle values, e.g., 21m/s for $\beta$ , 25mm for dzw, and 1.0 for $\alpha$ ―the study was not a full factorial design. The order of testing of the three parameters was beta-coefficient, then dead zone width, then exponent. The order of testing of the 3 levels within a parameter was randomized for each subject. For each level of a parameter, subjects completed one trial (30 targets) with large targets (2m) and one trial (30 targets) with small targets (1m). The order of testing between the three parameters was not randomized because it is easier to learn beta-coefficient first, then dead zone and finally exponent.
2.5 Outcome measures
The trial time was the time to complete the last 24 of the 30 targets in the trial; the first 6 targets were ignored and considered warm-up. After all 6 trials for a parameter (3 levels and 2 target sizes) were completed, participants removed the HMD and filled out a usability questionnaire. The questionnaire rated four statements: “I had excellent control”, “I had no shoulder fatigue”, “I had no hand-arm fatigue”, and “I did not feel sick” using a 5-point visual analog scale (1=strongly disagree to 5=strongly agree). Subjects also ranked their preferences for each level (1 being the condition liked the most, 3 the least). Subjects were also invited to provide open-ended comments on the levels tested.
2.6 Statistical analysis
For each parameter, differences in completion time between levels were compared using repeated measures ANOVA and stratified for the small and large targets. If significant, follow-up pair-wise comparisons were performed with the Tukey test to adjust for multiple comparisons. Differences between levels within a parameter for subjective usability and preference ratings were evaluated using the Skillings-Mack test, and, if significant, pair-wise comparisons were performed empirically using a Bonferroni adjustment for multiple comparisons.
3 Results
Fifteen subjects completed the study; 5 additional participants were unable to complete the study due to nausea so their data was not included in the analysis. Seven of the 15 were female, one was left-handed, and the age range was 18 to 25 years.
Summary results comparing the outcomes between levels for the 3 parameters are presented in Tables 1 to 6. For each parameter there are 2 tables one for the small target size and one for the large target size.
Results for beta-coefficient parameter differences for small targets
Beta coefficient (small target) (m/s) P value
$\beta$ =12 $\beta$ =21 $\beta$ =30
Total time (s) 74.5 (48.0) 62.6 (18.3) 82.0 (31.4) 0.18
Excellent control1 4.5 (0.5)ab 3.8 (0.9)a 3.6 (0.7)b 0.0001
No shoulder fatigue1 4.2 (0.9) 4.1 (1.0) 4.2 (1.1) 0.65
No hand fatigue1 3.9 (1.1) 3.9 (1.1) 3.8 (1.1) 0.70
No motion sickness1 4.4 (0.8) 4.4 (1.1) 4.3 (1.2) 0.20
Preference2 1.4 (0.7)a 1.9 (0.6)b 2.7 (0.5)ab 0.002
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
Results for beta-coefficient parameter differences for large targets
Beta coefficient (large target) (m/s) P value
$\beta$ =12 $\beta$ =21 $\beta$ =30
Total time (s) 48.0 (11.5)a 58.0 (21.6) 61.0 (23.8)a 0.011
Excellent control1 3.9 (0.8)a 3.7 (1.0)b 3.1 (1.0)ab 0.012
No shoulder fatigue1 4.3 (1.0) 4.3 (1.1) 4.2 (0.9) 0.38
No hand fatigue1 4.2 (1.0) 4.2 (0.9) 4.1 (1.1) 0.80
No motion sickness1 4.3 (1.1) 4.5 (1.1) 4.3 (1.2) 0.34
Preference2 1.5 (0.6)a 1.9 (0.7)b 2.6 (0.7)ab 0.010
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
Results for dead zone width parameter differences for small targets
Dead zone width (small target ) (mm) P value
dzw=10 dzw=25 dzw=40
Total time (s) 71.9 (22.8) 71.0 (32.5) 73.0 (38.2) 0.94
Excellent control1 3.5 (0.7) 3.8 (0.9) 4.2 (0.9) 0.15
No shoulder fatigue1 4.4 (1.0) 4.2 (0.9) 4.3 (1.0) 0.59
No hand fatigue1 4.1 (1.1) 4.3 (0.8) 4.3 (1.0) 0.41
No motion sickness1 4.3 (1.1) 4.3 (1.2) 4.2 (1.1) 0.10
Preference2 2.2 (0.9) 2.2 (0.7) 1.6 (0.9) 0.19
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
Results for dead zone width parameter differences for large targets
Dead zone width (large target) (mm) P value
dzw=10 dzw=25 dzw=40
Total time (s) 54.5 (16.6) 50.5 (17.5) 46.9 (18.8) 0.18
Excellent control1 3.3 (1.0) 3.8 (0.9) 3.7 (1.1) 0.26
No shoulder fatigue1 4.0 (1.3) 3.9 (1.3) 3.9 (1.4) 0.31
No hand fatigue1 3.9 (1.4) 3.9 (1.3) 4.1 (1.5) 0.53
No motion sickness1 4.3 (1.3) 4.3 (1.2) 4.3 (1.2) 0.81
Preference2 2.4 (0.8) 1.7 (0.8) 1.9 (0.7) 0.19
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
Results for exponent parameter differences for small targets
Exponent value (small target) P value
$\alpha$ =0.5 $\alpha$ =1.0 $\alpha$ =1.7
Total time (s) 92.9 (61.6)ab 65.5 (35.7)a 60.1 (19.4)b 0.017
Excellent control1 3.7 (0.9) 4.2 (0.9) 3.7 (1.1) 0.077
No shoulder fatigue1 4.2 (1.0) 4.4 (1.0) 4.4 (0.9) 0.072
No hand fatigue1 4.1 (1.0) 4.3 (1.1) 4.3 (1.0) 0.13
No motion sickness1 4.3 (1.2) 4.4 (1.1) 4.1 (1.2) 0.62
Preference2 2.4 (0.7) 1.6 (0.8) 2.0 (0.8) 0.12
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
Results for exponent parameter differences for large targets
Exponent value (large target) P value
$\alpha$ =0.5 $\alpha$ =1.0 $\alpha$ =1.7
Total time (s) 50.0 (13.9) 43.4 (8.9) 54.0 (27.3) 0.17
Excellent control1 3.1 (1.1)ab 3.9 (0.6)a 4.3 (1.0)b 0.001
No shoulder fatigue1 3.9 (1.2) 3.9 (1.3) 4.1 (1.2) 0.74
No hand fatigue1 3.7 (1.4)a 3.9 (1.0) 4.2 (1.0)a 0.019
No motion sickness1 4.2 (1.3) 4.3 (1.0) 4.2 (1.5) 0.41
Preference2 2.8 (0.4)ab 1.5 (0.5)a 1.7 (0.8)b 0.001
1 Subjective Ratings: 1=strongly disagree, 5=strongly agree (higher is better); 2 Preference: 1=favorite and 3=least favorite. Values are presented as mean±SD. Groups with the same superscript in a row are significantly different from each other (n=15).
4 Discussion
Small finger motions can control locomotion in VE, but the usability was influenced by 2 of the 3 parameters tested for mapping fingertip position to velocity. While the size of the dead zone had no significant effect on productivity, usability or preference, three of the participants reported that the smallest dead zone size (±10mm) made it difficult to control movement. In addition, there was a trend for participants to prefer the middle or larger size dead zone. It may be that the smallest dead zone size is near to a limit that challenges finger motor control. The participants in the study did not have prior experience using finger motion to control velocity; more experience may have led to a different outcome. In a prior study, evaluating the same parameters as this study, but with velocity controlled by moving the whole hand and forearm forward or backward, the dead zone size also had little effect on productivity, usability or preference[6].
The slope of mapping of fingertip position to velocity, $\beta$ , significantly influenced usability. For the large target size, the small $\beta$ (12m/s) was associated with the fastest time to complete the tasks compared to the large $\beta$ (30m/s) (Table 2). However, for the small target size there was no significant difference in task time between $\beta$ levels (Table 1). The small and midsize $\beta$ (12m/s and 21m/s) provided the best control, were the most preferred and were associated with the least fatigue. This difference was most evident for the large target size. The highest $\beta$ (30m/s) led to overshooting the target and therefore increased the time to complete the task and increased frustration for participants. In addition, the slower speed may provide a more comfortable experience for those prone to motion sickness or disorientation. The VR experience was new for most participants and they may have been inclined to move with caution; therefore, the optimal $\beta$ may change based on the users experience in the VR environment. In the prior study, involving moving the whole arm to control velocity, the effect of $\beta$ on performance was influenced by target size and target distance[6]. Small $\beta$ (7m/s) was associated with the best performance for small target size (1m) and short distance between targets (3m and 9m), while mid and large $\beta$ (14m/s and 21m/s) led to the best performance for larger distances between targets (15m) and the large target size (2m).
Overall, for the exponent ( $\alpha \right)$ , the usability of the lowest value ( $\alpha$ =0.5) was poor compared to the two larger exponent values, and there was little difference in usability between the two larger values ( $\alpha$ =1.0, 1.7). For the small targets, the completion time was 50% longer for $\alpha$ =0.5 compared to the larger values (Table 5). In addition, for the large targets, the 0.5 exponent was associated with poor subjective control and fatigue and was the least preferred. In the prior study involving whole arm movement[6], the exponent of 2.0 was associated with the slowest time to complete the task and there was little difference between the 1.0 and 1.4 exponents. Subjects in the prior study noted that with the 1.0 exponent there was an abrupt forward movement when the hand moved beyond the dead zone but with exponents of 1.4 or 2.0 the transition was smoother. However, with the exponent of 2.0 high speeds could occur too easily and were associated with loss of control. Overall, the findings of both studies suggest that some exponent greater than 1.0 may be useful but not too much. Perhaps values between 1.2 and 1.7 should be studied in more detail.
Subjects were immersed in the virtual world for most of the 1.5h of the study. Five of the 20 subjects were unable to complete the study due to motion sickness. This was somewhat surprising since movement direction was only forward or backward and participants did not have to turn or move up or down during forward travel.
Some study limitations should be noted. First, the depth camera was not always able to track the fingertips when the fingers were curled in the hand and this led to unexpected movements of the subject in VE. Second, participants were relatively young, self-selected and comfortable trying new technology. The study findings may be different for an older population. Third, this was not a full-factorial design study. That is, not all of the possible interactions between dead zone, $\beta$ , and exponent were evaluated. Fourth, the subject’s hand size was not measured and hand size may have influenced the response to different parameters. For example, larger hands may have more difficulty with movement control with the small dead zone than smaller hands. Finally, this experiment only evaluated forward or backward movement along a straight road. If movement direction is determined by head position and forward velocity is determined by finger posture, then control of movement through 3-space would be more complex and the study findings may be different.
Although few studies have evaluated velocity-control using hand gestures, it is an important application given that other studies have found that velocity-controlled locomotion is often a preferable way to move in the VR environment. For example, Ware et al. used a device called the Bat for navigation, and found that velocity control was preferred over other methods of self-travel, such as “scene in hand” or “eyeball in hand”, because it made the experience more “movie” like[1,7]. They also found that non-linear mapping of position to velocity was preferred and allowed control over a wider range of velocities. A study by Card et al. had subjects use a joystick to compare a quadratic velocity slope to logarithmic ones[8]. Although the quadratic-like exponent of 1.7 was not significantly different from the other two conditions tested, it and the linear exponent were preferred over the more logarithmic counterpart. In another study, five different techniques for velocity-controlled travel were tested, including a gesture-controlled condition involving full arm motion[5]. Speed was a linear mapping of distance from hand to the head. The full arm motion caused fatigue, and therefore it was not preferred even though objective measures with gesture were better. In our study, the finger motions used for velocity control were not associated with fatigue. Other studies have recommended hand micro-gestures based on the ease of forming gestures and less fatigue compared to arm movements[9,10].
It would be useful for future studies to evaluate forward-backward velocity control combined with control of movement direction based on head posture. It would also be useful to directly compare finger or hand postures for control of locomotion in VE to other methods such as a hand held controller. As Ware and Osbourne[1] suggested, and was observed in this study, velocity control may be better over longer distances than short. It may be useful to evaluate different mapping functions at different distances; position control for high-precision short distances and velocity control for long distances.
In conclusion, this is the first study to evaluate how the design of small finger movements can control locomotion in a VE. Small finger movements are less fatiguing than large arm or whole body movements and it provides an alternative input compared to a hand-held controller. The size of the dead zone, where fingertip position was ignored and did not trigger locomotion had little influence on usability. Beyond the dead zone, where fingertip position was linearly mapped to velocity, a slope of the fingertip position to velocity of 12m/s per mm was associated with the best usability and productivity. A higher exponential shape (1.0 or 1.7) of the fingertip position to velocity curve provided better usability than a smaller exponent (0.5). These findings provide some guidelines in the design of finger gestures for locomotion in VEs.

Reference

1.

Ware C, Osborne S. Exploration and virtual camera control in virtual three dimensional environments. ACM SIGGRAPH computer graphics, 1990, 24(2): 175–183 DOI:10.1145/91394.91442

2.

Riecke B E, Bodenheimer B, McNamara T P, Williams B, Peng P, Feuereissen D. Do we need to walk for effective virtual reality navigation? physical rotations alone may suffice. In: International Conference on Spatial Cognition, Springer, Berlin, Heidelberg, 2010: 234–247 DOI:10.1007/978-3-642-14749-4_21

3.

Maggioni C. A novel gestural input device for virtual reality. In: Proceedings of IEEE Virtual Reality Annual International Symposium, IEEE, 1993, 118–124 DOI:10.1109/VRAIS.1993.380789

4.

Baudel T, Beaudouin-Lafon M. Charade: remote control of objects using free-hand gestures. Communications of the ACM, 1993, 36(7): 28–35 DOI:10.1145/159544.159562

5.

Jeong D H, Song C G, Chang R, Hodges L. User experimentation: an evaluation of velocity control techniques in immersive virtual environments. Virtual Reality, 2009, 13(1): 41–50 DOI:10.1007/s10055-008-0098-6

6.

Nai W, Rempel D, Liu Y, Barr A, Harris-Adamson C, Wang Y. Performance and user preference of various functions for mapping hand position to movement velocity in a virtual environment. In: Virtual, Augmented and Mixed Reality, Springer International Publishing, 2017, 141–152 DOI:10.1007/978-3-319-57987-0_12

7.

Ware C, Slipp L. Using velocity control to navigate 3d graphical environments: A comarison of three interfaces. In: Proceedings of the Human Factors Society Annual Meeting. Sage CA, Los Angeles, 1991, 35(5): 300–304 DOI:10.1177/154193129103500513

8.

Card S K, English W K, Burr B J. Evaluation of Mouse, Rate-controlled isometric joystick, step keys, and text keys for text selection on a CRT. Ergonomics, 1978, 21(8): 601–613 DOI:10.1080/00140137808931762

9.

Chan E, Seyed T, Stuerzlinger W, Yang X-D, Maurer F. User elicitation on single-hand microgestures. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose, California, ACM, 2016: 3403–341 DOI:10.1145/2858036.2858589

10.

Card S K, Moran T P, Newell A. The psychology of human-computer interaction. Hillsdale, New Jersey: Lawrence Erlbaum Associates, 1983