Designing Effective Sound Feedback for Continuous Gesture Interaction in Path-following Tasks

Abstract

Human-computer interaction (HCI) is becoming increasingly diversified and complicated. With developments in technology, such as motion recognition and interactive feedback, continuous interaction has increased, especially in augmented reality/virtual reality or large displays. In the continuous interaction in which user and system are in a continuous process of information exchange, sound can be usefully utilized as feedback, because sound can easily attract user attention, be easily recognized, represent dynamic information well, and does not require cumbersome or expensive equipment. Sound feedback with these advantages can enhance performance of continuous interaction, however, it has been relatively neglected in the field and only a few studies have been devoted to its development or evaluation. This study aims at designing effective sound feedback for continuous gesture interaction by investigating related factors, especially in path-following tasks. To derive various factors of sound feedback that affect the performance of continuous interaction, many previous studies were reviewed. The derived factors included existence of additional sound as multimodal feedback, sound dimension, interaction task type and level of difficulty, amount of information from sound feedback, sound continuity, and sound presentation method. The factors were constructed into a conceptual diagram, based on Wickens’ human information processing model, to express information process of continuous gesture interaction with sound feedback. They were hypothesized to be verified by experiments in the perspective of sound feedback effectiveness and effect of information from sound feedback in continuous interaction. An experimental prototype for continuous gesture interaction was developed by UNITY with Leap Motion. The interaction task was path-following as a representative task of continuous gesture interaction. Participants followed a given track with a right-index finger gesture while maintaining constant speed along the center line of the track. The influences of the factors were confirmed with four task performance criteria: deviation representing vertical distance from the center line, task completion time, subjective ratings, and workload by NASA-TLX. To verify sound feedback effectiveness in continuous gesture interaction, the first experiment was performed, with hypotheses regarding the effects of additional sound as multimodal feedback (H1), sound dimensions (H2), and interaction task type and level of difficulty (H3). The experiment was conducted on a head-up display with 16 participants. The experiment had four types of sound feedback: no feedback (S1), sound-triggered feedback over the boundary (S2), continuous sound feedback with change of frequency (S3), and continuous sound feedback with change of duration (S4). The experiment also had two types of tracks: curved track with circles and straight track with squares. The results showed that sound-visual multimodal feedback enhanced the accuracy of performance by attracting the participants’ attention and concentration. Continuous sound feedback by frequency change was preferred to continuous sound feedback by duration change, because deviation could be easily recognized in the task. Moreover, track types with different levels of difficulty influenced accuracy of performance because participants showed different strategies according to the track type. The results imply that sound-visual multimodal feedback can overcome the limitation of using only visual feedback in continuous interaction. Moreover, they imply the best design of sound feedback for enhancing performance in the perspectives of sound dimension and task difficulty. To verify the effect of information from sound feedback in continuous gesture interaction, a second experiment was carried out, with hypotheses regarding the effects of the amount of information from sound feedback (H4), sound continuity (H5), and sound presentation methods for information (H6). The experiment was conducted on a monitor display with 20 participants. The experiment had five types of sound feedback: no feedback (F1), sound triggered over the boundary (F2), continuous sound with two-steps (F3), continuous sound with a stepwise-changing pitch (F4), and continuous sound with a constantly-varying pitch (F5). From F1 to F5, the amount of information that presents alternatives in a track increases ordinally. The results showed that more information from sound feedback that induced arousal, attention, and concentration, improved the accuracy of performance, however, the workload also increased. The results also showed that case-triggered sound was suitable to a task that did not require accurate controls, and sound presentation methods were related to sound perception, especially in linearly-varying sound feedback. The results implied that the amount of information from sound feedback should be determined at the appropriate level, taking into account the tradeoff between accuracy and workload. Moreover, they implied the best design of sound feedback for enhancing performance from the user perception and cognition perspectives, according to sound continuity and presentation methods. This study investigated various factors influencing performance improvement of continuous gesture interaction with sound feedback in path-following tasks, and verified influences of the six factors by experiments. Some important issues that interface designers should consider were identified by reviewing the factors. The outcomes of this study are expected to be actively utilized in various areas where sound feedback can decisively influence user attention and performance in continuous interaction with high interactivity. Although this study was conducted in both head-up and monitor displays as representative devices of future technology, other products or systems with continuous interaction can be suggested. The continuous interactions with well-designed sound feedback, based on the insights of this study, can induce remarkable improvement in performance: especially in a surgery by gesture interaction, which requires precise control; a lane departure warning system of a vehicle, which must follow center line of road; or a walking assistant for the blind.