Design of gesture algorithms for point and click using a 3D trajectory of fingertip

Abstract

As GUI was widely spread after 1980s, importance of point and click action and pointing device was increased. Studies for developing pointing devices were also increased. There are many representative pointing devices such as mouse, touchscreen, trackball, graphical tablet, etc. During the past some years, gesture interfaces received attention as alternative pointing devices to traditional pointing devices. Gesture-based interfaces eliminated cumbersome devices for user and can afford intuitive use. But still there were some problems should be solved. Gesture interfaces which based on user.s hand posture used pre-determined set of gestures. Users should memorize the gesture set to use these kind of interfaces. On the other hand, gesture interfaces which based on trajectory of user.s hands didn.t require users to learn pre-determined gestures. But this kind of interfaces weren.t designed for applications on Windows environment. They weren.t appropriate to click small targets like close button which normally located on the most upper-right edge of window. They were designed for simple structure of menu which consisted of big targets. And their gestures weren.t apt to operate double click or drag like mouse actions in Windows environment. In this study, new gesture algorithm was suggested as an alternative. The suggested gesture was deduced from normal people.s natural gesture to index or select target. So the developed gesture interface was operated with intuitive gesture. The pointing and the clicking actions were distinguished by movement on Z axis. If user moved his/her hand on X-Y plane, the interface operated pointing action. If user moved his/her hand on Z axis, the interface operated clicking action. Some additional algorithms were designed for more accurate clicking action. Cursor fixing function, which fixed the cursor on the screen when user clicked the target, was added to reduce wrong click. Pivot point function, which updated the reference point (pivot point) for cursor fixing function, was added to stabilize the clicking action. Forward angle function, which distinguished the pointing and clicking action when users moved their hands in the space between X-Y plane and Z axis, was added to provide the tolerances for point and click action. An experiment was conducted to figure out the performance of the interface and subjective satisfaction. On 12 locations of large screen (1720 mm × 1065 mm), targets were displayed. Participants pointed and clicked the targets one by one in random sequence. Target size had 3 levels (32, 53, 72 mm). The 12 locations were regularly divided as 4 × 3. Time between clicks and error rates were recorded for objective measures. Subjective ratings and ISO 9241-9 questionnaire were surveyed during and after experiment. The interface showed 3.1 for IP (index of performance) and more than normal (4 points) performance for ISO 9241-9 questionnaire. For the error rate, the interface has relatively high error rate for targets located on the side, especially the most right-below location. At the smallest target condition, the rate of wrong clicks was about 11%. At the largest target condition, the rate of wrong click was about 5%. As remedies, parameter resetting and click-plane updating were suggested.