On recognition of gestures arIsing in flight deck officer (FDO) training

Abstract

This thesis presents an on-line recognition machine RM for the continuous and isolated recognition of dynamic and static gestures that arise in Flight Deck Officer (FDO) training. This thesis considers 18 distinct and commonly used dynamic and static gestures of FDO. Tracker and computer vision based systems are used to acquire the gestures. The recognition machine is based on the generic pattern recognition framework. The gestures are represented as templates using summary statistics. The proposed recognition algorithm exploits temporal and spatial characteristics of the gestures via dynamic programming and Markovian process. The algorithm predicts the correspond-ing index of incremental input data in the templates in an on-line mode. Accumulated consistency in the sequence of prediction provides a similarity measurement (Score) between input data and the templates. Having estimated Score, some heuristics are employed to control the declaration in the final stages. The recognition machine addresses general gesture recognition issues: to recognize real time and dynamic gesture, no starting/end point and inter-intra personal tem-poral and spatial variance. The first two issues and temporal variance are addressed by the proposed algorithm. The spatial invariance is addressed by introducing inde-pendent units to construct gesture models. An important aspect of the algorithm is that it provides an intuitive mechanism for automatic detection of start/end frames of continuous gestures. The algorithm has the additional advantage of providing timely feedback for training purposes. In this thesis, we consider isolated and continuous gestures. The performance of RM is evaluated using six datasets - artificial (W_TTest), hand motion (Yang, Perrotta), Gesture Panel and FDO (tracker, vision). The Hidden Markov Model (HMM) and Dynamic Time Warping (DTW) are used to compare RM's results. Various data analyses techniques are deployed to reveal the complexity and inter similarity of the datasets before experiments are conducted. In the isolated recogni-tion experiments, the recognition machine obtains comparable results with HMM and outperforms DTW. In the continuous experiments, RM surpasses HMM in terms of sentence and word recognition. In addition to these experiments, a multilayer per-ceptron neural network (MLPNN) is introduced for the prediction process of RM to validate modularity of RM. The overall conclusion of the thesis is that, RM achieves comparable results which are in agreement with HMM and DTW. Furthermore, the recognition machine pro-vides more reliable and accurate recognition in the case of missing and noisy data. The recognition machine addresses some common limitations of these algorithms and general temporal pattern recognition in the context of FDO training. The recognition algorithm is thus suited for on-line recognition.