3D Human Gesture Tracking and Recognition by MEMS Inertial Sensor and Vision Sensor Fusion.

摘要

The aim of this dissertation is to describe a 3D human gesture tracking and recognition system that has been developed by fusing MEMS inertial sensor and CMOS image sensor in real-time to improve tracking and recognition accuracy over existing systems. This paper presents methodologies for calibrating the sensors and relative rotations between different coordinate frames. Direct Linear Transforms (DLT), Pose from Orthography and Scaling with Iterations (POSIT), and Perspective-n-Point problem (EPnP) have been evaluated for pose estimation. Extended Kalman filter (EKF) and unscented Kalman filter (UKF) have been applied to combine the complementary performances of these two sensors. Several EKF based algorithms have been developed and their performances for real-time gesture tracking discussed. An error model which has been proved to be effective in reducing the "drift" problem of inertial sensors when visual information is lost, has been built based on the stochastic errors of inertial sensor identified by Allan variance. In addition, since lower values of accelerations are associated generally with higher values of measurement noise, an adaptive measurement noise update model has been developed to reduce the effect of measurement noise on fusion results. For trajectory-based handwritten numeral recognition, I focus on the recognition of ten Arabic numerals. Discrete Fourier transforms (DFT) and direction Cosine transform (DCT) are applied for dimension reduction, while Principle Component Analysis (PCA) and a modified PCA are applied for feature extraction. Next, dynamic time warping (DTW) is applied for the trajectory-based handwritten numeral recognition. Also, the technique of Support Vector Machine (SVM) with some additional information except trajectories, such as velocities and accelerations has also been applied for numeral recognition.