In American Sign Language (ASL), the manual and the non-manual components play crucial semantical and grammatical roles. The design of systems that can analyze and recognize ASL sentences requires the recovery of both these manual and non-manual components. Manual signs in ASL are constructed using three building blocks---handshape, motion, and place of articulation. Only when these three are successfully estimated, can a sign be uniquely identified. The first part of my research is to define algorithms to recognize manual signs based on the recovery of these three components from a single video sequence of two-dimensional images of a sign. The 3D handshape is obtained with a structure-from-motion algorithm based on the linear fitting of matrices with missing data. To recover the 3D motion of the hand, a robust algorithm is defined which selects the most stable solution from the pool of all the solutions given by the three point resection problem. Faces of the signers in the video sequence are detected, with which the coordinate system with respect to the signer is defined and hence we recover the place of articulation of the sign. Based on the recognition results of the three recovered components, the manual signs are recognized using a tree-like structure. For the non-manual component of ASL, we need to provide an accurate and detailed description of external and internal facial features. The second part of this research focuses on the precise detailed detection of faces and facial features. Learning to discriminate the features from their context permits a precise detection of facial components, which is the key point of the feature detection algorithm. And because the shape and texture of facial features vary widely under changing expression, pose and illumination, the detection of a feature versus the context is challenging. This problem is addressed with the use of subclass division, which is employed to automatically divide the training samples of each facial feature into a set of subclasses, each representing a distinct construction of the same facial component. This approach is combined with edge and color segmentation to provide an accurate and detailed detection of the shapes of the major facial features. This proposed detection algorithm is used to obtain precise descriptions of the facial features in video sequences of ASL sentences, where the variability in expressions can be extreme. With the proposed algorithms, the modelling and recognition of ASL manual signs using the three manual components are achieved, and the non-manuals of ASL are detailedly and precisely modelled, which provides data for the analysis and recognition of the non-manuals in ASL. The recognition of both the manual and the non-manual components enables human-computer-interface systems to understand ASL.
展开▼
