Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information

摘要

The boom of big data in education has provided an unrivalled opportunity for educators to evaluate the learners' cognitive state. However, most existing cognitive state analysis methods focus on attention, ignoring the roles of emotion in human learning. Therefore, this study proposes an emotion-sensitive learning cognitive state analysis framework, which automatically estimates the learners' attention based on head pose and emotion based on facial expression in a non-invasive way. The proposed framework includes two modules. In the first module, a multi-task learning implementation with a cascaded convolutional neural network (CNN) is presented for face detection, landmark location, and head pose estimation simultaneously. The located landmarks are used to align the faces as the preprocessing step for the facial expression analysis. The estimated head pose and landmarks are used to recognize the visual focus of attention of the learner. In the second module, an expression intensity ranking CNN is proposed to recognize the facial expression and evaluate its intensity using ordinal information of the sequences. Then, the learners' emotions are estimated based on the facial expression. Experimental results show that this method can estimate a learner's attention and emotion with correctness rates of 79.5% and 88.6%, respectively. The results obtained suggest that the method has strong potential as an alternative method for analyzing emotion-sensitive learning cognitive state.