Multi-modal detection engine of sentiment and demographic characteristics for social media videos

摘要

A system and method for determining a sentiment, a gender and an age group of a subject in a video while the video is being played back. The video is separated into visual data and audio data, the video data is passed to a video processing pipeline and the audio data is passed to both an acoustic processing pipeline and a textual processing pipeline. The system and method performs, in parallel, a video feature extraction process in the video processing pipeline, an acoustic feature extraction process in the acoustic processing pipeline, and a textual feature extraction process in the textual processing pipeline. The system and method combines a resulting visual feature vector, acoustic feature vector, and a textual feature vector into a single feature vector, and determines the sentiment, the gender and the age group of the subject by applying the single feature vector to a machine learning model.