Recent Developments for Multimodal Interaction by Visual Agent with Spoken Language

摘要

This paper presents our recent research on multi-modal interaction where four modes; speech recognition, speech synthesis, computer vision, and a visual agent generated by computer graphics, are integrated. The research includes studies on multimodal interaction gained by collecting data on speech and visual images expressed during human and machine interaction, statistical analysis of the facial images, and the development of a prototype of multimodal interaction system. Through this research, we have examined the practical application of the multimodal interaction system using such modes.