Virtual piano with real-time interaction using automatic marker detection

摘要

This paper aims to apply the object detection used in Augmented Reality (AR) technology to a real-time application for musical instrument, a virtual piano. The proposed application does provide the sounds of musical notes and also display the corresponding notations of the playing notes. Both features allow people with hearing disability or muscular weakness or even ones who cannot exert their pressure on the general keyboard instruments to play music, due to their physical disabilities. The experimental results demonstrated that the error of the playback sounds is only 0.5 percent which is only 2 times measured from tapping 50 markers for 400 times in real time. However, all errors are because of the wrong positions of the player's hand as some of his fingers accidentally covered over the unexpected markers during playing the virtual piano.