Neural Network 3D Body Pose Tracking and Prediction for Motion-to-Photon Latency Compensation in Distributed Virtual Reality

摘要

Distributed Virtual Reality (DVR) systems enable geographically dispersed users to interact in a shared virtual environment. The realism of the interaction is crucial to increase the feeling of co-presence. Latency, produced either by hard- or software components of DVR applications, impedes reaching high realism levels of the DVR experience. For example, the time delay between the user's motion and the corresponding display rendering of the DVR system might lead to adverse effects such as a reduced sense of presence or motion sickness. One way of minimizing the latency is to predict user's motion and thus compensate for the inherent latency in the system. In order to address this problem, we propose a neural network 3D pose tracking and prediction system with latency guarantees for end-to-end avatar reconstruction. We evaluate and compare our system against multiple traditional methods and provide a thorough analysis on real-world human motion data.