Integrating Machine Learning Based Operators in Visual Effects Toolsets

摘要

Post-production workflows rely heavily on image processing and computer vision algorithms for the implementation of their visual effect (VFX) tools. In these fields, machine learning has been shown to be disruptive. By integrating the domain statistics from a given training dataset, machine learning based operators may be more efficient at solving existing problems and may enable new problems to be solved, expanding the VFX toolset. In this paper we are sharing our experience on developing and integrating several machine-learning based operators into software for the Post-production industry. We will present a sky segmentation operator, a depth map estimation operator and an operator to compute face geometric maps (UVs, depth and normals) from sequences of images. More specifically, these operators consist in trained deep convolutional neural networks (DCNN) taking as input an RGB color image and outputting the associated maps, i.e. a matte, depth, normals, and/or UVs maps. Such maps permit to apply many different effects to the input image during color grading, including beautification or even relighting of faces. These works will serve as a case study to review and discuss the multiple challenges posed by the implementation, integration and deployment of machine learning based operators into a VFX toolsets.