A Novel Joint Optimization Method for Adaptive Hand-held Video Stabilization Based on Spatial-temporal Consistency

摘要

Video stabilization for hand-held camera is vital in many high-level video enhancement applications. Although quite a few proposed approaches have achieved remarkable success in recent years, many technical challenges still prevail for video (from hand-held camera) containing wide-range scenes and highly-variable objects. In particular, we are lacking effective and versatile strategies to adaptively handle saliency preservation, parallax diminution, self-adaptive smoothing, cropping area decrement and video completeness in a consistent spatial-temporal fashion. To ameliorate, this paper develops a novel, joint optimization method to successively respect spatial structure consistency and temporal feature constraints. Our aim is to devise a new adaptive video stabilization technique by resorting to new modeling strategies. This paper's key originality is hinged upon joint utility of both self-adaptive intrinsic mode functions (IMFs) based on empirical mode decomposition (EMD) in temporal domain for video signal and mesh-structure constraint enforcement in the spatial domain. As a result, our new approach can optimize camera trajectory of wobbly video, and synchronously fine-tune the camera path based on key features to make the shaking video much closer to the original trend. To validate our joint optimization approach for adaptive video stabilization, we conduct comprehensive experiments on public benchmarks, and make extensive and quantitative evaluations with available state-of-the-art methods as well as popular commercial software. All of our experiments demonstrate the advantages of the joint optimization method in terms of versatility, accuracy, and efficiency.