Lifelong CycleGAN for continual multi-task image restoration

摘要

Recent years have witnessed the great success of deep learning in the applications of image restoration. However, there are still some challenging problems. First, most deep learning methods rely heavily on paired training images which are difficult to capture in the real world. Second, most existing methods are generally designed for a specific task of image restoration and they suffer from catastrophic forgetting when learn multiple tasks continually. Third, for most multi-task image restoration methods, they learn an individual network for each task and require to know the type of distortion for both training and test, which costs a lot of computational time and memory. To address the above issues, we propose a new lifelong learning framework based on CycleGAN for continual multi-task image restoration, called Lifelong CycleGAN (LCGAN), which can enhance low-light images, deblur and denoise simultaneously. The model utilizes knowledge distillation and memory replay to transfer knowledge and replay information that learned previously to alleviate forgetting. In addition, to regularize the unpaired training, we introduce the local discriminator and feature consistency constraint to preserve the color, edge and texture of input images. The proposed method can continually learn the three tasks using one network model and doesn't need to prejudge the type of distortion, which has low time and memory requirements. Experimental results demonstrate that LCGAN can achieve better visual and numerical results across the three tasks. (c) 2021 Elsevier B.V. All rights reserved.