A comprehensive perspective of contrastive self-supervised learning

摘要

Self-supervised learning (SSL), as a new unsupervised representation learning paradigm in machine learning, recently has received extensive attention, which is also regarded as the future of machine learning by the Turing Award winner, LeCunn [1]. SSL learns representation from unlabeled data using "pretext" tasks that provide free supervision, with the aim of performing well on semantic relation agnostic downstream (supervised) tasks. It usually divides two stages: first learning as general/invariant representation/feature as possible with the auto-annotation pretext tasks (the core), then transferring the learned knowledge to downstream tasks (the ultimate goal) [2]. As the core of SSL, a series of pretexts have been developed. Among them, contrastive SSL (cSSL) has become a mainstream, dominant methodology, since it naturally consists with the cognitive development in children [3], which forms the concept class by contrasts. cSSL typically learns the representation by contrasting latent representations of different cheap transformation augmentations or cluster assignments of images, which even surpasses the supervised counterparts in certain settings. Such excellent performance has attracted the attention of numerous researchers, and next we briefly review its recent advances [4].