Optimizing Deeper Transformers on Small Datasets

摘要

It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets. people usually use shallow and simple additional layers on top of pre-trained models during tine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train 48 layers of transformers, comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training. we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-updatc initialization scheme (DT-Fixup), inspired by the prior T-Fixup work (Huang et al.. 2020). Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.