Meet in the Middle: A New Pre-Training Paradigm for Language Models to Enhance Text Infilling
This research introduces a novel pre-training paradigm for language models (LMs), termed “Meet in the Middle” (MIM), which optimizes data utilization by integrating both prefix and suffix contexts while preserving autoregressive properties. MIM employs a dual approach, training forward and backward LMs concurrently on a shared corpus, with an agreement regularizer to ensure consistency in token probability distributions. This method enhances data efficiency and model agreement, allowing for improved performance in text infilling tasks. Evaluation across various domains confirms MIM’s superiority over traditional models, showcasing its potential to redefine LM pre-training and application.