Boosting language models for real-word error detection

Abstract

With the introduction of transformer-based language models, research in error detection in text documents has significantly advanced. However, some significant research challenges remain. In the present paper, we aim to address the specific challenge of detecting real-word errors, i.e., words that are syntactically correct but semantically incorrect given the sentence context. In particular, we research three categories of frequent real-word errors in German, viz. verb conjugation errors, case errors, and capitalization errors. To address the Real-word errors refer to words in texts that exist in the underlying dictionary but are incorrect in the context of the sentence. One open issue in detecting real-word errors is that there is limited data available for training the models, especially for languages other than the dominant language in research (namely English). To counteract this limitation, we propose to incorporate synthetic data in the training process for three categories of real-word errors frequently encountered in German text, viz. conjugation errors in verbs, wrong case selection, and capitalization errors. The first contribution of this paper is that we generate high-quality synthetic data from a real-world text data set provided by a Swiss proofreading agency that can be used for model training. In addition to the introduction of a novel and large-scale synthetic data set, the second major contribution of this paper is that we propose to incorporate ensemble learning methods for language models. Actually, a few approaches have been proposed that combine ensemble learning methods with language models. One such strategy, known as boosted prompting, is inspired by classical boosting algorithms. This method iteratively augments the prompt set with new prompts that better generalize regions of the target problem space where the previous prompts underperform (Pitis et al., 2023). Another approach is to train multiple models and combine them for the final output. In (Li et al., 2019), CNN-based and transformer-based models were combined to tackle the challenge of grammatical error correction. In the present paper, we propose to employ boosting techniques to enhance the training process of language models and ultimately improve the accuracy of language models for detecting real-word errors. To the best of our knowledge, this is the first time that boosting is used in combination with language models in this specific way and for this particular task.