Assessing the Quality and Screening Ethical Issues in Peer-Review Comments

Abstract

Peer-review is a cornerstone of modern scholarly communication. It is a process that relies on the voluntary work of researchers to evaluate the manuscripts authored by their peers. However, poor quality reports, misconduct, fraud, bias and other ethical issues are a growing concern. This is witnessed by an increasing number of post-publication retractions of papers in recent years, largely fueled by the uncovering of large paper mills. Several computational approaches have been previously proposed to support editors and publishers in assessing the quality of peer-review reports. Further, several studies have shown that state-of-the-art natural language processing (NLP) methods can be used to extract certain features, including politeness or social biases, from texts. However, the literature is often focused solely on the technical aspects of the proposed approaches and lacks the inclusion of business requirements and the human operators, i.e., the editors and publishers. The study thus aimed to address this gap by establishing the requirements and design principles for an effective information system that supports editors and publishers in screening and assessing peer-review reports on a large scale.