When scientists prefer the provisions of ChatGPT

Could artificial intelligence one day review scientific articles, or even do what we call peer review? For now, the first scholars examined on this topic favor ChatGPT’s judgments over those of their colleagues.

Let us remember that this is an essential part of the process by which scientific knowledge is constructed: whether it is a supposed discovery, hypothesis, confirmation, or All of this must be supported by published research. Ideally, this research should be reviewed by other experts in the field before publication. This is referred to as “peer review.”

However, this has its limitations: you have to find experts who are able to understand what the research is talking about, and these experts have to have time. Traditionally, months pass between the time a researcher submits an article to a scientific journal and the time of publication.

Can ChatGPT replace human reviewers? This is what researchers led by James Zhou, a machine learning expert at Stanford University in California, wanted to test. They asked ChatGPT-4 to provide “constructive criticism” for more than 3,000 studies published in 2022-2023 by one of the Nature group journals (and, therefore, peer-reviewed), and for 1,700 articles from an international conference on machine learning (International Conference on Learning Representations). They compared robot reviews with human reviews. As a second step, they asked ChatGPT the same for a few hundred research papers that had not been reviewed by anyone, and asked about 300 of their authors (all in the field of AI or IT) to rate the bot’s criticisms of them.

In summary, Which was previously published On October 3 on the ArXiv server (which, ironically, means it has not been peer-reviewed), they wrote first for more than half of the published texts and for more than three-quarters (77%) of the published texts. In the conference transcripts, the bot pointed out things that at least one of the reviewers had also pointed out. But the strongest result was for unpublished, unedited manuscripts: 82% of authors said they found criticism of ChatGPT more useful than criticism they had received in the past about other papers.

But the work faces major drawbacksIn the second part of the research, the researchers’ evaluation of the criticisms directed at them by the robot is a purely subjective evaluation and does not allow it to be compared with the criticisms that a human might direct for the same work. The first part of the research provides few details about what this information is that is not specified by ChatGPT, but is reported by humans. That remains to be seen Whether scientific journals are satisfied with these AI reviews Without having a guarantee that the robot has not “forgot” something important – but this question also exists in real peer review, which is far from infallible.

Subscribe to our sprawling newsletter