Anonymizing peer review makes the process more just

When manuscript authors’ identities and affiliations are blocked from peer reviewers, unconscious bias is less likely to influence peer review than when that information is available, a study finds.

The research¹ finds that ecology manuscripts by authors from lower-income nations, or ones with lower English proficiency, fare worse during the review process than do studies by authors in the same field from higher-income, English-speaking nations. Anonymizing authors eliminates much of this bias, the analysis finds.

Previous work has shown that manuscripts by authors from lower-income countries are treated differently. Bedoor AlShebli, a computational social scientist at New York University Abu Dhabi in the United Arab Emirates who examines racial inequalities in scientific publishing found that papers by authors from Asia, Africa and South America spend more time in the peer-review process².

Racial inequalities in journals highlighted in giant study

Charles Fox, an entomologist at the University of Kentucky in Lexington who led the latest study, says that he wanted to examine whether unconscious bias influences how reviewers evaluate research, and whether double-blind peer review — in which neither authors nor reviewers know each others’ identities — solves the problem.

In 2019, Fox, who was at the time executive editor of Functional Ecology, together with the journal’s editorial team, began a randomized trial to evaluate the costs and impacts of shifting the journal to double-blind peer review.

Hidden identities

In the study, authors who submitted a manuscript to the journal during a three-year period from 2019 to 2022 were required to submit the title page — containing the authors’ identities and any other identifying information, including contact details, institutional affiliations and acknowledgements — separately from the rest of the document. Authors were also advised not to identify themselves in the text in an obvious way. A manuscript-tracking system randomly assigned the papers to be reviewed either double blind or single blind (in which case the authors’ identities are known to the reviewers). A total of 3,689 papers were included in the study.

Stop the peer-review treadmill. I want to get off

During the study, 40.2% of the submitted papers progressed to peer review. For 34.3% of these papers, the authors were invited to make revisions; 16.7% of the papers reviewed were declined but the authors were allowed to resubmit; and 48.9% were rejected without the option of resubmission.

The study found that reviewers gave double-blind peer-reviewed papers lower scores than they gave to papers whose author identities they knew. Papers with identified authors were 24.2% more likely than those reviewed double blind to prompt invitations to submit a revision. They were also 15.2% more likely to have an overall positive outcome — with the authors being invited either to submit revisions or to resubmit the whole paper.

Loss and gain

Fox and his colleagues also found that authors from high-income and English-speaking countries benefit when reviewers know their identities, and that they lose those benefits when their identities are hidden. Papers by identified authors from the world’s richest nations, such as the United States, were 68% more likely to be sent for peer review than were papers by identified authors from less-affluent countries. Authors from high-income countries also received higher review scores and were more likely to be asked to resubmit manuscripts than were authors from lower-income nations — but only when reviewers knew their identities.

Nature is trialling transparent peer review — the early results are encouraging

For example, authors from wealthy nations were 28% more likely than those from less-wealthy ones to be asked to revise or resubmit when their identities were known, but only 4% more likely when their identities were hidden. By contrast, authors from low-income or non-English speaking countries, as a group, experienced similar peer-review outcomes whether their identities were hidden or not.

“Double-blind review certainly seems to be a promising way to mitigate biases,” says AlShebli, who was not involved in the current study. Academic publishers can also help to stop favouritism by enforcing policies that block editors from handling manuscripts submitted by their recent collaborators, she adds.

Certain biases did not seem to be in play in the study, however. Notably, the researchers found that double-blind peer review did not affect the outcome of papers by female authors. Such manuscripts were just as likely to progress to peer review — and even received slightly higher reviewer scores — than were papers by male authors, whether identities were blocked or not.

Prestige bias

Fox says the study shows that reviewers are not prejudiced against researchers from low-income or non-English-speaking countries. Rather, authors from rich nations get a boost when reviewers know their identity. He suggests that researchers from wealthy nations benefit from prestige bias — whereby reviewers expect work from researchers from certain institutions or countries to be of high quality and give deference to them.

In a statement published on 4 April, the British Ecological Society, which publishes seven journals, including Functional Ecology, said in response to Fox and his colleagues’ findings that it would start requiring all its journals to use double-blind peer review.

However, Fox warns against allowing authors to voluntarily opt for anonymous review — which the journal Nature Ecology and Evolution, for example, does. That will not resolve the bias, he says. Authors from lower-income nations will benefit only if authors from richer nations remain anonymous, — and for that to happen, it’s likely that double-blind peer review will need to be mandatory.

“When people are anonymized, the papers are actually being reviewed more honestly,” says Fox. “It would be good for science if all were treated the same.”