Britain’s Royal Society published the world’s first academic journal Philosophical Transactions in 1665, and established publishing procedures which have been handed down till today: registration of contributions, peer-review (which was initially only conducted by Royal Society council members, until review procedures were improved in the 1930s), publication, and filing. Today, peer-review has become the most commonly applied evaluation system in academia. This process is extensively used in paper reviews and evaluation, fund allocation, talent selection, disciplinary evaluation and other activities. In 1978, the Derekde Solla Price Award winner, Francis Narin, for the first time pointed out the relationship between peer-review and quantitative indicators. Though the results of peer-review are not as satisfactory as quantitative indicators, which include numbers of papers published and numbers of times cited, peer-review reveals the true state of evaluated academic works. This idea is universally acknowledged by scientists, and many scientific practices prove its efficacy. For example, throughout the 100+ years since the Nobel Prize in Science was first awarded, the prize has held a high position within academia. Its selection process has entirely relied on peer-review rather than quantitative indicators.

However, peer-review in scientific evaluation is not yet perfect. Factors such as personal feelings and preferences, benefit-oriented evaluations, competition, an acquaintance society dominated by personal ties and relationships, all disturb the results of the peer-review process.

First, whether or not the study is scientific is judged by peers according to their personal experience and logic, which does not necessitate a test or verification by reviewers on their own. In poring over research papers submitted in the contribution process, the reviewers usually make judgments based solely on their own experiences and logic, and the experiment performed in the research paper does not need to be re-conducted. Due to this flaw in the review system, some experimental results which are unrepeatable smoothly pass through the peer-review process. When reviewing journal papers and conference papers that have already been published, reviewers are usually inclined to trust the previous peer-review which evaluated the papers before they were published. As a result, their judgments are made according to personal experience, the journals in which papers are published, and the reputation of conferences where the paper was presented. Again, this process does not require reviewers to repeat experiments conducted in the original research. This defect in the system creates opportunities for academic misconduct.

Second, while giving recognition to scientific discoveries in a professional way as the gatekeepers of science, editors and reviewers of academic journals lead the direction of science. However, according to research results published in Proceedings of the National Academy of Sciences of the United States of America (PNAS) by Kyle Siler and others from the University of Toronto, scientific gatekeepers are able to weed out inferior research through the peer-review process, but they are hardly able to discern which scientific discoveries are pioneering contributions, much less steer innovation. The peer-review process does not improve the core creativity of research papers.

Despite this, peer-review is still more reliable than quantitative indicators, and it should be continued as it offers more nuance than whether or not a paper is published in the Science Citation Index (SCI).

Above all other proposed reforms to the peer-review system, the selection of experts should adopt a system which prohibits the review by acquaintances, to avoid the influence of social ties and administrative forces on review results. In addition, the meta-review system should be improved, which means experts performance as peer-reviewers and the results themselves should be reviewed as a benchmark in selecting experts in follow-up stages.

In addition, it is necessary to supervise the peer-review process in an open way. Since openness is not a prerequisite for peer-review, and it may bring unwanted exposure to reviewers, peer-review is conducted anonymously. But to ensure that reviewers are not impacted by non-academic factors, public oversight of the review process is a feasible solution.

An open peer-review usually includes one of the three approaches: open access to reviewers’ identities, open access to peer-review reports, or open access to reviewers’ participation in the process. The National Natural Science Foundation of China has adopted open access to reviewers’ identities in evaluating scientific research programs and talent programs. The Public Library of Science (PLoS), BioMed Central press (BMC), Nature Communications, European Molecular Biology Organization (EMBO), and other prominent academic publishing groups, journals, and organizations, have adopted open peer-review reports as a part of their peer-review processes. F1000Prime, which is affiliated with Faculty of 1000 (F1000), has adopted open access to reviewers’ participation in the peer-review process, when the peer-review is conducted after papers are published. An open peer-review system could solve problems embedded in the traditional reviewing process, which is non-transparent, and to some extent, lacks incentives.

Li Jiang is a professor from the School of Information Management at Nanjing University.