In February, 2020, the Ministry of Education and the Ministry of Science and Technology jointly released Opinions on Regulating the Use of Relevant Index of SCI Papers in Universities for Correct Evaluation Benchmarks. The “persistent concept” that the Scientific Citation Index (SCI) should be the supreme benchmark for evaluating a scholar’s academic competence must be weeded out to purify the academic atmosphere and optimize the academic environment. In the Reply to Proposal No. 2776 (Education No. 246) in the Third Session of the 13th National Committee of the Chinese People’s Political Consultative Conference (CPPCC), the Ministry of Education made clear the basic ways to further improve methods for evaluating scholars’ academic ranks and titles for the promotion process—by dint of their representative papers. The document calls for adoption of quantitative data-based and evidence-based expert evaluation methods which should not take SCI, ESI (Essential Science Indicators) or other related indices as the direct basis for judging academic competence. In evaluating a scholar’s academic titles for the promotion process, it is stipulated that representative papers should include a certain proportion of papers that are published in Chinese journals. It is encouraged for leading studies to be published in Chinese journals first.

The document has rapidly gained extensive attention and discussion from all social sectors. It is an urgent task to implement the methods suggested in this reply, and to make them a reality. To achieve this, the following approaches are advised.

The first approach is to bring new concepts to evaluation. Academic recognition can be taken as the first priority when formulating new quantitative evaluation indictors for measuring representative papers. It is now a pervasive phenomenon for papers to be evaluated according to the benchmarks of the journals where they are published, which simplifies the assessment of each paper’s quality. Regarding this, evaluation indicators design can consider the following factors: the degree of academic recognition received in earlier stages, including the quality of journals in which they are published, the papers’ appeal to leading scholars, if the papers were selected for publication in first-class journals sponsored by prominent academic associations, and the average degree of recognition for the papers; the degree to which the papers are recognized by academia in the subsequent stages, including comments from academia after papers are published, citations by authoritative scholars, and awards from associations. Only by winning recognition from leading scholars in Chinese academia, can papers truly be called “representative work.”

The second is to improve the peer review system. The peer review system began as early as the 17th century. With time-honored traditions, it is able to evaluate how innovative a paper is by examining the content, and it occupies a very important position in evaluating academic performance. However, a look at its practices both in China and internationally, shows that the system has some problems. A sound academic environment is therefore needed to resolve these. This requires a stronger sense of academic responsibility from scholars, and both incentives and rules are needed for motivating and regulating academics. Furthermore, a series of mechanisms such as expert selection mechanisms, review mechanisms, and dispute settling mechanisms are needed.

Currently, there are two methods for peer review which are noteworthy. One is informed peer review, which has been implemented in some European countries (such as Britain and Italy) for many years with good effects. It is based on quantitative information, for example, the degree of papers’ recognition in both the early and subsequent stages. This information is provided to experts, assisting them in more prudent decision-making. The other peer review method is transparent peer review, which opens communication between those being evaluated and evaluators to the public, thus academic reviews are conducted in an open and frank way. Different from open peer reviews, which asks evaluators to make their identities known to the public, transparent peer reviews allow evaluators to remain anonymous. These two can be adopted in a combined way in China. No matter what kind of peer review is carried out, it should be combined with quantitative evaluation. The last peer review method to discuss involves integrated data resources. The operating mechanisms for preprint databases should be optimized. A preprint is a full draft of a research paper that is shared publicly before it has been peer reviewed. Preprints may be lightly edited or screened, but they are typically not typeset or built into web pages. China has set up several preprint service systems, but the effect is not satisfactory as these systems have received limited attention. Many scholars do not even know of their existence. These preprint systems should thus be integrated for an efficient and appealing operating system. Scholars should be encouraged, and even required, to release their academic results in the preprint system with both Chinese and English versions of their abstracts attached. This will speed up the dissemination of knowledge both domestically and internationally. In addition, it is high time for a distinctive Chinese index and evaluation system to be set up for papers collected both at home and abroad. The most expedient way is to cooperate with international scientific data service providers and establish a paper index system which is China-based and globally-oriented.

Ma Ruimin is executive director and professor of the Scientific Evaluation and Research Center at Shanxi University.