Open Up: a Survey on Open and Non-anonymized Peer Reviewing
Our aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields. We ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions. Our main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/ , and the figures and follow-up work on http://tiny.cc/OpenReviews . While the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.
Which would you trust more, a research article posted as a preprint, or one that has been published after peer review? The reality is that all science communicated via either mechanism should be read with a discerning and critical eye.
This Article examines the effect of abolishing peer review on the changed incentive structure and the likely effects on the behaviour of individual scientists, and concludes that, abolishing peer review has overall slightly positive results.
Webinar: Scholarly Communication & COVID-19: Closing the Loop for Effective Peer Review
We are pleased to announce the next OASPA webinar which will explore recent steps to increase efficiency and speed in the publication of COVID-19 research (Wednesday 24th June 2020, 4.00 pm Central European Time).
The Limitations to Our Understanding of Peer Review
Peer review is embedded in the core of our knowledge generation systems. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, this paper assesses where the major gaps in the theoretical and empirical understanding of peer review lie.
The study replicates the NIPS experiment of 2014, showing that the ratings of peer review are not robust, and that altering reviewers leads to a dramatic impact on the ranking of the papers. This paper also shows that innovative works are not highly ranked in the existing peer review process, and in consequence are often rejected.
The public call for rapid sharing of research data relevant to the COVID-19 outbreak is driving an unprecedented surge in (unrefereed) preprints. To help pinpoint the most important research, Nature launched Outbreak Science Rapid PREreview, an open-source platform for rapid review of preprints related to emerging outbreaks.
Many of the words used by scientists when reviewing manuscripts, job candidates and grant applications - words such as incremental, novelty, mechanism, descriptive and impact - have lost their meaning.