The National Institutes of Health uses small groups of scientists to judge the quality of the grant proposals that they receive, and these quality judgments form the basis of its funding decisions. In order for this system to fund the best science, the subject experts must, at a minimum, agree as to what counts as a “quality”proposal. We investigated the degree of agreement by leveraging data from a recent experiment with 412 scientists.
The Peer Review Game: an Agent-based Model of Scientists Facing Resource Constraints and Institutional Pressures
This paper looks at peer review as a cooperation dilemma through a game-theory framework. We built an agent-based model to estimate how much the quality of peer review is influenced by different resource allocation strategies followed by scientists dealing with multiple tasks, i.e., publishing and reviewing.
OAI11 - CERN-UNIGE Workshop on Innovations in Scholarly Communication (19-21 June 2019)
The CERN-UNIGE Workshop on Innovations in Scholarly Communication will be held at University of Geneva in June 19th-21st 2019. The main theme of this edition is: Open Science - its impact and potential as a driver for radical change.
Assessing Peer Review by Gauging the Fate of Rejected Manuscripts: the Case of the Journal of Artificial Societies and Social Simulation
This paper investigates the fate of manuscripts that were rejected from JASSS- The Journal of Artificial Societies and Social Simulation, the flagship journal of social simulation. We tracked 456 manuscripts that were rejected from 1997 to 2011 and traced their subsequent publication as journal articles, conference papers or working papers.
Big Qual - Why We Should Be Thinking Big About Qualitative Data for Research, Teaching and Policy
When social scientists think about big data, they often think in terms of quantitative number crunching. However, the growing availability of ‘big’ qualitative datasets presents new opportunities for qualitative research.
This paper investigates the impact of referee behaviour on the quality and efficiency of peer review. We focused on the importance of reciprocity motives in ensuring cooperation between all involved parties. We modelled peer review as a process based on knowledge asymmetries and subject to evaluation bias. We built various simulation scenarios in which we tested different interaction conditions and author and referee behaviour. We found that reciprocity cannot always have per se a positive effect on the quality of peer review, as it may tend to increase evaluation bias. It can have a positive effect only when reciprocity motives are inspired by disinterested standards of fairness.
Making Research Open and Reproducible: An Early Career Researcher's Perspective
As an early career researcher (ECR), making the transition from the “traditional” way of doing science into methods that are more open, reproducible, and replicable can be a daunting prospect. We know something needs to change about our workflow, but where do we start?
Does Incentive Provision Increase the Quality of Peer Review? An Experimental Study
Although peer review is crucial for innovation and experimental discoveries in science, it is poorly understood in scientific terms. Discovering its true dynamics and exploring adjustments which improve the commitment of everyone involved could benefit scientific development for all disciplines and consequently increase innovation in the economy and the society.
Saint Matthew Strikes Again: An Agent-based Model of Peer Review and the Scientific Community Structure
This paper investigates the impact of referee reliability on the quality and efficiency of peer review. We modeled peer review as a process based on knowledge asymmetries and subject to evaluation bias.
Science is never the work of one person; it is the collaborative effort of students, technicians, professors, librarians and the support networks around them. This week, millions of girls and women around the world who have been told science is not for them found a new role model in Bouman - a new data point that told them yes you can.
Fourteen universities from five European countries started a collaboration to set up University Journals as an alternative to the current journal system that requires authors to transfer their copyright or charges article processing charges.
Although there is growing concern about the urgent need for a better life-work balance when doing science, there are not many examples about how this could be achieved in practice. In this article, 10 simple rules are introduced to make the working environment of research labs more nurturing, collaborative, and people-centered.
Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-Making Mechanisms for Allocation of Research Grants
By forming a pool of funding applicants who have minimal qualification levels and then selecting randomly within that pool, funding agencies could avoid biases, disagreement and other limitations of peer review.
Attitudes of Referees in a Multidisciplinary Journal: An Empirical Analysis
Paper finds that the disciplinary background and the academic status of the referee have an influence on their reviewing tasks. Articles that had been recommended by a multidisciplinary set of referees were found to receive subsequently more citations than those that had been reviewed by referees from the same discipline.