While the concept of peer review is widely considered the most appropriate system for regulating scientific publications, it is not without its critics. Some feel that the system's reliance on impartiality and the lack of remuneration for referees mean that in practice the process is not as open as it should be. This may be particularly apparent when referees are asked to review more controversial ideas that could damage their own standing within the community if they give their approval.
Questioning referee competence
Stefan Thurner and Rudolf Hanel at the Medical University of Vienna set out to make an assessment of how the peer-review system might respond to incompetent refereeing. [snip]
The researchers created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the "correct" who accept the good papers and reject the bad. There are the "altruists" and the "misanthropists", who accept or reject all papers respectively. Then there are the "rational", who reject papers that might draw attention away from their own work. And finally, there are the "random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.
[snip]
Within this model community, the quality of scientists is assumed to follow a Gaussian distribution where each scientist produces one new paper every two time-units, the quality reflecting an author's ability. At every step in the model, each new paper is passed to two referees chosen at random from the community, with self-review excluded, with a reviewer being allowed to either accept or reject the paper. The paper is published if both reviewers approve the paper, and rejected if they both do not like it. If the reviewers are divided, the paper gets accepted with a probability of 0.5.
Big impact on quality
After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. When just 10% of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.
"Our message is clear: if it can not be guaranteed that the fraction of rational and random referees is confined to a very small number, the peer-review system will not perform much better than by accepting papers by throwing (an unbiased!) coin," explain the researchers.
[snip]
Don't forget the editors
But Tim Smith, senior publisher for New Journal of Physics at IOP Publishing, which also publishes physics world.com, feels that the study overlooks the role of journal editors. "[snip]. In relation to this study however, one shouldn't ignore the role played by journal editors and Boards in accounting for potential conflicts of interest, and preserving the integrity of the referee selection and decision-making processes," he says.
Michèle Lamont a sociologist at Harvard University who analyses peer review in her 2009 book, How Professors Think: Inside the Curious World of Academic Judgment, feels that we expect too much from peer review. [snip]
When asked by physicsworld.com to offer an alternative to the current peer-review system, Thurner argues that science would benefit from the creation of a "market for scientific work". He envisages a situation where journal editors and their "scouts" search preprint servers for the most innovative papers before approaching authors with an offer of publication. [snip]
[snip]
Related
"Peer Review And Journal Models" / Paolo Dall'Aglio
[http://arxiv.org/abs/physics/0608307]
About the author
James Dacey is a reporter for physicsworld.com
[http://physicsworld.com/cws/article/news/43691]
!!! Thanks To Antonella De Robbio For The HeadsUp !!!
