PrintMay 3, 2009
Peer Review Needs Improvement
Scientific issues often deal with items that are too complex and technical for many people to grasp. When these issues touch on public policy, the problem of educating the public is compounded because partisan special-interest groups often mischaracterize or disregard the best available evidence in order to further their agendas. Obviously, this can often result in the enactment of ineffective or counterproductive laws or government programs and also unnecessarily excite the public.
A National Academy of Engineering report says, "As a society, we are not even fully aware of or conversant with the technologies we use every day. In short, we are not ‘technologically literate.'" (1)
Peer review is supposed to help in this area. What is it?
"Peer review is the process by which research and scholarship are evaluated by other experts in one's field. The depth and breadth of formal peer review varies by field; for example, the number of reviewers, as well as whether or not they and article authors will remain anonymous to one another, differs across science, social science, and humanities disciplines. Informal peer review also takes place after research results are published in an article; others in the field weigh in with observations and experiences that question, critique, or support the authors' assertions," reports Leslie Madsen. (2)
Yet, as John Moore writes in Nature,
"It's been peer reviewed so it must be right, right? Wrong! Not everything in the peer-reviewed literature is correct. Indeed, some of it is downright bad science. Professional scientists usually know how to rate papers within their own fields of expertise (all too often very narrow ones nowadays). We realize that some journals are more stringent than others in what they will accept, and that peer review standards can unfortunately be too flexible. A lust for profits has arguably led to the appearance of too many journals, and so it can be all too easy to find somewhere that will publish poor-quality work." (3)
Peer reviewers merely give advice to the editor as to whether a paper should be published. There is no warranty that the results are correct or that they can be reproduced. The reviewers whom the editor picks say yes, we would like to see this in the journal; that is the start and finish of it. (4)
Yet, the term ‘peer review' is often equated with ‘gold standard'. Hence, the politically motivated, lazy or unscrupulous can use the peer-reviewed literature selectively to make arguments that are seriously flawed, or even damaging to public policy. Chris Mooney, in The Republican War on Science (Basic Books, 2005) provides several examples of how this operates in the political world. (3)
Ragner Levi observes,
"As every experienced medical reporter knows, there is no guarantee that published studies are reliable-far from it. Though peer reviewed articles are generally less biased than non-reviewed articles, even highly unreliable results can be published in peer reviewed scientific journals. In a randomized controlled trial looking at peer review, a paper with eight significant errors was sent to 420 reviewers in JAMA's database. None spotted more than five errors, and most not more than two." (5)
Here's another example. Another report in JAMA found that one-third of studies published in three reputable peer reviewed journals didn't hold up. John Ioannidis looked at 45 studies published between 1990 and 2003 and found that subsequent research contradicted the results of seven of those studies, and another seven were found to have weaker results than originally published. In other words, 32% did not withstand the test of time.
This translates into a lot of medical misinformation! Ioannidis reviewed high-impact journals including The New England Journal of Medicine, The Journal of the American Medical Association (JAMA) and Lancet, along with a number of others. Each article had been cited at least 1,000 times, all within a span of 13 years. In a number of cases, the explanation for the discrepancies was in precisely what you'd suspect, sample size. The smaller the group, the shorter the study, the more likely it was that subsequent, deeper investigation contradicted or altered the original thesis. Where was the peer review? (6)
Garrett Lisi notes,
"This old system persists because academic career development often depends on which journals scientists can get their papers into, and it comes at a high cost-in money, time and stress. I think a better peer review system could evolve from reviewers with good reputations picking the papers they find interesting out of an open pool, and commenting on them. (7)
There is hope for improvement. In his book The Great Betrayal, Horace Freeland Judson agrees there is a distressing downward spiral in the peer review system, but holds out hope that ‘open review,' which prevent reviewers from hiding behind anonymity, and open publication on the Internet rather than in peer reviewed journals, may solve some of the problems. (8)
Notes
- Technically Speaking: Why All Americans Need to Know More About Technology, G. Pearson and A. T. Young, Editors, (Washington, DC, National Academy Press, 2002), 1
- Leslie Madsen Brooks, www.blogher.com/peer-review-science-it-broken, March 14, 2009
- John Moore, "Perspective: Does peer review mean the same to the public as it does to scientists?," Nature, (2006), doi: 10.1038/nature05009
- Stephen McIntyre and Ross McKitrick, "The Hockey Stick Debate: Lessons in Disclosure and Due Diligence," (Washington, DC, George Marshall Institute, May 11, 2005)
- Ragner Levi, Medical Journalism, (Ames, Iowa, Iowa State University, 2001), 64
- John P. A. Ioannidis, "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research," JAMA, 294 (2), 218, July 13, 2005
- Greg Boustead, "Garrett Lisi's Exceptional Approach to Everything," seedmagazine.com, November 17, 2008
- Horace Freeland Judson, The Great Betrayal: Fraud in Science, (New York, Harcourt, Inc., 2004)
