Peer Review - Politics of Science?

Categories

Legislation

Science

According to a recent article on BushGreenwatch.org, the White House is looking for ways to more closely control what scientists are allowed to say in studies that are to be used by the US government in forming policy in the areas of health and the environment.

The peer review system, whereby a scientific article is scrutinized by a scientist's colleagues - actually often by an anonymous selection of "guardians of orthodoxy" who work with the major scientific magazines - has already come under fire for hampering the open scientific discussion of new ideas. With government getting into the act to "select" the peer reviewers, we may see science degrading into a political tool, rather than a way to ascertain the truth about issues of interest.

Not that government is new to attempts at "damage control" by changing scientific findings. Here a recent report from AmericanProgress.org.

"DOCTORING DATA: A new report reveals that political appointees in the Department of Health and Human Services (HHS) doctored a study that revealed racial disparities in healthcare. The report, commissioned by Rep. Henry Waxman (D-CA) and seven colleagues, compares the draft report prepared by HHS scientists to the final version, edited by political appointees. In an effort to minimize the problem, political appointees deleted key sections of the report. Deleted: conclusion by HHS scientists that healthcare disparities are "national problems." Deleted: findings by HHS scientists on the social costs of healthcare disparities (including lost productivity, needless disability and early death). Deleted: key examples of health care disparities, including findings that racial and ethnic minorities are more likely to be diagnosed with late-stage cancer, die of HIV and be subjected to physical restraints in nursing homes."

January 22, 2004

White House Seeking Control Over Scientific Peer Review of Environmental, Health Research Which Shapes Federal Policies

Environmental and health studies conducted for or used by the federal government would require White House approval before their release, under a proposal now under review at the Office of Management and Budget (OMB). The plan would also give the White House authority to select which scientists take part in the system known as peer review -- the process by which fellow researchers evaluate the validity and reliability of studies before they are published.

Critics fear such a plan would undermine the impartiality of research that guides government policies and regulations. For example, it would open the door for the Administration to hand-select industry-friendly scientists to review studies that investigate the safety of chemicals in our food and consumer products, or studies that examine the environmental impact of energy plant emissions. The White House has frequently expressed its commitment to easing regulations for American industries. [1]

In a January 9 letter to the OMB, 20 former top federal agency officials, from both Democratic and Republican administrations, urged the White House to drop its proposal. The letter -- signed by former EPA Administrators Carol Browner and Russell Train; former Secretary of Labor Robert Reich; former Assistant Secretaries for Occupational Safety and Health Eula Bingham and Gerard Scannell; and others -- warned that the proposal, "in its current form, could damage the federal system for protecting public health and the environment." [2]

Currently, each federal agency controls peer review of its own projects. The government's rules to ensure research quality are already less stringent than those used by leading biomedical journals. For example, these journals require authors to disclose who paid for the research; and the journals will only publish studies done under contracts in which the investigators have the right to publish regardless of the results. Federal agencies do not have these requirements, nor do they consistently attempt to find out who paid for the studies. [3]

Far from ensuring the validity of the peer review process, the plan's critics assert that allowing the White House to control it would only add a layer of politics to what should be a purely scientific process. [4]

###
SOURCES:
[1] "White House Seeks Control on Health, Safety," St. Louis Post-Dispatch, Jan. 11, 2004.
[2] Letter to Joshua B. Bolton, Director of the OMB from 20 former agency officials, Jan 9, 2004.
[3] Michaels D, Wagner W. Disclosure in Regulatory Science. Science. 2003; 302:2073.
[4] "Peer Review Plan Draws Criticism," Washington Post, Jan. 15, 2004.

from: http://www.bushgreenwatch.org/mt_archives/000033.php

- - -

How to Fix Peer Review
Separating its two functions - improving manuscripts and judging their scientific merit - would help
(published in The Scientist)

By David Kaplan

Despite its importance as the ultimate gatekeeper of scientific publication and funding, peer review is known to engender bias, incompetence, excessive expense, ineffectiveness, and corruption. A surfeit of publications has documented the deficiencies of this system.1-4 In September, the fifth in a series of international congresses concerned with how peer review can be improved will convene in Chicago. Yet so far, in spite of the teeth gnashing, nothing is being chewed.

Investigation of the peer-review system has failed to provide validation for its use.1 In one study, previously published articles were altered to disguise their origin and resubmitted to the journals that had originally published the manuscripts.5 Most of these altered papers were not recognized and were rejected on supposed "scientific grounds." Other investigators found that agreement among reviewers about whether specific manuscripts should be published was no greater than would be expected by chance alone.6

Peer review subsumes two functions. First, peer reviewers attempt to improve manuscripts by offering constructive criticisms about concrete elements such as the application of a technique, the strength of results, or the cogency of an argument. The second function of peer review is to render a decision about the biological significance of the findings so that the manuscript can be prioritized for publication. I propose reforming peer review so that the two functions are independent.

Review of a manuscript would be solicited from colleagues by the authors. The first task of these reviewers would be to identify revisions that could be made to improve the manuscript. Second, the reviewers would be responsible for writing an evaluation of the revised work. This assessment would be mostly concerned with the significance of the findings, and the reviewers would sign it.

After receiving the final assessments from several different reviewers, the authors could decide to submit to a journal, sending the manuscript and the signed reviews together. The editors, carrying out the second function of peer review, would then decide to publish or not based solely on this material. The reviewers' identities would be revealed in the publication.

I believe there would be several significant effects of this change in peer review. First, the authors would submit only positive assessments. Consequently, reviews would emphasize why a manuscript should be published instead of why it shouldn't be. Second, investigators would be less likely to publish insignificant findings. They would have to ask colleagues to put their names on the manuscript; consequently, the tendency would be to ask for support for more complete and more compelling sets of findings.

Third, reviewers would be forced to account for their comments. They could not perform just a cursory look without the authors realizing the review was not insightful and did not represent an honest effort. Fourth, although it would be possible to have close friends and relatives review a manuscript, the editors would see who was supporting publication. In their deliberations, the editors would consider the breadth of the reviewers and their relationships to the authors and to the conceptualization promulgated in the manuscript.

Fifth, the editors would be free from adjudicating between authors and reviewers. They could concentrate on the specific arguments put forth for publication. Moreover, the process would be considerably streamlined, since there would be no need to send the manuscript out for review. This revision of peer review would change the incentives for all involved. The authors would tend to publish results that represent more complete findings and be more satisfied with the outcome, because they could exert lots of control over the review process. The reviewers would tend to be more honest in their evaluations, not wanting to praise work they consider flawed, because their names would be attached to it. Reviewers would not give a cursory and willfully negative evaluation, because the authors could simply not forward their comments. It would be in the reviewers' best interests to help improve manuscripts that have flaws but are potentially important.

The editors would emphasize publication of manuscripts that have the broadest support among scientists in the relevant community or that have the greatest potential to influence the community. Their jobs would be easier because the number of manuscripts submitted would be fewer, although of more substance. This tendency would be facilitated by editors' publicizing the stringent acceptance requirements. For example, editors could request manuscripts with support from reviewers from the same institution and from other institutions. They could request reviewers in the same field and reviewers in related fields.

Peer review is broken. It needs to be overhauled, not just tinkered with. The incentives should be changed so that: authors are more satisfied and more likely to produce better work, the reviewing is more transparent and honest, and journals do not have to manage an unwieldy and corrupt system that produces disaffection and misses out on innovation.

References
1. T Jefferson et al, "Measuring the quality of editorial peer review," J Am Med Assoc 2002, 287: 2786-90. [Publisher Full Text]