Complement Ther Med. 1999 Mar; 7(1): 19-23.

Reviewer bias against the unconventional? A randomized double-blind study of peer review.

Ernst E, Resch KL.

University of Exeter, Department of Complementary Medicine, UK.

OBJECTIVE: To test the hypothesis that there is a reviewer bias against publication of a test of an unconventional drug. DESIGN: Randomized, controlled, double-blind study of peer review. PARTICIPANTS: Convenience sample of 291 medical doctors from a wide variety of specialties drawn from a list of conference participants. METHODS: Reviewers were randomly assigned to receive one of two versions of a manuscript. Version M related to an in-vitro experiment on a mainstream drug (Metoprolol). The otherwise identical version V used a highly unconventional drug (beef spleen cell extract) for the same experiment. Reviewers were asked to complete a standardised evaluation sheet including visual analogue scales (VASs) on a set of predefined quality criteria. All participants were debriefed after completion of the study. RESULTS: The response rate was 61%. There were no significant differences in VAS ratings between the two versions of the manuscript. Ratings covered the entire range of the VASs. CONCLUSION: In the present setting, there was no evidence for a reviewer-bias against testing an unconventional drug. The low inter-rater reliability, however, suggested inadequate validity of peer review.