Testing for the Presence of Authorship Bias in Peer Review

NCT02739737 | Completed

• 1 other identifier: 46129-EA
• Study Type: interventional
• Target: 119
• Locations: 0 countries
• Sites: N/A

Brief Summary

No consensus exists among biomedical journals on the subject of blinding during the peer review process. Some journals attempt to remove all identifiers of authorship from potential manuscripts before delivering to referees for peer review while others prefer to leave the authorship transparent. Although peer review remains the gold standard for manuscript evaluation, the lack of a standardized blinding process between editorial offices may be a source of publication bias and make comparisons of manuscripts published in different journals more difficult to interpret. Proponents of a blinded peer review system believe that knowledge of authorship may leave referees vulnerable to biases about those authors' previous research, perceived expertise, institution, nationality, or gender. This shifts judgment away from the merits of the scientific work and introduces pre-conceived notions about the identity and background of the author into the review process. Conversely, supporters of transparent authorship argue that knowledge of author identity makes it easier for the referee to provide more appropriate critiques. Open author identification allows referees to compare the current manuscript to previously published work by the author, and to recognize or identify potentially important conflicts of interest. The investigators therefore propose an experimental study to address the question of whether blinding affects the likelihood a reviewer will recommend acceptance of a study being peer reviewed. The investigators plan to work in partnership with a journal to have a fabricated manuscript peer reviewed by a large number of reviewers; the authors will send this "test" manuscript out either in blinded form (authors' names/institutions not shown to reviewers) or in open form, with the names of several well-known, well-regarded authors and their institutions visible to peer reviewers. The manuscripts will otherwise be identical. The partnering journal, Clinical Orthopaedics and Related Research allows both open and blinded peer review, regularly employing both approaches (authors can choose what kind of review they prefer when they submit to CORR), and so the reviewers are unlikely to be troubled by seeing a manuscript in either format. The primary goal of this study is to determine whether the identification of a manuscript with a prominent group of authors (open authorship) will affect the likelihood that reviewers will recommend the manuscript for publication. Additionally, the investigators plan to purposely place several errors in the manuscript, and tally how often reviewers detect these mistakes, to determine whether reviewers read the work of prominent authors less critically. The investigators will also compare the reviewers' grading of the (identical) methods section, to determine whether the identification of a manuscript with prominent authors results in higher reviewer grades for methodological rigor. This study tests the following specific hypotheses:

1. 1.The visibility of prominent author names and institutions on test manuscripts will be associated with increased likelihood that the manuscript will be recommended for publication by peer reviewers. (Primary Study Endpoint)
2. 2.The visibility of prominent author names and institutions will be associated with a decreased likelihood that purposefully placed "errors" in the experimental manuscript will be detected by the reviewers.
3. 3.The visibility of prominent author names and institutions will be associated with increased scores given by reviewers for the methods section, despite the fact that the methods sections of the experimental manuscript will be identical.

Conditions

Peer Review

Keywords

Bias; Orthopedics; Peer Review

Outcome Measures

Primary Outcomes (1)

• Manuscript acceptance rate

  13 months

Secondary Outcomes (2)

• Error detection rate

  13 months

• Methods section score

  13 months

Study Arms (2)

Blinded Reviewers

EXPERIMENTAL

Subjects receiving randomized sham manuscript for review

Other: Blinded manuscript

Unblinded Reviewers

EXPERIMENTAL

Subjects receiving randomized sham manuscript for review

Other: Unblinded manuscript

Interventions

Blinded manuscript OTHER

Blinded version of sham manuscript

Blinded Reviewers

Unblinded manuscript OTHER

Unblinded version of sham manuscript

Unblinded Reviewers

Eligibility Criteria

• Age: 30 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• Registered peer reviewers in relevant subspecialty sections of CORR's reviewer database.

You may not qualify if:

• Reviewers who opt out (electronically, by phone, or by post).

Sponsors & Collaborators

• University of Washington: lead

Related Publications (6)

• van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999 Jan 2;318(7175):23-7. doi: 10.1136/bmj.318.7175.23.

  PMID: 9872878 BACKGROUND

• McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990 Mar 9;263(10):1371-6.

  PMID: 2304216 BACKGROUND

• Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998 Jul 15;280(3):240-2. doi: 10.1001/jama.280.3.240.

  PMID: 9676668 BACKGROUND

• Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review. JAMA. 1994 Jul 13;272(2):143-6.

  PMID: 8015127 BACKGROUND

• Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med. 2010 Nov 22;170(21):1934-9. doi: 10.1001/archinternmed.2010.406.

  PMID: 21098355 BACKGROUND

• Okike K, Hug KT, Kocher MS, Leopold SS. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA. 2016 Sep 27;316(12):1315-6. doi: 10.1001/jama.2016.11014. No abstract available.

  PMID: 27673310 DERIVED

Study Officials

• Seth L Leopold, MD

  University of Washington

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: DOUBLE
• Who Masked: PARTICIPANT, OUTCOMES ASSESSOR
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Professor, Dept of Orthpaedics and Sports Medicine

Study Record Dates

• First Submitted: April 4, 2016
• First Posted: April 15, 2016
• Study Start: December 1, 2013
• Primary Completion: August 1, 2015
• Study Completion: August 1, 2015
• Last Updated: April 15, 2016

Record last verified: 2016-04

Data Sharing

• IPD Sharing: Will not share