For our next set of Monday afternoon talks on March 3rd, Lars Grunske will give a talk on his research area “Understanding Software and DAW Input Spaces”, and we will have a guest lecture by Paul Ralph titled “Evidence Standards Improve Reliability in Scholarly Peer Review“. The talks will take place in the Humboldt-Kabinett of Johann-von-. Neumann-Haus, Rudower Chaussee 25, 12489, Berlin.
Abstract:
Background. Scholarly peer review is “the lynchpin about which the whole business of science is pivoted” (Ziman 1968). Most researchers believe peer review is effective (Ware 2008), but empirical research consistently shows that reviewers cannot reliable distinguish methodologically sound from fundamentally flawed studies (Cole 1981; Peters & Ceci 1982; Lock 1991; Rothwell and Martyn 2000; Price 2014; Ralph 2016). Consequently, we created comprehensive evidence standards and tools to improve peer review in software engineering and related fields. Objective. The objective of this study is to investigate the impact of evidence standards on scholarly peer review. Method. A randomized controlled experiment was conducted at an A-ranked software engineering conference. The program committee was randomly divided into two groups: one using a typical conference review process; the other using a standardized process based on the ACM SIGSOFT Empirical Standards for Software Engineering Research (https://acmsigsoft.github.io/EmpiricalStandards/) Results. Evidence standards significantly improve inter-reviewer reliability without harming authors’ or reviewers’ attitudes toward the review process. Reviewers using evidence standards gave more praise and focused more on research methods than style. Discussion. Asking reviewers to write free-text comments about a paper and score it on a 6-point scale from strong reject to strong accept produces data statistically indistinguishable from random noise. This means that decisions are determined entirely by reviewer selection, not the merits of the research. Conventional review processes are therefore scientifically and morally indefensible. While not a silver bullet, evidence standards significantly improve reliability, and the data collected in this study facilitates further refinement of the standards and tooling toward still greater reliability.
Dr. D. Paul Ralph, PhD (British Columbia), B.Sc. / B.Comm (Memorial), is an award-winning scientist, author, consultant, and Professor of Software Engineering at Dalhousie University. His cutting-edge research at the intersection of software engineering, human-computer interaction, and project management explores the relationship between software teams’ social dynamics and success.
