When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review

0541889 - ÚI 2022 RIV US eng J - Článek v odborném periodiku
Erosheva, E. - Martinková, Patrícia - Lee, C. J.
When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review.
Journal of the Royal Statistical Society Series A-Statistics in Society. Roč. 184, č. 3 (2021), s. 904-919. ISSN 0964-1998. E-ISSN 1467-985X
Grant CEP: GA ČR(CZ) GA21-03658S
Institucionální podpora: RVO:67985807
Klíčová slova: Bayesian estimation * grant peer review * inter-rater reliability * maximum likelihood estimation * measurement * mixed-effects models
Obor OECD: Statistics and probability
Impakt faktor: 2.175, rok: 2021
Způsob publikování: Open access
http://dx.doi.org/10.1111/rssa.12681

Considerable attention has focused on studying reviewer agreement via inter‐rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top‐quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for assessing peer review quality. First, we demonstrate that estimating local IRR from subsets of restricted‐quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top‐quality proposals rather than bottom‐quality proposals are considered. However, zero estimates from range‐restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, despite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero.
Trvalý link: http://hdl.handle.net/11104/0319384