Počet záznamů: 1
When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review
- 1.
SYSNO ASEP 0541889 Druh ASEP J - Článek v odborném periodiku Zařazení RIV J - Článek v odborném periodiku Poddruh J Článek ve WOS Název When Zero May not be Zero: A Cautionary Note on the use of Inter-rater Reliability in Evaluating Grant Peer Review Tvůrce(i) Erosheva, E. (US)
Martinková, Patrícia (UIVT-O) SAI, RID, ORCID
Lee, C. J. (US)Celkový počet autorů 3 Zdroj.dok. Journal of the Royal Statistical Society Series A-Statistics in Society. - : Wiley - ISSN 0964-1998
Roč. 184, č. 3 (2021), s. 904-919Poč.str. 16 s. Forma vydání Online - E Jazyk dok. eng - angličtina Země vyd. US - Spojené státy americké Klíč. slova Bayesian estimation ; grant peer review ; inter-rater reliability ; maximum likelihood estimation ; measurement ; mixed-effects models Obor OECD Statistics and probability CEP GA21-03658S GA ČR - Grantová agentura ČR Způsob publikování Open access Institucionální podpora UIVT-O - RVO:67985807 UT WOS 000641375900001 EID SCOPUS 85104530726 DOI 10.1111/rssa.12681 Anotace Considerable attention has focused on studying reviewer agreement via inter‐rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top‐quality grant proposals, we use real data from a complete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for assessing peer review quality. First, we demonstrate that estimating local IRR from subsets of restricted‐quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top‐quality proposals rather than bottom‐quality proposals are considered. However, zero estimates from range‐restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, despite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero. Pracoviště Ústav informatiky Kontakt Tereza Šírová, sirova@cs.cas.cz, Tel.: 266 053 800 Rok sběru 2022 Elektronická adresa http://dx.doi.org/10.1111/rssa.12681
Počet záznamů: 1