Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability

0585407 - ÚI 2025 eng J - Článek v odborném periodiku
Bartoš, František - Martinková, Patrícia
Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.
British Journal of Mathematical & Statistical Psychology. Online 15 April 2024 (2024). ISSN 0007-1102. E-ISSN 2044-8317
Grant CEP: GA ČR(CZ) GA21-03658S; GA MŠMT(CZ) EH22_008/0004583
Institucionální podpora: RVO:67985807
Impakt faktor: 2.6, rok: 2022
Způsob publikování: Open access
https://doi.org/10.1111/bmsp.12343

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.
Trvalý link: https://hdl.handle.net/11104/0353116