Testing Heterogeneity in Inter-Rater Reliability

0531172 - ÚI 2021 RIV CH eng C - Conference Paper (international conference)
Bartoš, F. - Martinková, Patrícia - Brabec, Marek
Testing Heterogeneity in Inter-Rater Reliability.
Quantitative Psychology. Cham: Springer, 2020 - (Wiberg, M.; Molenaar, D.; González, J.; Böckenholt, U.; Kim, J.), s. 347-364. Springer Proceedings in Mathematics & Statistics, 322. ISBN 978-3-030-43468-7. ISSN 2194-1009.
[IMPS 2019: The Annual Meeting of the Psychometric Society /84./. Santiago (CL), 15.07.2019-19.07.2019]
Grant - others:GA MŠk(CZ) LM2015042
Institutional support: RVO:67985807
Keywords : Inter-rater reliability * Mixed-effect models * Generalized estimating equations
OECD category: Statistics and probability

Estimating the inter-rater reliability (IRR) is important for assessing and improving the quality of ratings. In some cases, the IRR may differ between groups due to their features. To test heterogeneity in IRR, the second-order generalized estimating equations (GEE2) and linear mixed-effects models (LME) were already used. Another method capable of estimating the components for IRR is generalized additive models (GAM). This paper presents a simulation study evaluating the performance of these methods in estimating variance components and in testing heterogeneity in IRR. We consider a wide range of sample sizes and various scenarios leading to heterogenous IRR. The results show, that while the LME and GAM models perform similarly and yield reliable estimates, the GEE2 models may lead to incorrect results.
Permanent Link: http://hdl.handle.net/11104/0309894