Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance

0580447 - ÚI 2024 RIV BE eng A - Abstrakt
Štěpánek, Lubomír - Dlouhá, Jana - Martinková, Patrícia
Machine-learning prediction of test item difficulty using item text wordings: Comparison of algorithms’ and domain experts’ predictive performance.
The 10th European Congress of Methodology (EAM2023) Book of Abstracts. Ghent: Ghent University, 2023. s. 26-26.
[EAM2023: European Congress of Methodology /10./. 11.07.2023-13.07.2023, Ghent]
Grant CEP: GA ČR(CZ) GA21-03658S; GA TA ČR(CZ) TL05000008
Institucionální podpora: RVO:67985807
Klíčová slova: item difficulty * machine learning models * item text wording
Obor OECD: Education, general; including training, pedagogy, didactics [and education systems]
https://eam2023.ugent.be/images/eam2023_abstracts_book.pdf

ZÁKLADNÍ ÚDAJE: The 10th European Congress of Methodology (EAM2023) Book of Abstracts. Ghent: Ghent University, 2023. s. 26-26. [EAM2023: European Congress of Methodology /10./. 11.07.2023-13.07.2023, Ghent]. ABSTRAKT: Various properties of text wording of a given test item determine how difficult the item is for a test-taker. While the item difficulty is commonly estimated using item response theory (IRT) models based on test-takers’ responses, information on item difficulty is encoded in its text and could be predicted using machine-learning algorithms. In this work, we used text wordings of test items of the reading comprehension part of a test of English as a foreign language. For each item, we tokenized and lemmatized item text, removed stopwords, and calculated various features such as word counts, readability indices, lexical frequencies, and measures of item parts’ similarity. Then, the resulting dataset containing text features in rows was enriched by item difficulty estimated using the Rasch model. The item difficulty was predicted using multiple machine-learning supervised algorithms of regression task. Firstly, we applied regularization algorithms, i.e., LASSO, ridge regression, and elastic net, to select appropriate features, reduce dimensionality, and predict the (continuous) difficulty. Besides that, we employed support vector machines, regression trees and forests, and neural networks. Once we categorized the difficulty into disjunctive intervals, we switched the regression into a classification task, also applying the naïve Bayes classifier. To compare the algorithms to each other and domain experts’ difficulty predictions, we learned algorithms many times within cross-validation and estimated root mean square errors and predictive accuracies for each approach. Regularization algorithms in regression tasks and random forests in classification seemed to outperform other algorithms and predicted item difficulty similarly to domain experts
Trvalý link: https://hdl.handle.net/11104/0349220