Feature Selection for Performance Estimation of Machine Learning Workflows

Abstract

Performance prediction of machine learning models can speed up automated machine learning procedures and it can be also incorporated into model recommendation algorithms. We propose a meta-learning framework that utilizes information about previous runs of machine learning workflows on benchmark tasks. We extract features describing the workflows and meta-data about tasks, and combine them to train a regressor for performance prediction. This way, we obtain the model performance prediction without any training, just by means of feature extraction and inference via the regressor. The approach is tested on OpenML-CC18 Curated Classification benchmark estimating the 75th percentile value of area under the ROC curve (AUC) of the classifiers. We were able to obtain consistent predictions with $R^2$ score of 0.8 for previously unseen data.