Using Item Response Theory to evaluate feature relevance in missing data scenarios

Abstract

Item Response Theory (IRT) has been historically used to evaluate the latent abilities of human respondents to a set of items. Recently, e orts have been made to propose solutions that use IRT to solve classification problems, where the respondents are classifiers and the items are the instances of a dataset. Most of the initial works that tried to tackle this problem used a dichotomous IRT model, which is capable of modelling the classification problem only in terms of correct and wrong predictions. B3-IRT o ers a powerful tool to analyze datasets and classifiers, as the response is continuous, so instead of representing the predictions just as right or wrong answers, the response can be represented by the probability of a correct prediction. Although the IRT formulation can provide rich information about the behavior of the models towards the instances of a dataset, no previous work has investigated the application of IRT to rank features in an instance-based approach, or even to evaluate how missing data can impact the IRT parameters for instances (diculty and discrimination) and classifiers (ability). We propose a workflow that uses B3-IRT in missing data scenarios to evaluate the relevance of features both locally for each instance of a dataset, and globally for the whole dataset. In this workflow, data is missing at test time, and missing values are filled out with imputed values, in order to evaluate how much the missing data can a ect the ability of classifiers and di culty and discrimination of instances. This novel application represents an alternative to feature selection and feature ranking techniques that is capable to provide an overview of feature relevance both at global and instance level.