Γ-IRT : an item response theory model for evaluating regression algorithms

Abstract

Item Response Theory (IRT) is used to measure latent abilities of human respondents based on their responses to items with different difficulty levels. Recently, IRT has been applied to algorithm evaluation in Artificial Inteligence (AI), by treating the algorithms as respondents and the AI tasks as items. The most common models in IRT only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). Hence they are not adequate in application contexts where responses are recorded in a continuous scale. In this dissertation we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination.