Improvement of the AUC metric in classifier ROC analysis

Matjaž Majnik (2011) Improvement of the AUC metric in classifier ROC analysis. EngD thesis.

Preview

PDF Download (1585Kb)

Abstract

AUC measure, which is used for classifier evaluation and represents one of the main tools of ROC analysis, has certain shortcomings. It ignores scores of instances and considers only their ranking order. The consequence is its unreliability in evaluating instance sets, where the differences between scores are negligible. Another disadvantage of the AUC is its low informative quality when mutually comparing sets containing the same number of errors. For these reasons, researchers have proposed improvements of the AUC, which also take score values into account. In this thesis we address four such measures. However, it has been ascertained that they do not solve all the problems and may even introduce new ones. In the case of these variants, the improper influence of the attributes of considered instance sets on their behavior may arise. The essential purpose of this thesis has been to acquire a new measure from the basic AUC and its derived variants eliminating the disadvantages mentioned and offering trustworthy and more informative evaluation of classifier performance. To achieve the goal, we also have intended to control and adjust the abovementioned influences of the instance sets. Thus, the main result of this thesis is the new measure for the evaluation of classifiers, which has been tested in supervised environment. Considering the diversity of scores obtained from different classifiers we have included parameters allowing us to adjust the performance of the new measure. As a result, the measure should be more generally applicable. On the basis of final comparison of the variants and taking some presumptions into account we believe that the newly proposed measure has certain advantages over the existing variants.

Actions (login required)