Attribute evaluation on imbalanced data sets

Domen Rački (2011) Attribute evaluation on imbalanced data sets. EngD thesis.

Preview

PDF Download (1420Kb)

Abstract

We analyze the performance of attribute evaluation measures on imbalanced datasets at different levels of imbalance. We sample real world datasets at ratios 1:5, 1:10, 1:50, 1:100, 1:500 and 1:1000. We build decision tree models and for each attribute evaluation measure compute AUC with stratified 5x2 cross validation. To test significance of the difference we use Friedman's test. With Nemenyi's test we determine and graphically display the similarities and differences. We find that the best performing measure at unaltered class ratios is MDL, for class ratios 1:5 the best measure is the angular distance. For ratios 1:10 and 1:50 the beast measure is ReliefF and for class ratios 1:100, 1:500 and 1:1000 the best performing measure is information gain. The worst performing measure on all class ratios is accuracy.

Actions (login required)