Widget for measuring classifier performance based on curves

Miha Biček (2009) Widget for measuring classifier performance based on curves. EngD thesis.

PDF Download (1374Kb)

Abstract

In machine learning we are often faced with a task of evaluating and comparing classifiers based on their performance. Alongside scalar measures various graphical methods are being used. Graphical methods such as Receiver Operating Characheristic Curve (ROC) or Precision-Recall (PR) are already well known and used in the machine learning community. This thesis explores the options of using alternative graphical methods to help us explore and compare classifiers. The first chapter describes two of most commonly used graphical methods along with the algorithm to produce the set of points for this methods. Later on it depicts examples of two less conventional methods that could be used along side ROC and PR curve. End of this chapter covers methods that help us estimate the measure of variance. The second chapter begins with a description of a generic algorithm that can be used to define methods to evaluate and compare classifiers. Afterwards it describes an implementation of this algorithm as a widget in a data mining software called Orange. It also describes the user interface of the widget and some guidelines how to use it. The third chapter gives some examples how the widget could be used. The fourth chapter is the conclusion of the thesis. It consists of assessing if the goals of the thesis were met and pointing out some flaws of the widget.

Actions (login required)