ePrints.FRI - University of Ljubljana, Faculty of Computer and Information Science

Evaluating reliability of individual classifications with local methods

Darko Pevec (2009) Evaluating reliability of individual classifications with local methods. EngD thesis.

[img] PDF
Download (1364Kb)


    In this thesis we take upon different approaches for estimating reliability of individual classification predictions made by classifiers based on supervised learning. The general definition of the term reliability is the ability to perform required functions under stated conditions. In machine learning, we refer to accuracy, as in the ability to provide accurate predictions. We face the problem that measures of reliability are not quantitatively defined. We can therefore only conceive estimates. Reliability estimates of individual predictions provide valuable information that could be beneficial in individual predictions assessment of use. For the needs of our thesis we develop several methods for reliability estimation based on existing approaches of local methods and the variance of a bagged model. We test our methods on various available real-life and artificial datasets and compare our methods with those based on inverse transduction. Methods were tested on 20 different datasets on 7 classification models and the estimates were calculated using 11 measures of similarity. We applied three statistical methods to our results. We came to a conclusion that these tests do not give clear results, as Q-Q plots only vaguely support calculated correlation. Correlation tests show potential of estimates LCV and BAGV as they demonstrated best on average performance. Second-comers with good result were estimates TRANS1 and CNK, while other estimates failed to excel.

    Item Type: Thesis (EngD thesis)
    Keywords: machine learning, classification, reliability estimates, classification accuracy.
    Number of Pages: 86
    Language of Content: Slovenian
    Mentor / Comentors:
    Name and SurnameIDFunction
    prof. dr. Igor Kononenko237Mentor
    Link to COBISS: http://www.cobiss.si/scripts/cobiss?command=search&base=50070&select=(ID=7296596)
    Institution: University of Ljubljana
    Department: Faculty of Computer and Information Science
    Item ID: 904
    Date Deposited: 09 Sep 2009 15:38
    Last Modified: 13 Aug 2011 00:35
    URI: http://eprints.fri.uni-lj.si/id/eprint/904

    Actions (login required)

    View Item