Reliability estimation of individual predictions for incremental models

Anže Javornik (2016) Reliability estimation of individual predictions for incremental models. EngD thesis.

Preview

PDF Download (2782Kb)

Abstract

The thesis deals with the reliability estimates of individual predictions of incremental models. The purpose of reliability estimates is to enrich the models predictions with additional information. This information might have a critical meaning, especially if wrong predictions can have serious consequences. Classical prediction models are built on examples from a problem domain for which they provide predictions. Classical methods of the reliability estimation are constructed in a similar way, on examples representing the learning set for the prediction model. Models and reliability estimate scores built in this way in short term provide good results. In the long term use there is a high probability that a change happens in the problem domain that adversely affects the performance of conventional predictive models and estimates of reliability. Building a new predictive model to adapt to the changes can be time consuming or even not an option, if the data stream is such that we can observe each example only once – e.g. network communication. Therefore, for the analysis of such problem domains we choose to use the incremental prediction models, which are able to adapt to changes. In this thesis we present three known methods of the reliability estimation and propose their versions with an incremental character. These methods were tested on twenty-two fixed problem domains and eight data streams (generators) with and without drift, which represents a change in the problem domain. The resulting estimates were statistically analysed. In the statistical analysis, we were interested in statistically significant correlations between factual error and the reliability assessment. Obtained results show that the proposed incremental methods of reliability estimation offer almost the same results in fixed problem domains and much better results in data streams, especially if the drift is applied. Best results are offered by the reliability estimate iCNK. Good results are also offered by the reliability estimate iBAGV. An incremental version of the reliability estimate LCV, iLCV, does not deviate from its classical version.

Actions (login required)