Information-Theoretic Exploration and Evaluation of Models

Abstract

No information-theoretic quantity, such as entropy or Kullback-Leibler divergence, is meaningful without first assuming a probabilistic model. In Bayesian statistics, the model itself is uncertain, so the resulting information-theoretic quantities should also be treated as uncertain. Information theory provides a language for asking meaningful decision-theoretic questions about black-box probabilistic models, where the chosen utility function is log-likelihood. We show how general hypothesis testing can be developed from these conclusions, also handling the problem of multiple comparisons. Furthermore, we use mutual and interaction information to disentangle and visualize the structure inside black-box probabilistic models. On examples we show how misleading can non-generative models be about informativeness of attributes.