Attribute Interactions in Machine Learning

Abstract

To make decisions, multiple data are used. It is preferred to decide on the basis of each datum separately, afterwards joining these decisions to take all data into consideration, for example by averaging. This approach is effective, but only correct when each datum is independent from all others. When this is not the case, there is an interaction between data. An interaction is true when there is synergism among the data: when the sum of all individual effects is smaller than the total effect. When the sum of individual effects is lower than the total effect, the interaction is false. The concept of an interaction is opposite to the concept of independence. An interaction is atomic and irreducible: it cannot be simplified or collapsed into a set of mutually independent simpler interactions. In this text we present a survey of interactions through a variety of fields, from game theory to machine learning. We propose a method of automatic search for interactions, and demonstrate that results of such analysis can be presented visually to a human analyst. We suggest that instead of special tests for interactions, a pragmatic test of quality improvement of a classifier is sufficient and preferable. Using the framework of probabilistic classifier learning, we investigate how awareness of interactions improves the classification performance of machine learning algorithms. We provide preliminary evidence that resolving true and false interactions improves classification results obtained with the naive Bayesian classifier, logistic regression, and support vector machines.