1 Beyond Independence Probabilistic Models for Query Approximation on Binary Transaction Da

smyth@ Heikki.Mannila@ pavlovd@
We investigate the problem of generating fast approximate answers to queries for large sparse binary data sets. We focus in particular on probabilistic model-based approaches to this problem and develop a number of techniques that are signi cantly more accurate than a baseline independence model. In particular, we introduce a novel technique for building probabilistic models from frequent itemsets. The itemsets are treated as constraints on the distribution of the query variables and the maximum entropy principle is used online to build a joint probability model for attributes in the query. We show that the resulting probability model de nes a Markov random eld (MRF) and that the time taken to answer a query scales exponentially as a function of the induced width of the associated MRF graph. We empirically compare the MRF model to other probabilistic models, such as the independence model, the Chow-Liu tree model, the Bernoulli mixture model, and the ADTree model. Experimental results on both simulated and real-world transaction data sets demonstrate that the choice of a particular model depends on the parameters of the data set at hand and available resources. The results illustrate various fundamental tradeo s between approximation error, model complexity, and the online time required to compute a query answer. Binary transaction data, query selectivity estimation, probabilistic model, maximum entropy, Markov random eld.
I. Introduction
2
only a small fraction of pages, making the data set sparse. Within a single day popular Web sites can produce millions of records. Query selectivity estimation for such binary data can be de ned as follows. Let R = fA1; : : : ; Ak g be a table header with k 0/1 valued attributes (variables) and r be a table of n rows over header R. We assume that k n, and that the data are sparse i.e., the average number of 1's per row is substantially smaller than the number of attributes. By de nition, a row of the table r satis es a conjunctive query Q if and only if the corresponding attributes in the query and in the row have equal values. We are interested in nding the number of rows in the table r satisfying a given conjunctive query Q de ned on a subset of its attributes. We can view this query selectivity estimation problem in a probabilistic light and pose the problem as estimating the true frequency of Q in the table r using an approximate probability model PM . Query selectivity estimation is an important practical problem in the optimization of database management systems where query pro lers and query optimizers routinely rely on fast and reasonably accurate estimates of the query counts 1]. Query selectivity can also be important in interactive data mining applications and exploratory data analysis, where a data miner may be exploring various hypotheses in the data and would prefer to get an approximate count to a query in real-time rather than waiting for an exact count to be generated. Any Boolean query can be answered using a single scan through the data set. While this approach has linear complexity and works well for the small data sets, it becomes infeasible for real-time queries on huge data sets not residing in main memory. Consequently we would like to have approximate algorithms that allow us to trade accuracy in the estimate PM (Q) with the time and memory taken to calculate it. There are two main contributions of the paper: 1. We introduce and study a probabilistic modeling technique that is based on frequent sets (e.g., 2], 3]), demonstrate how maximum entropy can be used in a principled manner to infer in an online manner a joint probability model on the query attributes using the frequent sets, and prove that the resulting model de nes a Markov random eld (MRF). We further show how to alleviate the computational requirements for the maximum entropy online computation by applying bucket elimination and clique tree approaches to leverage any graphical structure that exists in either the models or the queries. 2. We perform a detailed experimental evaluation of various probabilistic models for query approximation, including the maximum entropy/MRF model, the independence model as a baseline, the Chow-Liu tree model, the Bernoulli mixture model, and the ADTree model based on the inclusionexclusion principle. Our experimental study allows us to draw a variety of conclusions on the general characteristics of each approach in terms of memory requirements, the online time taken to answer a query, and the accuracy of the resulting estimate. Previous work on this problem has mainly investigated the use of wavelets (see, e.g. 4], 5], 6]), multidimensional histograms ( 7], 1]) and sampling ( 8], 9]). However, there have been no systematic investigations of using di erent probabilistic modeling techniques for the query selectivity problem. In this paper we explore several variants of probabilistic models and investigate di erent aspects of their performance. A speci c class of probabilistic models, mixtures of Gaussian independence models, have previously been proposed for generating approximate queries on real-valued data sets from relatively lowdimensional data cubes (5 or fewer dimensions) 10]. We generalize this approach to include not only mixtures of independence models (for binary data) but also several other probabilistic models, and we demonstrate how these models can be used for approximate querying on data sets involving several hundred dimensions. The rest of this paper is organized as follows. In Section II we introduce some general notation and discuss the statistics used to compare the models. Section III introduces the maximum entropy model