An Empirical Investigation of the Impact of Discretization on Common Data Distributions

Proceedings of the Third International Conference on Hybrid Intelligent Systems (HIS’03): Design and Application of Hybrid Intelligent Systems, A. Abraham, M. Koppen and K. Franke (Editors),Pages 692-701, December 2003, IOS Press.An Empirical Investigation of the Impact ofDiscretization on Common DataDistributionsMichael K. Ismail Vic CiesielskiDepartment of Computer ScienceRMIT University, MelbourneVIC 3001, AUSTRALIAAbstract.This study attempts to identify the merits of six of the most populardiscretization methods when confronted with a randomly generated datasetconsisting of attributes that conform to one of eight common statistical distributions.It is hoped that the analysis will enlighten as to a heuristic which identifies the mostappropriate discretization method to be applied, given some preliminary analysis orvisualization to determine the type of statistical distribution of the attribute to bediscretized. Further, the comparative effectiveness of discretization given each datadistribution is a primary focus. Analysis of the data was accomplished by inducing adecision tree classifier (C4.5) on the discretized data and an error measure was usedto determine the relative value of discretization. The experiments showed that themethod of discretization and the level of inherent error placed in the class attributehas a major impact on classification errors generated post-discretization. Moreimportantly, the general effectiveness of discretization varies significantlydepending on the shape of data distribution considered. Distributions that are highlyskewed or have high peaks tend to result in higher classification errors, and therelative superiority of supervised discretization over unsupervised discretization isdiminished significantly when applied to these data distributions.IntroductionDiscretization of continuous attributes not only broadens the scope of the number of datamining algorithms able to analyze data in discrete form, but can also dramatically increasethe speed at which these tasks can be performed [1]. There have been many studies thathave evaluated the relative effectiveness of the many discretization techniques along withsuggested optimizations [4,6,7]. However, these experiments have usually been carried outwith application to real-world datasets where no attempt has been made to identify the typeof probability density function of the continuous attribute.The effectiveness of various discretization methods as applied to a variety of data will be evaluated. All data has been artificially constructed and conform to common statisticaldistributions. Discretization and data mining operations will be performed using the WEKAdata mining software [11]. The comparative responsiveness of each distribution todiscretization is the primary focus of the experiments and will be achieved by the analysisof classification errors resulting from discretization. Thus, it is hoped that by datavisualization alone we can make inference as to the effectiveness of any subsequentdiscretization apriori and if applicable, isolate the most supportive discretization method toadopt. There are eight statistical distributions chosen that will be analyzed, along with sixdiscretization methods.1. Data Generation MethodologyEach attribute was formed with 10,000 instances as this provided accurate measures in terms of achieving the desired symmetry in distributions. Each class attribute was formed with an inbuilt error or class overlap (1%, 5%, 10%, 20%, 35%).For instance, if we have a numeric attribute, for example personal assets that contains 10,000 instances ranging from $50,000,000 to $50, for a 2-class label problem (sad, happy) we find the median value 5000th e.g. ($50,000). If we say that everyone below the median value (poor) is sad and everyone above the median value (rich) is happy, then anyone who is poor but happy and rich but sad contravenes our hypothesis. To illustrate this point further, consider the dataset of 10 instances in Table 1.Table 1. 0% error overlap dataset with 10 instancesValue 2 4 6 8 10 12 14 16 18 20Class small small small small small large large large large Large Intuitively, we can see that any sensible discretization method should recognize that a partition should be placed between values 10 and 12 as the class label changes from small to large. If all datasets were this simple our work would be done. However if we swap the two middle class labels (Table 2) there is some confusion as to where we should place partitions.Table 2. 20% error overlap dataset with 20 instancesAttribute 2 4 6 8 10 12 14 16 18 20Class small small small small large small large large large large Given that the middle 2 out of 10 values in the dataset are in dispute, we have used the phrase ‘error overlap’ to represent the level of uncertainty deliberately placed into the class labels when forming each class. Hence, the error overlap for Table 2 is 20% or 2/10. Figure 3 demonstrates as personal assets increase, at some point there is a change of mood from sad to happy, with the error overlap represented by the intersection of class 1 and class 2.Class 1 - sad Class 2 - happypersonal assetsFigure 1. 5% class error overlap (class 1 class 2) with 10,000 instancesTable 3 shows the number of errors built into the data (shaded rows) as the error overlap increases for a dataset of 10,000 instances.Table 3. Built in classification error rates2 class problem 1% 5% 10% 20% 35%Poor & sad 4950 4750 4500 4000 3250Poor & happy 50 250 500 1000 1750Rich & sad 50 250 500 1000 1750Rich & happy 4950 4750 4500 4000 3250where Poor personal assets < $50,000Rich personal assets > $50,000For 3 and 5 class label attributes a similar methodology was followed to build in an error rate into the data. For instance, 3 classes in the above case would result in class labels of: sad, content, happy and for a 5 class problem: very sad, sad, content, happy, very happy. 2. Data DistributionsFigure 2. Normal Figure 3. Uniform Figure 4. Leptokurtic Figure 5. PlatykurticFigure 6. Bimodal Figure 7. Skewed Figure 8. Exponential Figure 9. ZipfMany datasets conform to a Normal Distribution as depicted in Figure 2. They are characterized by a symmetrical bell shaped curve with a single peak at the median value which is also approximately equal the mean value. The normal probability distribution sets a benchmark for center, spread, skewness, and kurtosis that other shapes/distributions are measured. The probability density function used to generate a normal distribution is given in [10].Figure 3 shows a randomly generated Uniform Distribution. Uniform distributions are characterized by a roughly equal frequency count across the data range as every value has the same probability of occurrence.Figure 4 shows a Leptokurtic Distribution. This distribution is similar to a normal distribution however, the central area of the curve is more pronounced with a higher peak. Thus, more data values are concentrated close to the mean and median value. A co-efficient of kurtosis of magnitude greater than 3 formally recognizes a data’s distribution as being leptokurtic [10].The Platykurtic Distribution is the opposite of a leptokurtic distribution; it is characterized by a flatter than normal central peak and as such a more evenly symmetrical distribution of values across the data range as exhibited in Figure 5. Formally, if the coefficient of kurtosis is significantly less than three, it is regarded as platykurtic. Note, a uniform distribution is a special case of a platykurtic distribution.A Bimodal Distribution is a distribution that contains two distinct peaks or modes. Unlike the previous distributions, the data can become asymmetric. Two distinct peaks are prevalent in Figure 6, either can be regarded as the mode.In the real world, many datasets will not exhibit the symmetry of a normal distribution. Very often there will be a greater proportion of values either side of the mean or the distance between the mean and the extreme values either side of the mean differ. When this occurs, we are said to have skewed data. Formally, if the coefficient of skewness is significantly greater/less than zero, we have a skewed distribution [10]. Figure 7 depicts a positively Skewed Distribution characterized by a long right tail.Exponential Distributions are often used to model expectancy and lifetimes such as the life expectancy of humans, car engines, or light bulbs as depicted in Figure 8.A Zipf Distribution follows a straight line when frequency and rank are plotted on a double-logarithmic scale, highlighting the inverse correlation between ordered frequencies. An example of data that exhibits this pattern is the frequency of words in a book. Words such as ‘the’ and ’and’ have very high frequency counts in most long text files, whilst other words such as ‘alleviate’ occur rarely. As a Zipf distribution, all the words would be sorted into frequency order where each spike in the histogram represents a word. Figure 9 depicts an approximated Zipf distribution where there are many items with few occurrences and few items with many occurrences.4. Discretization Techniques4.1 Unsupervised DiscretizationThe simplest means of discretizing continuous features involves a class blind approach where only knowledge of the feature to be discretized is required. Here the user specifies the number of intervals or bins.4.1.1 Equal Interval Width/Uniform BinningThis method relies on sorting the data and dividing the data values into equally spaced bin ranges. A seed k supplied by the user determines how many bins are required. With this seed k, it is just a matter of finding the maximum and minimum values to derive the range and then partition the data into k bins.4.1.2 Equal Frequency/Histogram BinningPartitioning of data is based on allocating the same number of instances to each bin. Dividing the total number of instances n by k the number of bins supplied by the user does this.4.2 Supervised DiscretizationSupervised discretization on the other hand makes use of the instance/class labels during the discretization process to aid in the partitioning. Prior knowledge of the class label of each instance is incorporated at each iteration to refine partition breakpoint estimation of each bin.4.2.1 Holte’s 1R DiscretizerThis method uses an error-based approach using error counts to determine when to split intervals [4]. The attribute is sorted into ascending order and a greedy algorithm that divides the feature into bins where each contains only one instance of a particular class label is used. The danger inherent in such a technique is that each instance may end up belonging to a separate bin. To combat this problem, a minimum number of instances of a particular class for each bin as specified by the user (except for the upper most bin) is implemented. Hence any given bin can now contain a mixture of class labels andboundaries will not be continually divided leading to overfitting. Each bucket grows i.e. the partition shifts to the right until it has at least 6 instances of a class label, and continues until the instance to be considered is not part of the majority class label. Empirical analysis [4] suggests a minimum bin size of 6 performs the best.4.2.2 C4.5 DiscretizerC4.5 uses a bottom up approach that builds a complete tree and then prunes the number of intervals [8]. Information gain theory is used to determine at which threshold value the gain ratio is greatest in order to partition the data. A divide and conquer algorithm is then successively applied to determine whether to split each partition into smaller subsets at each iteration. Every non-leaf sub tree is examined from the bottom and if the predicted error is lower if a leaf replaces the sub tree, then a leaf replaces the sub tree.4.2.3 Fayyad and Irani’s Entropy Based MDL MethodThis method uses a top down approach whereby multiple ranges rather than binary ranges are created to form a tree via multi-way splits of the numeric attribute at the same node to produce discrete bins. Determining the exact cut off points for intervals is based on the Minimum Description Length Principle (MDLP) that biases a simpler theory that can explain the same body of data and favors a hypothesis which minimizes the probability of making a wrong decision assuming a uniform error cost. No pruning is applied to the grown tree.An information entropy/uncertainty minimization heuristic is used to select threshold boundaries by finding a single threshold that minimizes the entropy function over all possible thresholds [3]. This entropy function is then recursively applied to both of the partitions induced. Thresholds are placed half way between the two delimiting instances. At this point the MDL stopping criterion is applied to determine when to stop subdividing discrete intervals.4.2.4 Kononenko’s Entropy Based MDL MethodThis method is virtually identical to that of Fayyad and Irani except that it includes an adjustment for when multiple attributes are to be discretized. This algorithm provides a correction for the bias the entropy measure has towards an attribute with many values.5. Running Algorithms in WEKAThe dataset generated contained 8 attributes representing each data distribution. There are 15 class columns: 5 classes for the simplest 2 class problem with either a 1%, 5%, 10%, 20% or 35% error rate built into the class labels; and similarly 5 classes for a 3 class label and 5 class label problem. Each discretization method was tested with 1 attribute and 1 class column run together via the C4.5 classifier using 10-fold cross-validation. Each distribution was then analyzed by filtering out the appropriate columns with all 720 combinations run, (i.e. 6 discretization methods * 8 attributes * 15 classes).Equal width binning was run with the number of bins used set at the default setting of 10 bins. No attempt was made to optimize this value. The C4.5 algorithm was then applied to the discretized data to produce a tree. This classifier was chosen over a Bayesianclassifier as there are no normality constraints on the data required, especially relevant given the focus of the experiments.Equal frequency binning was run with the number of bins used set at the defaultsetting of 10 bins. The C4.5 algorithm was then applied to the discretized data to produce a tree. Holte’s 1R Discretizer was run using a default minimum bucket size set at the default setting of 6. C4.5 Discretizer was run using a confidence factor set to 0.25 as per the default value. Fayyad and Irani’s Entropy Based MDL Method was achieved using WEKA 3-2-3 with the use MDL parameter set to true. The C4.5 algorithm was used as the wrapper function. Kononenko’s Entropy Based MDL Method was achieved by setting the use MDL parameter to false. The C4.5 algorithm was used as the wrapper function.6. Analysis of ErrorsIn most practical situations, the best numerical prediction method is still the best no matter which error measure is used [11]. The root relative squared error was used as the basis for comparison. This measure gives extra emphasis to outliers and larger discrepancies are weighed more heavily than smaller ones. Further, it was able to detect subtle differences in the various error rates of each method, especially when the total error count or confusion matrix was the same. This measure is given by:r.r.s.e =(p 1 – a 1)2 + .. + (p n – a n )2 where p = predicted value, a = actual value, (a 1 – )2 + .. + (a n – )= 1/n i a i average value of a i .7. ResultsFigure 10. Normal dist. errors for 2-class labels Figure 11. Skewed dist. errors for 2-class labelsFigure 12. Unsupervised(Equal-width) vs. Supervised(Fayyad&Irani) average errors (Y-axis) for Exponentialand Uniform distributions plotted against number of class labels (X-axis) and error overlap (Z-axis)Figure 13. Average root relative squared error for all distributions and discretization methods7.1 Varying the Error OverlapFigure 10 plots the root relative squared error for each discretization method and error overlap for the 2-class label problem for a normally distributed continuous attribute. The first two unsupervised methods progressively deteriorate in predictive performance as the inherent error overlap is increased. On the other hand, the other methods have a stable performance as the error overlap is increased. Surprisingly, the MDL methods perform worse with an error overlap of 1% compared to when the error overlap is 35%. In contrast, Figure 11 shows a degenerative performance for the skewed distribution when the error overlap was increased, regardless of the discretization method used.7.2 Varying the Number of Class LabelsThe effect of increasing the number of class labels can be seen in Figure 11 where an example of the exponential and uniform distribution errors are highlighted when run with both an unsupervised (equal-width) and supervised method (Fayyad&Irani). When the number of class labels is increased from 2 to 3 and 5, the degenerative performance of the unsupervised methods becomes more pronounced. From Figure 12, the error rate rises sharply for both the exponential and uniform distributions. This pattern was replicated across the board for all distributions when unsupervised discretization was applied. The supervised methods provided mixed outcomes when the number of class labels increased. In Figure 12 we can see that the uniform distribution errors remained static for supervised discretization using Fayyad&Irani method even when the number of class labels increased. Only, the normal distribution followed this pattern (not shown). All other distributions followed a growth in the error rate similar to that shown for the supervised exponential errors in Figure 12.7.3 Comparison of Overall Error RatesFigure 13 shows a comparison of the performance of each discretization method as applied to each data distribution. Each bar represents the average root relative squared error bysumming across all overlaps and number of class labels and dividing by 15. The equal-interval binning technique appears to be more effective when the data distribution is flatter, as depicted by a uniform or platykurtic distribution. This concurs with the findings in [2] that this method of discretization is vulnerable to skewed data. Interestingly, the error rate of equal-interval binning converges to that of equal-frequency binning with flatter distributions as depicted in Figure 13. Equal interval binning often distributes instances unevenly across bins with some bins containing many instances and others possibly containing none. Figure 13 demonstrates the degenerative performance of equal-interval binning when the distribution being discretized is less even. It is thus no surprise that the leptokurtic and the zipf distribution are the worse performers applying this method, and the best performers being those distributions with a more even distribution, i.e. uniform and platykurtic distributions.Similarly, equal frequency binning can suffer from serious problems. Consider the event that we have 1000 instances and 10 intervals specified by the user. If there are 105 instances with a value between 10 and 20 corresponding to bin 2, then the remaining 5 instances are grouped into the next bin. This sort of occurrence undermines knowledge discovery in the sense that partitions then lose all semantic value. On the whole, the equal-frequency binning method shows relative indifference to the distribution of the data.The OneR discretizer works well with both a normal and uniform distribution. The OneR error rate is roughly 20% higher for all other distributions relative to the MDL methods. The problem with an error based method like OneR is that it may result in splitting intervals unnecessarily to minimize the error count and consequently blurs the understanding of classification results. Furthermore, multiple attributes that require discretization cannot be used with this algorithm.The C4.5 discretizer was the equal best performer for the normal and uniform distribution. This concurs with [5] who showed that discretization causes a small increase in error rate when the continuous feature is normally distributed. However, it was nearly 10% worse than the MDL entropy methods for the other six distributions. One factor that impacts on the overall tree generated is the confidence level. The default confidence parameter of 0.25 was used. Varying this value changes the level of pruning applied to the tree. The smaller this parameter the greater the level of pruning that occurs. It is a balancing act to prevent overfitting whilst trying to optimize this parameter in the context of knowledge discovery [5].The entropy based MDL methods combine both local and global information during learning, i.e. they use a wrapper function i.e. C4.5 in this case with the MDL criterion (global). The MDL pre-selects cut-points and the classifier decides which cut-point is the most appropriate in the current context. Using a global discretization as opposed to a local method (C4.5) makes the MDL methods less sensitive to variation from small fragmented data [2]. These methods were consistently high performers across all data distributions as depicted in Figure 13.7.4 Selection of the Number of BinsIn the experiments k bins set to the default value of ten was used. The choice of k has a major effect on error rates [6]. Identifying the best value of k to use is a trial and error method and cannot be applied universally to each attribute. In some cases, error rates may even compare favourably to supervised discretization, however, finding the optimum cut off thresholds by supervised discretization may involve 2an calculations where a is the number of attributes and n is the number of instances [7]. The unpredictability ofunsupervised methods as shown in the overall summary error rates for all distributions seems to make them a poor choice for discretization.Figure 14. Error rate for number of bins manually chosen for equal-width binning strategy for normaldistribution of 2 class problem with 35% error overlap.Figure 14 depicts the effect of the error rate on classification when the seed for an unsupervised method is varied. Here the seed represents a manually chosen number by the user that is the number of bins requested from the data. Figure 14 shows that the default value of 10 bins (used in all runs) is not the best selection of the number of bins in order to produce the smallest error rate in this instance but may be in others.7.5 Identifying a HeuristicInitial thoughts were that error rates were correlated positively with the level of kurtosis or skewness. The correlation coefficient for kurtosis against the root relative squared error was 0.45, indicating that there appears to be a significant relationship as shown in Figure 15. The relationship between skewness and the error rate as depicted in Figure 16 was less pronounced with the correlation coefficient at 0.22. Prudence is taken with influence regarding the size of the coefficients given the error measure used and the assumption of a linear relationship via a linear regression model. In general terms, we can deduce a positive relationship between the elevation (kurtosis) and symmetry (skewness) of the data and classification errors resulting from discretization. However, a word of caution is that discretization methods make a poor job of separating the two underlying populations in a variable that has a bimodal distribution [9].Figure 15. Kurtosis vs. average error rate : r2 = 0.45 Figure 16. Skewness vs. average error rate : r2 = 0.22 Via closer inspection of the shape of each distribution, it seems that distributions that have either:• a long thin tail (skewed, zipf and exponential distributions) or,• a high peak/concentration of data (leptokurtic, bimodal distribution)benefit the least from discretization. Similarly, these distributions gain the least from using a supervised method compared to an unsupervised method of discretization. For instance, the root relative squared error rate can be improved by 86% for a normal or uniform distribution, compared to 19% for a skewed distribution and 25% for a bimodal distribution (Fig.13).8. ConclusionFrom analysis the 720 runs it was found the distributions that benefit the most from discretization in ranked order are: (1) uniform, (2) normal, (3) platykurtic, (4) bimodal, (5) exponential, (6) zipf, (7) leptokurtic, (8) skewed.The higher ranked distributions, namely the uniform and normal distributions exhibit escalating error rates for the unsupervised discretization methods when the error overlap and/or the number of classes labels increases. These unsupervised methods were shown to be highly susceptible to an arbitrarily selected number of bins. In contrast, the error rates for the supervised discretization methods remained constant as both the error overlap and/or the number of class labels introduced increased.At the lower end of the ranked distributions, the classification error rates grew as the error overlap grew as expected, regardless of which discretization method was adopted. Unexpectedly, as more class labels were introduced, error rates increased at lower error overlap levels whilst remaining relatively static at the higher end of the error overlap level.In summary, we have identified a heuristic that is able to determine the relative effectiveness of discretization apriori by data visualization, and by establishing that a positive correlation between the level of kurtosis and skewness and the error rate from classification in the data distributions exists, with a caveat for bimodal distributions.In this work we examined classification errors based on a single discretized attribute. In future work we will look at combinations of discretized attributes.References1. Catlett, J. (1991b) “On Changing Continuous Attributes into Ordered Discrete Attributes”, InY.Kodrtoff, ed., EWSL-91. Lecture Notes in Artificial Intelligence 482, pp.164-178. Springer-Verlag, Berlin, Germany.2. Dougherty, J., Kohavi, R., Sahami, M. (1995) “Supervised and Unsupervised Discretization ofContinuous Features” Machine Learning: Proceedings of the 12th International Conference, Morgan Kaufmann, Los Altos, CA.3. Fayyad, U.M., Irani, K.B. (1993) “Multi-interval discretization of continuous valued attributes forclassification learning” In Proceedings of the 13th International Joint Conference on Artificial Intelligence, pp.1022-1027, Los Altos, CA: Morgan Kaufmann.4. Holte, R.C. (1993) “Very simple classification rules perform well on most commonly used datasets”5. Machine Learning 11, pp.63-90.6. Kohavi, R., Sahami, M. (1996) “Error-based and Entropy-based Discretization of ContinuousFeatures” In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, pp.114-119. Menlo Park:AAAI Press.7. Kononenko, I., Sikonja, M.R, (1995) “Discretization of continuous attributes using ReliefF”Proceedings of ERK'95 , Portoroz, Slovenia, 1995.8. Pazzani, M. (1995) “An iterative improvement approach for the discretization of numeric attributesin Bayesian classifiers” KDD-95 pp.228-2339. Quinlan, J.R. (1993) “C4.5 : Programs for Machine Learning”, Morgan Kaufmann, Los Altos, CA10. Scott, P.D., Williams, R.J., Ho, K.M. (1997) “Forming Categories in Exploratory Data Analysis andData Mining”, IDA 1997, pp.235-246.11. Smith, P.J. (1997) “Into Statistics” Springer-verlag, pp.205-206, 304-306, 323-329.12. Witten I., Frank, E. (2000) “Data Mining: Practical Machine Learning Tools and Techniques”Morgan Kaufmann, pp.80-82, 147-150, 238-246.。