Mixture of Experts of ANN and KNN on the problem of Puzzle 8

Mixture of Experts of ANN and KNN on the problem of Puzzle 8Guiwen Hou Jingyue Zhang Jiahong ZhouComputing Science Department University of Alberta Edmonton, Candada , T6G 2E8 {guiwen,jingye,jiahong}@cs.ualberta.caNovember 14,2002Abstract: In this project we present a heuristic learning process by training ANN(artificial neural network) and KNN (k-nearest neighbor) using the best number of steps, gained from A*, from randomly generated states to the goal. After training ANN and KNN, the mixture of Experts is discussed and the empirical data are collected to demonstrate the feasibility and accuracy of combination of ANN and KNN in different areas. In our project, we present a novel method to combination of ANN and KNN by using the confidences. In order to achieve this goal, different approaches are adopted to converge the ANN.Key words:ANN (Artificial Neural Network), KNN (k-nearest neighbor), Auto-encoding ANNContents1.Problem formulation and Challenges .....................................................................3 2.Related work .........................................................................................................4 2.1 8/15puzzle search.........................................................................................4 2.2 Artificial Neural Network and its confidence................................................5 2.3 Function Approximation...............................................................................5 2.4 KNN and its confidence ...............................................................................5 2.5 Mixture of experts........................................................................................6 3. Combination of ANN and KNN (Noveld Methods)...............................................6 3.1 Generate training samples by A*: .................................................................6 3.2 Build ANN and train ANN ...........................................................................7 3.3 Scaling .........................................................................................................9 3.4 Momentum...................................................................................................9 3.5 Learning rate..............................................................................................10 3.6 Using Auto-encoding network to calculate the confidence of ANN.............10 3.7 Ensemble of auto-encoding ANNs .............................................................12 3.8 Training KNN ............................................................................................13 4 Empirical result and analysis................................................................................15 4.1 Converge of ANN ......................................................................................15 4.2 Combine ANN and KNN ...........................................................................17 4.3 Apply this method to other domain.............................................................18 5 Summary .............................................................................................................21 5.1 Conclusion.................................................................................................21 5.2 Discussion..................................................................................................22 5.3 Future work................................................................................................22 Acknowledgement ..................................................................................................23 Bibliography...........................................................................................................241.Problem formulation and ChallengesAdmissible heuristics grantee the optimal path solutions in search algorithms such as A*, but typically the more accurate and effective the heuristic is, the more expensive the computation will be, especially in real-world application the bounded time and memory available is critical. Furthermore, finding an effective admissible heuristic function for certain domain sometimes requires a long time. Previous works have been done to use ANN as well as KNN to construct look-up tables as heuristic function to speed up the search. The traditional artificial feed-forward neural network (ANN) is a memoryless approach. This means that after training is complete all information about the input patterns is stored in the neural network weights and input data are no longer needed, i.e. there is no explicit storage of any presented example in the system. On the contrary, some methods as the k-nearest-neighbors (KNN) [37], the Parzen-window regression [40], represent the memory-based approaches. These approaches keep in memory the entire database of examples and their predictions are based on some local approximation of the stored examples. The neural networks can be considered global models, while the other two approaches are usually considered local models [44]. Both of the two approaches have advantages and disadvantages. A global model provides a good approximation of the global metric of the input data space. However, if the analyzed function, f, is too complicated, there is no guarantee that all details of f will be represented. Thus, the global model can be inadequate because it does not describe equally well the entire state space with poor performance of the method being mainly due to a high bias of the global model in some particular regions of space. The ANN variance can also contribute to poor performance of this method [39]. The local models are based on some neighborhood relations and these methods are more pertinent to exploring a fine structure of the analyzed function, i.e. they can have a lower bias compared to a global model. However, while applying these methods the difficult question is how to correctly determine the neighborhood relations in the analyzed space. The analyzed input data, especially in practical applications, can have large number of dimensions and the actual importance and contribution of each input parameter to the final representation is generally not known. Out of this consideration we propose a combined solution: calculate the distribution of confidences for the estimation of ANN and KNN and get the combination of the advantage of both. In our project, we use puzzle 8 as test bed. We present a heuristic learning process by training ANN and KNN using the best number of steps, gained from A*, from randomly generated states to the goal. After training ANN and KNN using training samples <s, h>, where s denotes state of puzzle 8, and h is the optimal steps tosolve the problem, the mixture of Experts is discussed and the empirical data are collected to demonstrate the feasibility and accuracy of this approach. In order to reach this goal there are certain obstacles we must settle down. First, even though the confidence of KNN constructor is obviously reachable, the feathers by which to quantify the effectiveness of the ANN constructor are hard to define. Secondly, how to make a compromise between the two constructors to give out satisfactory results, which outweigh the effort consumed. Thirdly, when we train samples <s, h> gained from A*, we found that only a little change in s will lead h to be totally different. It is very difficult to make ANN to converge to a little error. The accuracy of any single approximator will be low. So how to improve the accuracy of ANN and make it converge? The rest of the paper is organized as follows. First we will introduce the related work in this area, second we will present our approach to solve the problem, then we will introduce the empirical result, finally we will give some conclusions, discussions and future work.2.Related workIn this project, we focus on a mixture expert on single agent search. We use well-known 8 puzzle as our test bed by gaining the heuristic function through ANN and KNN, then combining them together and applying it to the puzzle search. So our related search include following:2.1 8/15puzzle search8/15 puzzle search have been fully explored among mathematicians and AI researchers since their advent. The sliding-tile puzzles are among the most commonly used domains for testing heuristic search algorithms . The reason for the popularity of sliding-tile puzzles is the simplicity of their specification combined with the very large size of their associated search space. An NxN puzzle problem has an associated search graph of size N2!/2. The optimizing NxN puzzle problem is NP-Hard, although some research efforts have been made in finding optimal solutions to the 15-puzzle [11] and the 24-puzzle [12]. It is not difficult to devise a domain-specific algorithm for solving the NxN puzzle problem [13]. Ian Parberry designed an algorithm using a greedy and divide and conquer techniques for n2-1 puzzle [4]. The basic idea to solve this kind of problems is to use heuristic search approach such as A*, STA*, STRA*,DTA*, IDA* etc. An effective heuristic function plays an important role in Puzzle search. Lev Finkelstein use a Selective Macro-learning Algorithm to solve NXN puzzle problem [5]. The basic idea of MICRO-HILLARY is to acquire macro-operators for escaping from local minimal, i.e., macros that lead from a local minimum to a state with a better heuristic value.2.2 Artificial Neural Network and its confidenceANN has proved to be an effective approach in solving no liner program such as pattern recognition, automatic control. How to find the confidence in the ANN is an interesting topic. SiRichard Dybowski had made a (apprehensive extensive) survey about assigning confidence intervals to feed-forward neural networks. He listed several methods, namely, bootstrap estimation, maximum likelihood estimation, and Bayesian statistics [16]., and also propose a mixture model via Markov Chain Monte Carlo sampling to enable non-Gaussian variances to (avoid) introducing the bias caused by maximum likelihood. Mitch Weintraub introduced a neural network based measures of confidence for word recognition [15]. Luren Yang and Paul F.M. de Groot [23] had compared and evaluated the confidence estimation methods for Neural Networks in various situations But all this approaches are use to calculate the confidence intervals based on distribution.2.3 Function ApproximationIn these days, many applications require the use of generalizing function approximators. There are many approaches to achieve this goal, such neural networks, decision-trees, or instance-based methods. Many efforts in function approximation have been made in estimating a value function, with the action-selection policy represented implicitly as the “greedy”policy with respect to the estimated values (e.g., as the policy that selects in each state the action with highest estimated value). The value-function approach has worked well in many applications, such as Tesauro's backgammon player [51], but has several limitations. An arbitrarily small change in the estimated value of an action can cause it to be, or not be selected. Such discontinuous changes have been identified as a key obstacle to establishing convergence assurances for algorithms following the value-function approach [50]. For example, Q-learning have all been shown unable to converge to any policy for simple MDPs and simple function approximators [52]. In our project, we try to build a function approximation for a states in puzzle 8 to get optimal steps to goal in order to direct each move at a state. We use bagging algorithm to speed up the converge of ANN.2.4 KNN and its confidenceNearest neighbor analysis examines the distances between each point and the closest point to it, and then compares these to expected values for a random sample of points from a CSR (complete spatial randomness) pattern. In the k - nearest neighbor rule, a test sample is assigned the class most frequently represented among the k nearest training samples. If two or more such classes exist, then the test sample is assigned the class with minimum average distance to it. It can be shown that the k - nearest neighbor rule becomes the Bayes optimal decision rule as k goes to infinity [19]. A KNN algorithm doesn't really "train" on the data; all of its work is performed in the prediction process.2.5 Mixture of expertsThere is some research on combining artificial neural network. Basically there are two approaches ensemble-based and modular [22]. An ensemble is made of a set of nets, each of which is a general function approximator. Modular method uses a divide and conquers technique to improve the performance. Differences between these approaches center mainly lie on how to create a diverse ensemble and how to combine the component classifiers. The most extensively studied approaches are Bagging by Breiman [31] and Boosting by Freund and Schapire [34]. Maclin and Opitz [49] have shown that Bagging and Boosting can improve the accuracy of neural networks. However, all the methods are focus on how to combine an ensemble of ANNs. As far as we know no past work related to the mixture of two experts of ANN and KNN as combined heuristic estimate exists. In this project we present a new technique to combine of ANN and KNN, and give some analysis of the empirical result and guidance for further research.3. Combination of ANN and KNN (Noveld Methods)3.1 Generate training samples by A*We have designed A* algorithm to generate the training samples for ANN and KNN. The training samples are outputted into a file with the following format. 6 4 2 7 0 8 3 1 5 18 4 5 6 3 1 8 7 0 2 16 . 20.97 seconds elapsed. Where each line represents a training sample, the first 9 numbers stand for the random generated initial state; the last one denotes the optimal steps to goal. For instance, the first line above, we denote the following initial state: 6 7 3 4 0 1 2 8 5The optimal steps from this state to goal state are 18. Then each line would be a sample item and the outputted file consists of 1000 samples of this kind with the time elapsed for computation recorded at the end of the file. We observe that usually 1000 samples take about 20 seconds to generate using A* with Manhattan distance as heuristic function. Hopefully after we train the ANN and KNNwith these samples, we can build up approximators, which can output estimation more accurate than Manhattan distance, thus by comparing the time consuming we can get the empirical result which would guide the work of mixture of ANN and KNN.3.2 Build ANN and train ANNIn this project, we use back propagation neural network to approximate the heuristic function. An input pattern is presented to the network, and the output pattern is computed within the network. The error vale is propagated backward [49] through the network so that the weights of connections between the layers of units are adjusted to accurate the output next time when a input pattern comes. This process is repeated continuously until the difference between the output pattern and target output pattern is very small.Figure 1. An example of neural network. Let’t introduce the back-propagation algorithm briefly. See figure 1. The dash line in above digram represents a neuron, which can be either a hidden or the output neuron. The outputs of n neurons (O 1 ...O n) in the preceding layer provide the inputs to neuron B. If neuron B is in the first hidden layer, then (O 1 ...O n) is simply the input vector. These outputs are multiplied by the respective weights (W1B...WnB), where WnB is the weight connecting neuron n to neuron B. The summation function adds together all theseproducts plus bias b to provide the input, IB, which is processed by the transfer function f (.) of neuron B. f (IB) is the output, OB, of neuron B. Following three funtions are the most commonly used transfer functions for backpropagation, but other differentiable transfer functions can be created and used with backpropagation if desired hese are several common used transfer functions.f = 2/(1+exp(-2*n))-1f = 1/(1+exp(-n))Figure 2. Four most common transfer function in ANN.Multilayer networks often use the log-sigmoid transfer function logsig. The function logsig generates outputs between 0 and 1 as the neuron's net input goes from negative to positive infinity. Alternatively, multilayer networks may use the tan-sigmoid transfer function tansig. The function logsig generates outputs between 0 and 1 even if the neuron's net input goes from negative to positive infinity. In our ANN, we use logsig function to represent transfer function in hidden layers in ANN, and use liner function to present transfer function in output layer. Before training a feed forward network, the weights and biases must be initialized. In our project, we set a scope to generate the weights and biases randomly by defining a range of the initial weights.3.3 ScalingSignals within the network are scaled to a range and precision, which suits the hardware or software platform on which the network is implemented.Figure 3 Scaling For example, a neural network implementation may converge quickly if the values of the attributes of each vector in training set lie in certain scope. Conversely, outputs from the network would be rescaled back to the range and precision expected by the outside world: In our project, w use a vector with 9 attributes to express the position of 8 puzzles. We have used the program to test that the maximum steps from a state to goal is 28. We normalize the output of ANN for training samples by dividing the output by 14 and then minus 1.This is the formula. O O = ( − 1) Where O is the output 14 Now we can have the scope of [-1,1]. Also we divide every number in each vector by 9, so that every attribute in each vector is between [0, 1]. These are some training samples 0.4444 0.8889 0.3333 0.7778 0.1111 0.6667 0.5556 0.2222 0.0000 0.4286 0.0000 0.1111 0.3333 0.4444 0.8889 0.2222 0.7778 0.5556 0.6667 0.0000 0.4444 0.0000 0.2222 0.7778 0.6667 0.5556 0.1111 0.8889 0.3333 0.3571 0.0000 0.6667 0.2222 0.3333 0.5556 0.8889 0.1111 0.4444 0.7778 0.2857 0.0000 0.2222 0.3333 0.1111 0.7778 0.6667 0.5556 0.4444 0.8889 -0.4286 0.4444 0.3333 0.6667 0.7778 0.1111 0.8889 0.5556 0.2222 0.0000 -0.1429 0.8889 0.5556 0.1111 0.4444 0.2222 0.3333 0.7778 0.6667 0.0000 0.2857 When we predict an output given a input pattern, we rescale back for the output usingO = (O + 1) • 14 O is the original output3.4 MomentumThe approximation used for the weight change is given by the delta rule: ∆w(t ) = −ε∂E / ∂w(t ) + α∆w(t − 1) where ∂E / ∂w(t ) is the error derivative for that weight, accumulated over the whole epoch., ε denotes the learning rate; α denotes the momentum factor. This derivative can represent the "slope" of that weight. Back propagation is very slow, especially when ε is small. If the learning rate ε is toolarge, the gradient descent will oscillate widely. One technique to speedup the learning process of a neural net and to avoid undue oscillations involves the use of a momentum term. The idea is to give each connection w a momentum, so that the gradient descent tends to change in the direction of the average downhill direction that it encounters, instead of oscillating.3.5 Learning rateIn our project, we also try to find a reasonable learning rate ε , for each individual weight as the training proceeds. First we give an initial value for the ε . When the program sees that the slope of the error surface averages out to be in the same direction for several iterations for a particular weight the program increases the ε value by an amount, κ . The network will then move down this slope faster. When the program finds the slope changes signs, it assumes that the program has stepped over to the other side of the minimum and so it cuts down the learning rate by the decay factor d. For instance, a d value of 0.5 cuts the learning rate for the weight in half. The running average at iteration i, ai is defined as: ai = (1 - θ ) * ∆Wi + θ * ai-1 Where ∆Wi is the slope of weight, θ a parameter which must be in the range 0<= θ <1.3.6 Using Auto-encoding network to calculate the confidence of ANNAuto-encoding (all so called auto-associative) neural nets are neural nets that try to learn the identity mapping. This means that the input is also the required output. When the hidden layer(s) between the input and output layer have fewer units, the network gets a hour-class shape. This kind of network can be used for dimension reduction because the network has found a more compact representation of the input data. It is clear that when fewer units in the hidden layer(s) are used the net will have to find a more complex representation of the data, so learning times will increase. At a certain point the network can no longer learn the required encoding and more units in the hidden layer(s) are needed. The hour-class shape of the network can be regarded as a joining of two networks. The first net will perform a encoding of the data to a lower dimension, while the second net will perform a decoding of the reduced data to the original data dimension. shape of the network can be regarded as a joining of two networks. The first net will perform a encoding of the data to a lower dimension, while the second net will perform a decoding of the reduced data to the original data dimension.Figure 4. An example of Auto-encoding ANN A key feature of the architecture is that the number of units in the middle layer must be less than the number of inputs (and outputs): otherwise the network would simply copy the inputs to the outputs. The smaller hidden layer forces the neural network to learn any relationships within the input data, and compresses the input data to a number of parameters equal to the size of the middle layer. In training, the network weights are adjusted until the outputs match the inputs, and the values assigned to the weights reflect the relationships between the various input data elements. This property is useful in, for example, data validation: when invalid data is presented to the trained neural network, the learned relationships no longer hold and it is unable to reproduce the correct output. Ideally, the match between the actual and correct outputs would reflect the closeness of the invalid data to valid values. This model can be use to capture the distribution of the given data set [48]. So in our project, we build an auto-encoding neural network to reproduce the states of training samples. We use the same samples to train ANN and auto-encoding ANN so that the auto-encoding can reproduce the original states given a state in samples. When a new state comes, we assume the ability that the auto-encoding ANN can reproduce the original state reflects the accuracy of ANN. Following the approach to get the confidence of ANN Offline 1) Train ANN using samples <s,h>, where s is the state of puzzle 8, h is the optimal steps from state s to goal. 2) Train Auto-encoding ANN using the same samples Online 1) Solving the puzzle using A*, but using the heuristic function gained by combination of ANN and KNN instead of Manhattan Distance. 2) Get a new state s. 3) Feed s into ANN to get a h’ 4) Feed s into Auto-encoding ANN to get s’ 5) Measure the distance d between s and s’ using Manhattan distance in our project 6) Measure the difference α by Manhattan distanceα =d max{d1 , d 2, ..., dn}where di is ith state’s Manhattan distance7) Confidence P = 1-α3.7 Ensemble of auto-encoding ANNsIn our auto-encoding network, we try to reproduce the state S’ give a state S. We hope that our network work well and the difference between S’ and S is very small, so we can use it to calculate the confidence of ANN, see above. But because of the hidden nodes in auto-encoding ANN is less than the input units. We do not want to provide too many hidden units, which allow the network to memorize the training examples rather than extracting the general features that will allow it to handle cases it has not seen during training. Neither do we want to force the network to spend a lot of extra time trying to find an optimal representation. In our project, we use 9-5-9 structure for auto-encoding ANN. Following is some training samples ( has been scaled). 0.4444 0.8889 0.3333 0.7778 0.1111 0.6667 0.5556 0.2222 0 0.4444 0.8889 0.3333 0.7778 0.1111 0.6667 0.5556 0.2222 0 But we find it is very hard to converge. After training 50000 iterations using 1000 samples, the average errors for each unit of out is 0.21. We also try 9-7-4-7-9, it only a little better than 9-5-9, the average error for each unit is 0.17, but it takes a longer time to train. So in our project, we choose 9-7-4-7-9 structure and try to use an ensemble of auto-encoding ANN to improve the accuracy of the estimation. Bagging approach is to build up composite classifier based on the voting of the component classifiers. With an ensemble of resampled training sets, bagging applies a basic learning algorithm to construct an ensemble of component classifiers. To classify an unlabeled pattern, it uses simple voting to combine the predictions of the component classifiers and assigns the output of the pattern to the class receiving the maximum votes. If two or more classes have same maximum vote, then one is randomly chosen one. The idea here is quick simple. Supposed that we have several diverse ANN, which can make different errors on new states. To see why accuracy and diversity are good, imagine that we have an ensemble of three ANN {A1; A2; A3} and consider a new state S comes. If the three ANNs are identical, then when A1 (S) is wrong, h2 (S) and h 3 (S) will also be wrong. However, if the errors made by the ANNs are uncorrelated, then when A1 (S) is wrong, A2 (S) and A3 (S) may be correct, so that a majority vote will correctly classify S. More precisely, if the error rates P <= 1/2, then more ANNs in an ensemble, the more probability that we have majority of correct predictions. Figure 5 [53] demonstrates this using a simulatedensemble of 21 classifiers, each having an error rate of 0.3. We can see that the probability of 1. 2, 3, … , 10 classifiers simultaneously wrong is high, and the probability that 11 classifiers wrong simultaneously is 0.026, which is much less than the error rate of the individual hypotheses.So the key point to combine the ensemble of ANN is to construct individual ANN with error rates below 0.5 whose errors are at least somewhat uncorrelated. In our project, we do a little revision by average the outputs of an ensemble of networks. First build 10 neural networks.We generate 10 x 1000 samples randomlyTrain each neural network in different 1000 samples with random initial weights When a new state comes, use this ten neural network to calculate the outputs and average them to get the output of the pattern3.8 Training KNNKNN is a simple algorithm that stores all training samples and classifies the new instance of patterns based on the distance of K nearest neighbors. In our project, training samples are pairs of state and heuristic value, which can be got from A*. When a new state comes, we calculate the distances between this state and all states in training set, find nearest K states, then classify the new state (in this project, get the heuristic value) based on the K states.There are two commonly used distances: one is Euclidean, the other is Manhattan distance. The confidence is percentage of samples in K that have most frequent heuristic value. Suppose there are n different classes in the dataset. For each sample with unknown class, the algorithm looks into a neighborhood of the unknown for k samples in the training set. If within that neighborhood, more samples lie in class i than any other classes, the unknown sample is then assigned as class i. The confidence level for each class is calculated by Ki/K, where Ki is the number of training samples that belong to class i within the neighborhood, and k is the total number of training samples in the neighborhood. Different values of K's might generate different confidence levels of the unknown belonging to a certain class. A majority voting is conducted to identify the class that majority of the K's vote for. In our project, we use Manhattan distance to calculate the similarity of the two states. Given a new state, we want to predict the heuristic value (steps to goal steps) and a confidence of the estimation. Because it is very difficult to find the same heuristic value in K nearest samples, the approach of calculating the confidence for the classification by voting is not suitable to our project. We use weighted KNN to get the heuristic value and confidence. Store all input/output pairs in the training set Search for the K nearest patterns to the input patterns using a Manhattan distance For estimation of heuristic function of new state S, the output value H is based on the average of the weighted output values of the K nearest patterns:H = ∑W i • Hii =1kWhere k is the number of nearest neighbor, Hi is the heuristicfunction for the ith neighbor, Wi is the weight of ith neighbor in the classification. Wi = Di / ∑ Di1 kWhere Di is the distance for ith neighbor to the new state SThe confidence for estimation is。