A Neural Network decision module for the Adaptive Strategic Planning Framework in RoboCup

A Neural Network decision module for the Adaptive Strategic PlanningFramework in RoboCupNikolaos Kalyviotis and Huosheng HuDepartment of Computer Science, University of Essex,Wivenhoe Park, Colchester CO4 3SQ, UKEmail: nkalyv@, hhu@Abstract: Co-operation and decision making play important roles in Multi-Agent Systems(MAS). This paper proposes a new decision module for the Adaptive Strategic PlanningFramework (ASPF) [1] in the RoboCup domain [4]. This new module is to improve the co-operative behaviour of the Essex Wizards simulation team that has been based on a purelyanalytical approach in the past. The open architecture of the ASPF shows how artificial neuralnetworks (ANNs) can be used in the decision module for a good performance. It also shows thatthe neural network decision module can outperform the decision module that is based on handcoded one.Keywords: Neural Networks, Multi-robot Systems, Simulation, RoboCup1. IntroductionRoboCup (Robot Soccer) is an initiative that started six years ago. The purpose of this initiative is to promote Artificial Intelligence (AI) and robotics research [4], providing a common test-bed (soccer), where various agent theories and algorithms can be tested. The technologies that are involved in RoboCup cover a wide spectrum of research such as design principles of autonomous agents, multi-robot collaboration, strategy acquisition, real-time reasoning and planning, intelligent robotics, sensor-fusion [3, 5, 6]. RoboCup is divided in four main leagues, namely Simulation league, Small size robots league, Middle size robots league, and Sony legged robot league.In Simulation league agents are purely synthetic and have great flexibility and many design and implementation possibilities. In addition the domain that RoboCup provides is very interesting for MAS research, since it comprises of a unique combination of characteristics [7]. The fundamental issue for RoboCup is to design a multi-agent system that behaves in real time, performing reasonable goal-directed behaviours. As the situations of the game change quickly and the state-space is huge, soccer agents should be able to play the game strategically [6]. Therefore, three challenges have to be accomplished: learning challenge, teamwork challenge, and opponent modelling challenge.In MAS planning is an important element that can result in efficient co-operative behaviours. In general, planning concerns the process for people to select a course of action, making choices, formulating and revising plans, dealing with problems and adversity, and eventually performing some actions [8]. As planning involves reasoning about actions, it has been approached by an AI point of view. This approach that begun in late sixties and early seventies gave birth to the “classical AI planning problem”[9]. Sometimes however, planning systems do not have to create a plan. They just have to retrieve a plan from their memory that is applicable to the problem they are trying to solve. These plans are called “canned plans” by Wilensky [8].The human brain can be considered as a very powerful and complex parallel computer [10]. Therefore it would be very beneficial if we could simulate the functionality of such a powerful processing system. The need to understand and to simulate the way that the human brains work, lead to the introduction of neural networks that can be considered as information processing systems. In general, ANNs are used to develop adaptive systems. An ANN is a massively distributedProc. of International Symposiumon Robotics and Automation, Toluca, Mexico, 1-4 September 2002processor that has a natural property for storing experiential knowledge and making it available for use. It resembles the brain in two respects. The first one is the knowledge that is acquired by ANNs through a learning process. The second one is the inter-neuron connection strength, known as synaptic weights, which are used to store the knowledge.The efficiency and power of ANNs have made them very popular. As a result of that ANNs have been used in many different kind of applications with great success, including classification, pattern matching, pattern completion, noise removal, optimisation and control [11]. For example in the medical area, ANNs have been used in the diagnosis and treatment of breast cancer [13]. In economics ANNs have been used in order to forecast the prices of the shares, and also to allocate assets in the global bond markets [14]. ANNs have also been used in telecommunications. An example is a system, which uses ANNs for routing traffic [15].In this paper, ANNs have been adopted to implement a decision module for the Adaptive Strategic Planning Framework (ASPF) in the RoboCup domain. In the next section the basic idea of the ASPF is briefly outlined. Section 3 describes how a decision module based on ANNs can be built. Some experimental results are presented in section 4. Finally brief conclusions and future work are summarised in section 5.2. Overview of the ASPFThe ASPF [1] is an extension of the Basic Strategic Planning [17], involves a mapping of“plans”to “situations”. It can be seen as many-to-one relationship such that a set of particular plans is mapped to a particular situation. A plan is mapped (or assigned) to a situation only if that situation is important. For instance, the importance of a situation can be determined by the frequency of its occurrence. An important situation is called “Strategic Situation”and the plans mapped to that are called “Global Strategic Plans”. In RoboCup, like in any other MAS, there are certain situations that occur repeatedly. Therefore a great advantage can be gained if plans are devised for such situation beforehand.In the Essex Wizards team, ASPF is used before the agent enters the actual thinking process (figure 1). If an agent is a part of a Global Strategic Planning (GSP) that can be executed, then the ASPF module will produce an action and sent it to the actuators. Otherwise the agent will enter the thinking process, and then the thinking process will produce an action and send it to the actuators.A situation that happens many times during a game is the “goalie clearance”situation. For this strategic situation we have devised 10 different GSPs that can be executed, as can be seen in figure 2.At some points the goalie should make a decision of which GSP will be executed. So far an analytical (hand coded) approach has been used. This approach is based on a risk factor RG (X) which shows how risky is to execute GSP X. In the process of decision between the ten GSPs, the risk factor of each GSP (RG) is calculated and then the GSP with the least RG value is chosen providing that the wining RG value is below a threshold that we have set. This decision module was used in the Essex Wizards 2001 simulation team [2]. If the winning value is above that threshold then no GSP is chosen. In the following section we will see how the decision module can be built using ANNs.Fig. 1. A high level view of the Essex Wizards agent3. Neural Network ApproachThe problem of choosing the best GSP in the “goalie clearance”situation can be viewed as a classification problem since each GSP can be classified as a good or a bad choice according to the state of the environment. In this section we will show how neural networks were used in order to build a decision mechanism for the goalie clearance situation.3.1 The methodologyWhen the goalie catches the ball, it can keep it no more than 250 cycles. During those 250 cycles the environment changes and therefore the effectiveness of each GSP changes as well. For instance at time T a good choice could be GSP 7, while at time T+100 GSP 10 could be the best choice to make. If we assume that an evaluation function E(x, t) gives the effectiveness of GSP X at time T, then during the 250 cycles there will be a cycle T best and a GSP X best that will produce the maximum value for E(x, t). In other words GSP X best at cycle T best will be the best choice during the 250 cycles that the goalie can keep the ball. As it was mentioned earlier on, the environment can change in every cycle and therefore at time T best the environment will be at state S best. So every time that the goalie catches the ball (i.e. in each “goalie clearance” situation) we can associate state S best with GSP X best (S best, GSP X best). If we gather many sets of such data, we can use them in order to train a neural network that uses the S best data as training examples and the GSP X best data as the correct output. Then the neural network should be able to decide whether at a state S one of the GSPs should be executed or not.3.2 The evaluation functionIn order to find the GSP that performs best at a given time we should be able to evaluate the performance of each GSP and choose the one with the best performance. For that to be done we willTeam-mate Goalie Ball Pass RunFigure 2. Ten GSPs for the “goalie clearance” strategic situationneed an evaluation function E(x, t) that will evaluate the performance of GSP X at time T. The evaluation function that we have constructed uses four criteria that are explained below:a) The risk factor RG(x, t) of GSP X at time T: thisshows how risky is to execute GSP X at time T.The value of RG(x, t) is a real number between 0 and 1. A value close to 1 means that there is a high risk for the GSP X to fail, while a value close to 0 means that the risk to fail is low.b) The congestion CON(x, t) of GSP X at time T: thisshows how much congested of opponents at time T is the player of GSP X that is going to receive the ball last. The value of CON(x, t) is a real number between 0 and 1. A value close to1 denotes a low congestion while a value closeto 0 denotes a high congestion.c) The usefulness USE(x, t) of GSP X at time T: thisshows how much useful is GSP X at time T based on the position of the player that is going to receive the ball last. The value of USE(x, t) isa real number between 0 and 1. A value close to1 denotes a position close to opponent’s goal,while a value close to 0 denotes a position close to our goal.d) The cost COST(x, t) of GSP X at time T: the valueof COST(x, t)is related to the number of passes involved in GSP X and is calculated as follows: COST(x, t) = 1/(number of passes involved in GSP X) Now we can define the evaluation function E(x, t) as follows:E(x, t) = (1-RG(x, t)) * 0.35 + CON(x, t) * 0.35 +USE(x, t) * 0.2 + COST(x, t) * 0.1The outcome of the evaluation function is a real number between 0 and 1. The closer to 1 it is, and the better is the performance of GSP X at time T. The values 0.35, 0.2, and 0.1, show the importance of each factor towards the calculation of E(x, t). For example RG(x, t) and CON(x, t) are the most important factors and therefore they get the value 0.35. The other two factors are less important and so they are assigned lower values.In figure 3 we can see the evaluation of the 10 GSPs for the “goalie clearance” situation in the time period of 250 cycles where the goalie keeps the ball. In this example the best performance is achieved by GSP 10 at time 100 (point A).3.3 The topology of the neural networkThere are three layers in the neural network that we have used, namely input – hidden – output, as shown in figure 4:The input layer: As it was mentioned earlier on in order to train the network we are using sets of (S best, GSP X best) data. The information that we are using in order to describe the state S best of the environmentFigure 3. Evaluation of the 10 GSPs for the “goalie clearance” situationis based on the first twocriteria of theevaluation function,namely the risk factorRG(x, t) and the CON(x, t)congestion. Althoughwe have 10 differentvalues of RG(x, t) for the10 available GSPs, weonly have 6 differentvalues of CON(x, t),since the number ofplayers that receive theball last in the 10 GSPsis 6. For example inGSP 1, GSP 3, GSP 4and GSP 10 the playerwho receives the balllast is always the same.Therefore we can describe the environment state S best by using these 16 numbers. Hence the input layer of the network will have 16 neurons, one neuron for each one of the 16 numbers.The output layer: The output layer of the neural networks has 10 neurons and each neuron corresponds to one GSP. The correct output presented in these neurons is 1 for the node that represents GSP X best and 0 for all other neurons.The hidden layer: The analysis of the results showed that 10 neurons in the hidden layer could produce satisfactory results.3.4 The training of the neural network96 sets of (S best, GSP X best) data were used in order to train the neural network. The algorithm that was used for training is the back propagation algorithm [16].4. Experimental ResultsThe performance of the Neural Network decision module was evaluated through the calculation of the average performance from 100 decisions that were made. The outcome was 0.76, which is quite good since the average performance of the data that were used in order to train the network was 0.8. We also evaluated the previous analytical (hand coded) decision module and a random decision module using the evaluation function E(x,t). It is to compare them with the Neural Network decision module. The average result that we got for the hand coded decision module (from 100 decision that were made) was 0.64 and the average for the random one (from 100 decision that were made) was 0.47 (figure 5). In the random decision module, the goalie keeps the ball for 100 cycles and then chooses randomly one of the 10 possible GSPs.As it was mentioned in section 3.3 the hidden layer consists of 10 neurons. In the experiments, we tried many other numbers of hidden neurons such as 20, 30, 50, 100, and 200. However there was nosignificant change in the performance of the neuralnetwork. Therefore, 10 hidden neurons were chosen since they make the training process of the neural network faster.5 Conclusions and Future WorkThis paper describes how neural networks can be adopted in the Adaptive Strategic Planning Framework that has been developed for the Essex Wizards team to participate in the RoboCup competitions. Integrating a neural network decision module to the ASPF has proved the openness of the framework that has been claimed before. Experimental results shows that a decision module based on neural networks can outperform the hand coded decision module that has been used in the past. During a game the ASPF should make decisions in many other strategic situations like “goal kick” and “crossing”.In the future we will implement the neural network decision technique to all the strategic situations. For each strategic situation a different neural network will make the decisions. References[1] N. Kalyviotis and H. Hu, ASPF: an AdaptiveStrategic Planning Framework in RoboCup,Proceedings IASTED International Conference on Robotics and Applications (RA 2001), Florida, November 2001.[2] H. Hu, K. Kostiadis, M. Hunter and N.Kalyviotis, Essex Wizards 2001 Team Description,In Birk A., Coradeschi S., and Tadokoro S., editors, RoboCup-01: Robot Soccer World Cup V. Springer Verlag, Berlin, 2002.[3] M. Hunter, K. Kostiadis and H. Hu. ABehaviour-based Approach to Position Selection for Simulated Soccer Agents, Proc. of the 1st European RoboCup Workshop, 28 May-2 June 2000.[4] H. Kitano, RoboCup: The Robot World CupInitiative, Proceedings of The First International Conference on Autonomous Agent (Agents-97)), Marina del Ray, The ACM Press, 1997.[5] K. Kostiadis and H. Hu, ReinforcementLearning and Co-operation in a Simulated Multi-agent System, Proc. of IEEE/RSJ Int.Conf. on Intelligent Robots and Systems, pages 990-995, Kyungiu, Korea, 17-21 October 1999.[6] H. Kitano, M. Tambe, P. Stone, M. Veloso, S.Coradeschi, E. Osawa, H. Matsubara, I. Noda, and M. Asada,The RoboCup Synthetic Agent Challenge, 97, Int. Joint Conf. on Artificial Intelligence, 1997.[7] P. Stone, Layered Learning in Multi-AgentSystems, PhD Thesis, Carnegie Mellon University, 1998.[8] R. Wilensky, Planning and Understanding,Addison-Wesley Publishing Company, Massachusetts, 1983[9] D. Wilkins, Practical Planning: Extending theClassical AI Planning Paradigm, Morgan Kaufmann Publishers, California, 1988. [10] S. Haykin, Neural Networks a comprehensivefoundation, Macmillan, USA, 1994.[11] P.K Simpson, Foundations of Neural Networks,San Diego, 1992.[12] I. Aleksander and H. Morton, An Introductionto Neural computing, Chapman & Hall, London, 1990.[13] N.A Bridgett, C.J Harris, Artificial neuralnetworks for use in the diagnosis and treatment of breast cancer, UK, 1995.[14] C. Diamond, J. Shadbolt, M. Azema-Barac andA.N Refenes, Neural network system fortactical asset allocation in the global bond markets, Econostat Ltd, UK, 1993.[15] H.E Rauch and T. Winarske, Neural Networkfor Routing Communication Traffic, 1988. [16] S.I Gallant, Neural Network Learning andexpert systems, MIT Press, USA, 1993 [17] N. Kalyviotis and H. Hu, A Co-operativeFramework for Strategic Planning, Proceedings Towards Intelligent Mobile Robots (TIMR) ’01, Manchester, 2001.。