Under Guidance Of

ROBOCUP SOCCER SIMULATIONReinforcement Learning – Single Shooter & GoalkeeperBy----------------------------------Ajit Kumar Akhilesh Kumar Sinha Department of Comp. Sc. & Engg Department of Comp. Sc. & Engg ***************.in***************.inUnder Guidance OfProf. Dr. Amitabha Mukherjee, Prof. Dr. R. M. K. Sinha, Mr. Mrityunjay GautamPeriod: March – April 2006.----------------------------------------------------------------------------------------------------This paper is our Project Report for CS365: Artificial Language Programming Course taken in Jan-April 2006.Project scope and Objectives:RoboCup Soccer Simulators are available on the internet. But all these codes are available only in C/C++ and built on compilers no longer publicly available. So we started with building the Soccer Server architecture in Java. Java has neat backward compatibility and hence Servers and Clients built on Java platform will be easier to use and moreover it also becomes platform independent.Central Server and the Client programs were written in a fashion that it can be easily extended to multi-player domain with inclusion of many more basics relevant to the game, like dribbling, passing and many others.Using Reinforcement Learning to learn the basics of the game.Make a shooter and a keeper learn in the same environment, by reward maximizing Reinforcement Learning, both having opposing goals.Compare and contrast the two learning rates and also compare the learned behavior in agents that have learned to different levels.With additional learning phases for dribbling and passing, it could be easily extended to the m-n keepaway games.ROBOCUP SOCCER SIMULATIONReinforcement Learning – Single Shooter & GoalkeeperAbstractIn the Markov Decision Process (MDP) formalization of Reinforcement Learning, a single adaptive agent interacts with the environment defined by a probabilistic transition function. Secondary agents can only be a part of the environment and are therefore fixed in their behavior. The framework of Markov games allows us to widen this view to include multiple adaptive agents with interacting or competing goals. This paper considers a step in this direction in which exactly two agents with diametrically opposed goals share an environment. It describes reward maximizing Q-Learning-like algorithm for finding optimal policies and demonstrates its application to a relatively complex two-player game in which the optimal policy is probabilistic. Learning of the two agents at different learned-level is also compared in a systematic way to further emphasize on the learning behavior of the agents at various stages of their Learning Life.1.IntroductionA soccer game is a specific but very attractive real time multi-agent environment from the viewpoint of distributed artificial intelligence and multi-agent research. Learning in this dynamically changing real world of soccer environment is to be done with Reinforcement Learning and Layered Learning.Here, we apply reward based Reinforcement Learning to a two agent, a Goal Keeper and one Shooter, with opposing goals in a single environment. Both agents learn simultaneously during the learning phase, when they play against each other and take random actions, based on these pair of random actions of both the players a reward or penalty, depending on the result of the action-reaction pair, is awarded to each of the agent. This reward helps to modify the learning matrix of each of the players.Various matrices each for different level of learning for both the agents, the Goal Keeper and the Shooter are used to compare and contrast the learning behavior of the two agents and the rate of learning in a single agent.Motivation:Building a Soccer Team for RoboCup Simulation League has already been done by many research groups. This involves virtual soccer players competing in a dynamic, complex, uncertain multi-agent environment. Major challenges involved in making the team more human like and better strategy planner are•The environment is highly dynamic.•The perception of each player is locally limited.•The role of each player can be different.•Communication among players is limited, therefore each agent is required to.behave very flexibly and autonomously.A soccer game is a specific but very attractive real time multi-agent environment from the viewpoint of distributed artificial intelligence and multi-agent research. Several scientific areas of interest which we will address in our project are the following: •Adjustable Autonomy. How do players decide on interpreting and acting on a coach's advice or supplied information?•Multi-Agent Modeling. How can a coach recognize its own team's and an opponent's strength's and weaknesses? How can it model their behavior?•Teamwork and Coordination. How can a group of agents work together, collaborating and coordinating, as an effective team? How do dynamic small sub-teams form, and how can they be made effective?•Adversarial Planning. How can agents plan to counter opponents' behavior? Currently many research groups across the glove, in many technological universities and laboratories are working on a similar domain. Several of them have also released their codes (Soccer Simulators are available at /projects/sserver/).2. DefinitionsReinforcement Learning algorithm works by letting the agents learn autonomously as to what is good and bad. This involves learning about and from the environment how to map situations to actions in such a way as to maximize the reward associated with the possible actions. The reward function is heuristically chosen which is in close accordance with actual goal. The whole learning is goal-oriented.Some very nice application of RL are:1. TD-Gammon: Tesauro, world’s best backgammon program.2. Elevator Control: Crites and Barto, high performance elevator controller.3. Dynamic Channel Assignment: Singh and Bertsekas, Nie and Haykin, high performance assignment of radio channels to mobile telephone calls.4. Traffic Light Control: Bhattacharya et al., high performance control of water level of regional water systems.Complete Agent is an agent that is temporally situated and continually learning andplanning, it affects the environment. Environment in our case is stochastic, uncertain and probabilistic. Agent’s action brings a change in the environment.Key advantages in having such a complete agent is that it is autonomous. Learner is not told which actions to take. Initial learning comes from trial and error search and theaction-reward pair in those trials. It considers the whole problem of a goal directed agent in an uncertain world. Long term and short term goals can also be given preferences depending on the game and goal selection. It also covers the possibility of exploiting and exploring during the randomized trial and error learning sessions.Advantages of using Reinforcement Learning in this project are that it is “online”, “autonomous” learning without a need for preprogrammed behavior or instruction. It can be used to learning to satisfy long-term goals and is applicable to many tasks.3.Theory And Application In Soccer Simulation(Single Shooter and Goalkeeper)In this section we describe how RL theory is finally applied to real world problems, how we define the proper states and the action. All these will done here in reference to the Soccer Simulation (Single Shooter and Goal Keeper) case.Elements of Reinforcement Learning:Policy: What to do?Here the primary focus is do properly define and formulate the map states and the possible action states.An MDP [Howard, 1960] is defined by a set of states S, and actions, A. A transition function, T : S × A PD(S), defines the effects of the various actions on the state of the environment. (PD(S) represents the set of discrete probability distributions over the set S) The reward function, R : S × A R, specifies the agents task.In broad terms, the agent’s objective is to find a policy mapping its interaction history to a current choice of action so as to maximize the expected sum of discounted reward,E{∑j=0∞ Y j . r t+j } where r t+j is the reward received j steps into the future. A discount factor, 0 < Y < 1 controls how much effect future rewards have on the optimal decisions, with small values of Y emphasizing near-term gain and larger values giving significant weight to later rewards.In the present paper and project work, the work is concentrated around two agents, the Shooter and the Goal Keeper. Lets name then Agent 1 and Agent 2 respectively for easier reference subsequently.Agent 1 has the position of the ball, the position of the crossbar (dimensions and spread of the crossbar) as the state of the environment. For the Agent 1 the environment is static till it takes the kick action, which changes the environment for both Agent 1 and Agent 2 and this change serves as the stimulus for the Agent 2 to take its action of trying to save the goal. During play Agent 1 chooses an action probabilisticly.Fig 3.1 Actions of the Agent 1 includes all the possible kicks at the various angles and height.Agent 2 shares the same environment as the Agent 1, but its action depends on the changed environment after the shooter kicks the ball towards the crossbars.To make the situation more complex and real like various variables like power into the shot, wind velocity and direction and gravity effects etc. can also be made a part of the environment.RewardIn this paper, we consider a well-studied specialization in which there are only two agents and they have diametrically opposed goals. As the game played just involves a single action by both the Agents and the reward is decided only on these two actions. Both these agents have different reward policy and each one of them tries to maximize its own rewards and further use it optimally to choose an action.Reinforcement Learning is primarily done by the rewarded editing and updating of the state action maps for both the Agents. After a single play of kick and save action of the Agent 1 and Agent 2 respectively, during learning phase, each of the state-action map is modified depending on the outcome of the last single play.Agent 1’s state-action map gets modified for action a if after the single play of Agent 1 and Agent 2, the final outcome comes to be a goal (success for Agent 1 and failure for Agent 2). Similarly Agent 2’s state-action map gets modified at the final outcome of not being a goal. As the game in our project just had a single play from both the Agents and each of these such games are completely independent of the earlier games hence r j+t is always zero. There is no long term goal for Agents in this model, hence Y = 0, for this Single Shooter and Goal Keeper model.ValueValue function is irrelevant in the approach we’ve taken, as the game is only action each and result is a well defined state of a goal or not a goal.ModelIn this model the actions by the two agents are taken sequentially. Agent 1 waits till it gets the start of game message from the central soccer server. It chooses an optimal action (kick) and this information is relayed through the soccer server to the Agent 2, which thereafter takes an optimal action for this changed state of the environment.4.Soccer Server and Client architectureAll the code is written in Java. The architecture is kept very close to that of the RoboCup Soccer Simulator available for learning and research purposes. The organization of the central server and the client programs have exactly similar communication policy and hence this can be easily expanded to include many other features of the game with the same basic architecture. This can also be used the official version of the RCSS server. One major difference is that there is no scope for communication strategy building among team players as this model incorporates only two opposing Agents. This is one field in which the maximum research is going on in many prestigious universities around the globe.Fig 4.1 Current architecture of the Soccer Server Client modelCentral ServerCentral server is concerned with setting up the environment and gives the information of all the states to the clients. Clients individually take actions that change the environment which is then again relayed to both the clients to take further action depending on the new states. This information is in form of a string that is interpreted by the clients to gather information about the states of the environment and the result of their actions. Information about the start of the play session or the learning session is passed by the Server to both the clients and thereafter the clients react depending on the phase they play is currently governed by the Server.Shooter Client 1Client 1 is the program that first gets connected to the Central Server. This behaves as the Shooter. During Learning phase it takes random decisions and relays it to the Server. Depending upon the result it upgrades its learning Matrix.During playing phase it uses this same learned Matrix to choose an optimal action. Parameters: it deals with these self explanatory variables.Start_of_learn: Start_of_play :Ball_position :Kick_height :Kick_direction : reward : result.Goal Keeper Client 2Client 2 takes no action till the changed environment due to Client 1’s action is communicated to it by the Central Server. Strategy for learning and playing is similar as in Client 1.Parameters it deals with:Start_of_learn : Start_of_play : kick_direction : kick_height : save_angle : save_height.5.Training and TestingThe random number that has been used during the trial and error learning phase shows truly random behavior and at the same time for large iterations it covers all the possibilities nearly uniformly.Fig 5.1 Uniformity in random number generator.Two tables of number of times one of the values was randomly taken.0 is 1002640is 3002521 is 1003561is 3008222 is 995892is 2995363 is 999633is 3002724 is 1005314is 3003855 is 1000025is 2996786 is 996716is 2997737 is 999107is 2994198 is 999518is 3000389 is 997629is 299823For 1000000 iterations For 3000000 iterationsFor very large iterations, the whole range of numbers were nearly uniformly chosen.Fig 5.2 Each range of values were uniformly chosen.Using this Random number generator initially the Shooter and the Goal Keeper are trained, using rewards when they favorable action and no change otherwise.The agents play the game according to the learned matrix they obtain after the learning phase. Different combinations of initial value and reward strategy leads to different levels of understanding and learning of individual games and opponents moves.Below we have taken various combinations of learning time, initial value functions and rewards to compare both agents learning rate and efficiency.Experiments and ObservationsTrial and Error Learning;Number of Iterations = 1000Reward = 4 for each goal to Shooter and for each save to Keeper.Initial value for goalie = 10, for Shooter = 500Fig 5.3 Number of goals for every 100 games.In this case we observe that level of learning is nearly same for both the GoalKeeper and the Shooter.Trial and Error Learning;Number of Iterations = 100000Reward = 4 for each goal to Shooter and for each save to Keeper till 1000th iteration, reward = 2 after that.Initial value for goalie = 10, for Shooter = 500Fig 5.4 Number of goals for every 100 games.We can see that in this case the Goalkeeper learns to much higher extent.Trial and Error Learning;Number of Iterations = 1000000Reward = 4 for each goal to Shooter and for each save to Keeper till 1000th iteration, reward = 2 after that till 100000 and reward = 1 after that.Initial value for goalie = 10, for Shooter = 500Fig 5.5 Number of goals for every 100 games.It is very evident that initially the Shooters learning rate is higher than Goalies but after a large number of iterations the goalie becomes very efficient and learns to stop most of the goals.Now we compare the agents with different level of learning experience.Trial and Error Learning;Number of Iterations = 1000 for Shooter, and Iterations = 100000 for Keeper. Reward = 4 for each goal to Shooter and for each save to Keeper till 1000th iteration, reward = 2 after that till 100000 and reward = 1 after that.Fig 5.6 Number of goals for every 100 games.The Goalkeeper is able to save most of the goals as it has learnt for more time than the Shooter.Trial and Error Learning;Number of Iterations = 100000 for Shooter, and Iterations = 1000 for Keeper. Reward = 4 for each goal to Shooter and for each save to Keeper till 1000th iteration, reward = 2 after that till 100000 and reward = 1 after that.Fig 5.6 Number of goals for every 100 games.The Shooter learns here over more iteration than the Keeper as a result it is able to convert most of the actions into goals.Hence learning over more iterations is beneficial for both the agents. Initial value function is also important to choose initially properly because not all actions have equal probability of converting into goals.6.Conclusion and Future WorkWe believe our present work shows that reinforcement learning can scale to complex domains such as simulated soccer with a minimal interjection of domain knowledge on the hand of the designers. However, we feel that more work is needed to make reinforcement learning truly robust in such domain.Soccer Server Architecture does not have the Coach and any graphical representation of the view of the online game being played. We plan to build a graphical interface to this online game, similar to the LogPlayer in RCSS simulator. Adding few other basics like accepting passes and dribbling will make the whole model much more realistic and many strategy building and co-operation work and research work could then be done. Building the whole SoccreServer and Clients on Java platform seems a very good idea for general research and algorithms testing as it is more robust and platform independent. There is a disadvantage of speed though in programming in Java. But it much easier to change and upgrade the servers because of the OOP in Java.Next step would be to build a keepaway play model.AcknowledgementsWe would like to thank Mr. Mrityunjay Gautum, Mr. Utsav Maitra and Mr. Gagandeep Pal for their help whenever we wanted some.Reference:"Co-Evolving Soccer Softbot Team Coordination with Genetic Programming,"[Farris et al, 97] J. Farris, S. Luke, G. Jackson, C. Holn, J. Hendler, InProceedings First International Workshop on RoboCup, IJCAI-97, W17, pp. 115-118, Nagoya Congress Centre, Nagoya, Japan, 1997.Multi-agent reinforcement learning: independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, Amherst, Massachusetts. Morgan Kaufmann.[Tan, 1993] Tan, M. 1993.An evolutionary approach for strategy learning in RoboCup soccer Nakashima, T.; Takatani, M.; Udo, M.; Ishibuchi, H.;Systems, Man and Cybernetics, 2004 IEEE International Conference on Volume 2, 10-13 Oct. 2004 Page(s):2023 - 2028 vol.2“Markov games as a framework for multi-agent reinforcement learning” Michael L. Littman, Brown University, Department of Comp. Sc.Brown University / Bellcore, 1994.Strategy Learning For A Team In Adversary EnvironmentsYasutake Takahashi, Takashi Tamura and Minoru AsadaDep. Of Adaptive Machine Systems, Osaka UniversityAn Evolutionary Approach for Strategy Learning in RoboCup Soccer Tomoharu Nakashima, Masahiro Takatani, Masayo Udo, Hisao Ishibuchi College of Engineering, Osaka Prefecture University2004 IEEE International Conference on Systems, Man and Cybernetics Reinforcement Learning for RoboCup SoccerPeter Stone, Richard S. Sutton, Gregory KuhlmannDep. Of Comp. Sc. University of Texas at Austin, Univ. of AlbertaLayered Learning for Evolving Goal Scoring Behaviour in Soccer Players Andrei Bajurnow and Vic Cieielski of RMIT University, Australia Proceedings of the 2004 Congress on Evolutionary Computation(CEC2004) Pages 1828-1835, Portland, June 2004, IEEE PressEvolving Goalkeeper Behaviours For Simulated Soccer Competition Chistopher Lazarus and Huosheng Hu, Univ. of Essex, Wivenhoe Park Proceedings of the 3rd IASTED International Conference on Artificial Intelligence and Applications (AIA2003) September 8-10, 2003, Benalmadena, Spain Extending Graphplan to Handle Uncertainity & Sensing ActionsDaniel S. Weld, Corin R. Anderson and David E. SmithAppeared in AAAI-98Websites:/http://www.cse.iitk.ac.in/users/pmitra/cs365/robocup.html/noda97soccer.htmlhttp://www.science.uva.nl/research/ias/research/multi_agents/robocupsim/index_e n.html/projects/sserver/http://www.ifi.unizh.ch/ailab/people/nitschke/RoboCup.htmlhttp://www.csl.sony.co.jp/person/kitano/RoboCup/RoboCup-old.html/overview/23.html。