Bayesian map learning in dynamic environments

Kevin P.MurphyComputer Science DivisionUniversity of CaliforniaBerkeley,CA94720-1776murphyk@AbstractWe show how map learning can be formulated as inference in a graphicalmodel,which allows us to handle changing environments in a naturalmanner.We describe several different approximation schemes for theproblem,and illustrate some results on a simulated grid-world with doorsthat can open and close.We close by briefly discussing how to learn moregeneral models of(partially observed)environments,which can containa variable number of objects with changing internal state.1IntroductionMobile robots need to navigate in dynamic environments:on a short time scale,obstacles, such as people,can appear and disappear,and on longer time scales,structural changes, such as doors opening and closing,can occur.In this paper,we consider how to create models of dynamic environments.In particular,we are interested in modeling the location of objects,which we can represent using a map.This enables the robot to perform path planning,etc.We propose a Bayesian approach in which we view the map as a(matrix-valued)random variable,which can be updated by Bayes rule in just the same way that the other hidden state variables,such as the robot’s pose(position and orientation),are updated in the widely-used techniques of Markov localization[FBT98]and Kalmanfiltering.To do Bayesian updating,we must specify the observation model(how the state predicts what the robot should see)and the transition model(how the state changes,both over time and in response to the robot’s actions).We use the formalism of factored POMDPs (Partially Observable Markov Decision Process)[CKK96,KLC98,BDH99];see Section2 for details.We must also provide a mechanism for efficiently performing the update equations;we discuss various approximation algorithms for this in Section3.In Section4, we show some results of applying our algorithm to a simulated dynamic environment,and in Section5we conclude and discuss future work.2The modelThe three main components of a POMDP are the transition model,the observation model, and the reward model(which induces a policy).In this section,we focus on the transitionmapa b cFigure 1:(a)A POMDP depicted as a graphical model.The dotted nodes (,,and )represent the fixed parameters.The square nodes (12)are the decision variables (actions).The clear nodes,(12)represent the hidden state,such as the robot’slocation.The shaded nodes (12)are the observations.(b)A POMDP in which the map is explicitely represented as a set of random variables.All the map nodes connect to and ,as represented by the thick arrow.(c)A model in which the dynamic state of the world,such as whether doors 1and 2are open or closed,is stored separately from the static map.and observation models,treating the sequence of actions as observed inputs;in this paper,we use a simple exploration policy to choose these actions,as described in Section 4.If the “external”world is considered static (for modeling purposes),we can treat it as a fixed (but unknown)parameter ;in this case,the state usually just consists of the robot’s position and orientation (see Figure 1(a)).If the state variables are discrete-valued,we can perform Bayesian updating using a Hidden Markov Model (HMM);otherwise we can use (extended)Kalman filtering,or non-parametric sample-based filtering (see e.g.,[KF98,TLF99]).For example,let us consider a simple HMM model in which the state consists of the robot’s location ,which is represented as a point on a discrete grid,and the actions consist of moving one step in each of the four compass directions.The observation model specifies the probability of each possible observation vector as a function of location.In this paper,we assume an observation is one of symbols,although it is simple to extend the technique to handle observations from I R .With a discrete set of observations,the observation modelcan be represented as a stochastic matrix,.Similarly,the transition model,which specifies the probability of moving from one grid cell to another as a function of the action taken,can be represented as a set of stochastic matrices:1.The parameters and implicitely encode the map;hence estimating these parameters using EM is equivalent to estimating the map [KS96,SK98,TBF98].If the external world can change over time,we must represent it as a random variable,instead of a fixed parameter.We then augment our (hidden)state to include the randomvariables 1,where means location contains object at time :see Figure 1(b).1To parameterize the model,we need to specify a transition matrix forpositionFigure2:A two-dimensional sensor model in which the robot receives observations from a33neighborhood.The double-ringed nodes are deterministic multiplexer nodes,and connect in1-to-1fashion to the observations.the’s.For example,if an open door can become a closed door with probability change, we can write1change.(We discuss a more parsimonious representation of objects with internal state,such as doors,in Section5.)If the robot can change the external environment(e.g.,it can open and close doors,or move obstacles),then 1will also depend on.We also need to specify how affects the observations.The simplest way to do this is to create an indexical representation of space,i.e.,in robot-centered coordinates.If the robot knew its location,it could simply extract the appropriate part of to build a model of its immediate surroundings,which we call a local map.We can implement this in a graphical model by using deterministic multiplexer nodes,with the location acting as the selector:this is illustrated in Figure2.The nodes in the local map are defined such that ∆∆∆∆,for∆∆101. Each entry in the local map then gets passed through a stochastic link to produce the actual observation.The local map can also be used to predict whether actions will succeed or not, since the probability of moving to a neighboring cell will depend on whether it contains an obstacle,and the probability of actuator failure.3InferenceNaive application of an exact inference algorithm,such as junction tree[Jen96],to the graphical model in Figure1(b)would be wildly intractable,because all the nodes in the map would become correlated by virtue of having a common child.Intuitively,the correlation arises whenever the robot is uncertain of its location,as we can explain by means of a simple example.Suppose at time1the robot sees some object,but it does not know if it is in cell or.Hence we need to increase and,but record the fact that these values are not independent.The posterior on now has(at least)two modes, corresponding to the possible worlds in which contains or contains.If,at time2,the robotfigures out its location is1(e.g.,because it recognizes a landmark),then it can infer that at time1its location was(with high probability),and hence should go up—but should go down to compensate,since they are correlated.The Bayesian approach will handle this automatically:one of the two modes in the posteriorwill“collapse”once the robot knows where it is.Unfortunately,an exact posterior over would be of size,so we need tofind a more efficient representation.If the robot always knows where it is,there will be no correlation amongst the’s,because of the deterministic nature of the local map(see Section3.1). Hence we can represent the posterior in factored form,11, which takes space.(If we had perfect sensors,we would know what object was present in each cell,and could represent the map in just space.)In practice,the robot will not know its exact location,but,as an approximation,we can still project the posterior down onto the factored representation at each step.In[BK98b],they show that the error introduced by this kind of iterated approximation does not grow with time.If we assume that the map changes slowly,we can treat is as a constant,at least over a short window of time,,and“compile”it into the parameters of the local map. We can then alternate between estimating the locations in this window given the map,and updating the map given the estimated locations.(This is just an online version of the standard EM approach cf.[BK98a].)The main advantage is that locations early in the window get to look at“future”evidence,which can help to disambiguate them(consider the example of moving down a corridor with known landmarks at each end),and this can lead to less uncertainty about which part of the map to update.However,the(standard) assumption of parameter independence means this approach cannot handle correlations. Some alternative approximations would be to use sample-basedfiltering algorithms(see e.g.,[KF98,TLF99])or the variational approximation of[GJ97].3.1Exact inference within a sliceWe will now discuss how to do inference in the two-dimensional graphical model in Figure2.Given the assumption that the’s are independent,we only need to compute the marginals on each separately,which can be done exactly and efficiently.The basic idea is that we will consider each location one at a time,and update the part of the map around with the information from the sensors,weighted by the probability that is the correct location.The conditional likelihood of the evidence from the sensors in the33neighborhood centered on is given by a product of factors:Hence∆∆101∆∆∆∆(1)∆∆101each term being just a diagonal matrix times a vector,which takes time to compute.So we can update the posterior on in time as follows:,where is a normalizing constant.Once we have the weighting matrix,we can update the map in2time usingwhere.The terms can be computed in constant time by exploiting the fact that,when and are known,only a33submatrix of the nodes get“connected”to the evidence(via the local map),the other nodes being effectively “disconnected”from the graph.Formally,we haveif101otherwisewhere the terms,etc.have already been computed in Equation1.If we only condition on the most probable values of,Figure3:A simple environment used for the experiments.Shaded blocks represent walls,R represents the starting location of the robot;to its north is an open door and to its north-east is a closed door.Figure4:A sample run of the algorithm for the environment in Figure3. instead of considering all possible values,we can update the map in time;this approximation is called bounded conditioning.4ResultsWe illustrate the behavior of the algorithm when the robot is placed in the simple envi-ronment shown in Figure3.We used change01and assumed sensors and actuators could fail with probability0.1.The policy was to always visit the nearest uncertain cell cf. [YSA99].In Figure4,we show the belief state of the robot at each time step.The letters in the center of each cell is the robot’s best guess as to what is there(W=wall,O=open door, C=closed door,F=free space,D=open or closed door,?=very uncertain).The true location of the robot is indicated by H,and the robot’s best guess of its location is indicated by h. The observations from a33grid of sensors centered on the true location are shown above each cell;an asterisk denotes an erroneous observation,i.e.,a label which is different from the true contents of the cell.The title indicates the goal the robot is heading towards,using (row,column)notation(so(1,1)represents the top left corner),and the action it intends to take.We now discuss some of the interesting steps in this example.At step1,the robot starts in(4,2),and knows this fact for sure.The two nearest,accessible,uncertain cells are(2,2) and(4,4);it arbitrarily decides to head towards thefirst.Because of the33sensing neighborhood,it is sufficient to reach(3,2),so this becomes the goal.It then moves to the upper corridor.At step5,the robot receives several observations that are inconsistent with its prior beliefs.Consequently,it gets“confused”,and decides to stay put(action=“-”). At step6,the robot’s uncertainty about the status of the door at(3,2)exceeds threshold, as shown by the fact that the’O’becomes a’D’,which represents the fact that the robot knows the object is a door,but is not sure whether it is open or closed.This increase in itsuncertainty about door objects(but not walls,etc.)happens because the transition matrix for the nodes specifies that open doors can become closed doors and vice versa.The robot will revisit the door,to check its status,after it has examined all the closer cells.After succesfully learning a map of the environment,the robot starts running from door to door, trying to keep its map up-to-date.In experiments on other,similar,environments,we have found that the robot always learns the correct map unless the noise levels become too large.In this case,the robot can get “lost”,and it starts updating the wrong part of the map.If the robot wanders into an area with known distinguishing characteristics,it can relocalize itself,and hence repair the damage to the map.5Discussion and future workThe problem of modeling dynamic environments has been open for quite a while.In 1996,Cassandra et al.[CKK96]wrote:“One of the big shortcomings of modeling the environment with POMDP models is that there is too much dependence on the world being static.”In1998,Thrun[Thr98]wrote:“It is an open question as to how to incorporate models of moving objects into a grid-based representation.”In this paper,we have shown how to model dynamic environments using graphical models.(The only previous work we know that has addressed the problem of learning maps in dynamic environments is[YB96].) However,our representation still leaves much to be desired.In particular,suppose we are interested in representing the location of individual objects,each with their own internal state.If there are such objects,each of which can be in states,the current approach would require each to have possible values,as we saw in our representation of doors(which could be open or closed).In addition,the transition matrix for the’s would be hard to specify.A more parsimonious representation would use the map to store (a distribution over)the identity(or type)of the object in each cell,and would store the state of each object in separate random variables.(Of course,uncertainty about object identity would induce correlation amongst all the state variables,just as uncertainty about location did in our previous representation.)A simple example of an object-centered representation is shown in Figure1(c),in which we store the state of each door(open or closed)in the nodes;now can take on values in12.The transition matrix for the’s becomes the identity matrix(since we assume an object in a given location doesn’t move or change identity),and the transition matrix for the’s has change on the diagonal and1change off the diagonal.As another example,consider the task of following a person through a maze-like environment;since they may disappear from view(e.g.,around a corner),it is useful to have a model of their current position and heading.The transition model for the person will be similar to that of the robot’s(since they share the same physical environment),but will be based on a local map centered on the person’s location.The example above assumed that we knew there were two doors,and further,that we knew 1corresponded to the one in location(3,2)and2to the other one(or vice versa).When an agent is placed in an unknown environment,it will need to decide whether its current observations are due to an existing object,and if so,which one,or whether it needs to create a new object.(This problem arises even in the simpler context of learning maps of unknown size—afixed graphical model is clearly inadequate.)The situation is very similar to the problem of data association and tracking with multiple targets[BSF88].The standard approach there is to model each object’s state(usually its position and velocity) with a Gaussian;when a measurement is observed,it is assumed to come from the nearest object;if,however,the observation does not lie inside’s1-Σconfidence ellipsoid,a new object is created.Note that this hard-thresholding approach eliminates the need to modelthe correlation between the different objects.If a new object is needed,the agent will have to decide what kind of object it is,too;if it does not have a pre-existing ontology,it will have to invent categories for itself,perhaps by clustering percepts.We hope to work on these problems in the future,to better understand how to build an embedded Bayesian agent. AcknowledgementsI would like to thank Stuart Russell for many useful discussions.This work was supported by grant number ONR N00014-97-1-0941.References[BDH99] C.Boutilier,T.Dean,and S.Hanks.Decision theoretic planning:Structural assumptions and computational leverage.J.of AI Research,1999.[BK98a]X.Boyen and D.Koller.Approximate learning of dynamic models.In Advances in Neural Info.Proc.Systems,1998.[BK98b]X.Boyen and D.Koller.Tractable inference for complex stochastic processes.In Proc.of the Conf.on Uncertainty in AI,1998.[BSF88]Y.Bar-Shalom and T.Fortmann.Tracking and data association.Academic Press,1988. [CKK96] A.Cassandra,L.P.Kaelbling,and J.Kurien.Acting under uncertainty:Discrete Bayesian models for mobile-robot navigation.In IEEE Intl.Conf.on Intel.Robotics and Systems,1996.[FBT98] D.Fox,W.Burgard,and S.Thrun.Active Markov localization for mobile robots.Robotics and Autonomous Systems,1998.[GJ97]Z.Ghahramani and M.Jordan.Factorial Hidden Markov Models.Machine Learning, 29:245–273,1997.[Jen96] F.V.Jensen.An Introduction to Bayesian Networks.UCL Press,London,England,1996. [KF98] D.Koller and ing learning for approximation in stochastic processes.In Intl.Conf.on Machine Learning,1998.[KLC98]L.P.Kaelbling,M.Littman,and A.Cassandra.Planning and acting in partially observable stochastic domains.Artificial Intelligence,101,1998.[KS96]S.Koenig and R.Simmons.Unsupervised learning of probabilistic models for robot navigation.In IEEE Intl.Conf.on Robotics and Automation,1996.[SK98]H.Shatkay and L.P.Kaelbling.Heading in the right direction.In Intl.Conf.on Machine Learning,1998.[TBF98]S.Thrun,W.Burgard,and D.Fox.A probabilistic approach to concurrent mapping and localization for mobile robots.Machine Learning,31:29–53,1998.[Thr98]S.Thrun.Learning metric-topological maps for indoor mobile robot navigation.Artificial Intelligence,99(1):21–71,1998.[TLF99]S.Thrun,ngord,and D.Fox.Sample-based hidden Markov models.In Intl.Conf.on Machine Learning,1999.[YB96] B.Yamauchi and R.Beer.Spatial learning for navigation in dynamic environments.IEEE Trans.Systems,Man and Cybernetics B,26(3):496–505,1996.[YSA99] B.Yamauchi,A.Schultz,and W.Adams.Integrating exploration and localization for mobile robots.Adaptive Behavior,1999.。