Human-oriented tracking for human-robot interaction

Human-Oriented Tracking for Human-Robot Interaction Christian Wengert1,Terrence Fong2, Sébastien Grange1, and Charles Baur11Swiss Federal Institute of Technology (EPFL), Lausanne, Switzerland{christian.wengert|sebastien.grange|charles.baur}@epfl.ch2The Robotics Institute, Carnegie Mellon University, terry@AbstractIn order for human-robot interaction to be effective in real-world environments,it needs to be natural and trans-parent.To achieve this,we need to facilitate direct,proxi-mal communication between humans and robots.Our approach is to use computer vision to enable robots to observe and react to human activity.In this paper,we describe the development of real-time,computationally effi-cient vision modules for human tracking,human identifica-tion, and static gesture recognition.1.IntroductionRobots continue to play an ever increasing role in our world,working for and in cooperation with humans. Beyond the simple industrial tasks performed in the past, robots now assist in a wide range of activities(health care, office work,etc.),often in immediate proximity to humans. Central to the success of these applications is close and effective interaction between humans and robots.Thus, although it is important to continue enhancing autonomous capabilities,we must not neglect improving the human-robot relationship.In particular,we need to develop tech-niques that allow humans and robots to interact naturally, effectively, and transparently.Our objective is to develop computer vision methods that facilitate proximal human-robot interaction.Specifically, we want to enable humans to communicate directly with robots,through movement and gesture,unencumbered by a user interface.At the same time,we want to provide robots with the capability to observe and react to human activity, so that they can respond quickly and in an appropriate way. In the following sections,we present color and stereo vision techniques for human tracking,human identification,and static gesture recognition.We have designed these tech-niques to be computationally efficient,so that they can oper-ate in real-time with low-latency on low-power,low-cost processing hardware.2.Related WorkA great deal of work has been performed in the field of human tracking,particularly for video-based surveillance [1][9].Many methods rely on static cameras to permit back-ground subtraction or template matching.Our approach is to use a combination of camera geometry,stereo vision,and color indexing to detect and localize humans.Gesture recognition has widely been studied as a means of computer input and has also been used for robot con-trol[6][10][11].In previous work,we developed a remote driving interface based on visual gesturing[2].In that sys-tem,we used colorfiltering and stereo to track and classify hand motions for remote driving.The work described in this paper is similar,except that we only recognize static ges-tures in order to reduce processing requirements.Some researchers have recently begun investigating vision-based human-robot interaction,particularly with regard to detecting human head pose and recognizing repeating motions(e.g.,cyclical hand gestures)[3][4].The difference between our work and these other systems is that we com-bine color and stereo vision to achieve better tracking and identification.3.Approach3.1ArchitectureOur system architecture is shown in Figure1.We capture sub-sampled images from a pair of color CCD cameras and then perform color processing(normalized color filtering and histogramming)and area-based stereo correlation(to compute a disparity map).This data is then used by the three primary system modules: HumanTracker,HumanID and GestureDetector,which per-form human tracking,color-based identification,and static gesture recognition.Collectively,these modules enable a single user to interact with a mobile robot through move-ment and hand gestures.Human Tracking.The HumanTracker tracks humans using stereo vision.Although other researchers have emphasized skin-colorfiltering,we prefer stereo because it is largely invariant to changes in color,background and illu-mination.However,we also employ color-based identifica-tion to guarantee that only one person is tracked at a time. Human Identification.In our research,we have chosen to focus on1:1human-robot interaction.Thus,when there are multiple people in the scene,we need to distinguish between them.The HumanID module does this with color indexing.The method works by computing a color histo-gram for the image region that contains the greatest separa-tion between the detected human and the background. Gesture Recognition.The GestureDetector recognizes a pre-defined set of static gestures.We use a discrete set of postures based on a simple geometrical model,which includes only the position of the user’s two hands relative to his head.Sensor fusion of depth and color information is used in order to extract head and hand pose in a robust man-ner.The output of GestureDetector is used to control the robot’s motion, e.g., to activate a following behavior.3.2HardwareMobile Robot.In our work,we are using a Pioneer2-AT mobile robot(Figure2).The Pioneer2-AT is skid-steered and capable of traversing moderately rough natural terrain. Our Pioneer2-AT has a hardware microcontroller,Pentium-based computing,802.11wireless ethernet,a variety of sen-sors(sonar,drive encoders,differential GPS,magnetome-ter, inclinometer) and cameras.Imaging system.Two Pacific Corporation CSC-740color CCD cameras are mounted on a mast(Figure2).The cam-eras are equipped with wide-angle lenses(98.7deg HFOV) and are connected to two ImageNation PXC200framegrab-bers.We capture and process sub-sampled images (160x120pixels)at15Hz and use the SRI small vision sys-tem to compute disparity maps.To guarantee that the robot can locate humans in a wide area and that it can easily view head/hands we employ a tilted camera configuration(see Figure3).This geometry also allows the HumanTracker to operate quickly,because it only needs to search a portion(i.e.,a horizontal band)of theimage to detect humans.The primary disadvantage of this system is that distantobjects tend to appear foreshortened in the captured images.Since we use small images and track nearby objects(i.e.,head and hands)whose size does not vary greatly,this dis-tortion does not pose significant problems for tracking.Wedo,however,perform a correction when computing dispar-ity maps, in order to obtain accurate depth information. Figure 1:System architectureFigure 2:Pioneer2-AT with camera mastFigure 3:Camera geometry4.Human TrackingOur human tracking system is based on stereo vision.Stereo offers many advantages:it is capable of detecting partially occluded objects;it provides real-world information (object size,pose,velocity);and it is fairly immune to the effects of shadow, varying illumination, and camera dynamics.Stereo vision,however,does have limitations as well.Ste-reo tracking is not human-specific (i.e.,non-human objects will also be found).Furthermore,close objects may be seen to be a single object.Finally,depth measurement is non-lin-ear, thus localization accuracy will vary.4.1Z-slice ExtractionTo address these problems,our stereo tracker processes image data in a number of steps.First,given a disparity map (computed from a stereo pair),we compute a disparity his-togram.This histogram contains a number of peaks,which we call “z -slices”, corresponding to objects in the scene.We characterize each z -slice in terms of peak height,area,width and gradient (Figure 4).If the area is smaller than a threshold,the slice is considered as noise and is ignored.The slice with the lowest disparity (i.e.,furthest range)is taken to be the background.All values below that threshold are ignored.The slice with the highest disparity represents correlation error and is also discarded.Because of image noise,it is common for a large number of z -slices to be extracted.Thus,after the initial segmentation,we perform additional processing to connect “close”slices.We have found that a heuristic,based on slice separation,works well.4.22D SegmentationOnce a z -slice has been extracted,we compute a binary image and perform 2D segmentation using a connected-compound labeling algorithm.The output of this stage is a list of objects (“blobs”),which contains information about each object in the image plane (distribution,position,etc.).We use an area filtering algorithm to remove image artifacts such as disconnected regions (caused by insufficient image texture)and stray pixels (caused by noise).Blobs that are very close to each other are connected into a single blob.4.33D Segmentation and Object ExtractionBy combining the histogram processing with 2D segmenta-tion,we are able to partition the 3D scene into layers (Fig-ure 5),each of which (except for the background)is guaranteed to contain at least one object.Based on this seg-mentation,we can determine which objects are humans by applying simple heuristics (physical dimensions,geometric constraints, etc.).Working by layer,we characterize each object in terms of its 2D pixel distribution,disparity,and world (3D)coordinates.Then,we perform a final global scan for overlapping objects.If any objects overlap,a merge is performed.The use of world coordinates greatly helps improve filter coher-ency and robustness.Because our system only uses pixel operations,filtering can be performed within the z -slice (peak),blob,and object spaces.Thus,the human tracker is very fast,achieving 10-15Hz (horizontal band processing only)on a 233MHz Pen-tium without any hardware acceleration.Figure 4:Disparity histogram and z-slice extraction Figure 5:Schema of 3D Object Extraction5.Human IdentificationThe HumanID module distinguishes different individuals through color indexing.Color indexing has proven to be an efficient,robust method for detecting and identifying col-ored objects[8].Our implementation is based on the use of color histograms,which are translation invariant and which vary slowly with changing view angle,scale and occlusion.5.1Normalized Color HistogramFor each person detected by the HumanTracker,we com-pute histograms of the image region centered on that per-son’s mid-section.To reduce the effect of varying illumination,the computation is performed using normal-ized colors (red and green).Additionally,each histogram is normalized to a scale of 100 to facilitate comparison.Figure 6contains two images of the same person and the corresponding normalized green histograms.Although the images show the person at different scales and under differ-ent illumination,we observe that the two histograms are very similar in shape.This indicates that we should be able to obtain a positive match.5.2Histogram IntersectionGiven a particular person to match,we compare histograms through intersection .If we consider a histogram to be a function,then intersection can be interpreted as the mini-mum value at each point of the functions[8].Thus,to com-pute the intersection,we simply select the pixels that have the same color in both histograms.To obtain a confidence value,0(no match)to 1(complete correspondence),we normalize the intersection by the num-ber of pixels in the template (model)histogram.In practice,we compute confidence values for both normalized color histograms and then calculate the total match confidence by averaging the two values.6.Gesture RecognitionThe GestureDetector locates head and hand position using skin-color filtering.It then compares the relative positions against a set of pre-defined,static gestures.Figure 7shows the geometric model and a small set of the gestures we use for robot motion control.The GestureDetectorisactivatedwhenever a human standsin front of the robot for a short period.When this occurs,the GestureDetector searches for the human’s head position.It begins the search based on the horizontal body position returned by the HumanTracker.Because the HumanTracker tracks the lower part of the body,we can assume that the head is located above this region,most likely within a one vertical region.The GestureDetector is designed to look for color blobs that fall within a pre-defined skin-color locus[7].We use a fairly large skin-color matching locus in the normalized red /nor-malized green plane.If the initial search fails (i.e.,a blob cannot be found above the horizontal body position),then the search area is widened to the entire image.In either case,the largest skin-colored blob is taken to be the head.Once the head position is known,GestureDetector then looks for the person’s two hands,which are assumed to be located to the left and the right of the head.As with head detection,we perform a skin-color based search.We also perform consistency checks to verify that the head-hand relationship (distance, angle, etc.) is physically possible.Figure 6:normalized color histogramsFigure 7:Left, human geometrical model;right, possible postures.7.Results7.1Human TrackingThe performance of the human tracker is strongly correlated to a number of parameters.The effectiveness of the z -slice extraction is dependent on experimentally determined thresholds for area,gradient,and separation.Also,for both the 2D and 3D segmentation phases,there is a trade-off between resolution (how precisely an object is localized)and accuracy (how well an object is segmented from other objects).The results of a tracking experiment is presented in Figure 8.As the figure shows,the tracker correctly distinguishes the two subjects when:(a)they are separated;(b)there is partial occlusion;and (c)they overlap.Moreover,the tracker performs well even if the two subjects are (d)adja-cent at the same distance from the camera (i.e.,within the same z-slice).7.2Gesture recognitionThe GestureDetector is very conservatively.We explicitly designed it to minimize false positive matches,i.e.to avoid inadvertently triggering an incorrect action.As a result,however,users sometimes report that gesture recognition responds slowly and that it excessively precise hand posi-tioning is required.Figure 9shows some gesture recognition results.In the first three images (a-c),the module correctly identifies the static gestures.In image (d),however,no gesture was found because a hand could not be detected (i.e.,due to saturation effects because of the strong background illumination).This indicates that skin color-based filtering,by itself,is inadequate for robust gesture recognition.A betterapproach would be to use a more sophisticated tracking method, perhaps combining color with stereo.7.3Human IdentificationIn our testing,we found that the color histogram method works well for distinguishing between multiple individuals (Figure 10).In the top-left image,the individual to be tracked is highlighted by a rectangle.The top-right and bot-tom-left images show a second person entering the scene and approaching the robot.The bottom-right image con-tains three people in the scene.In all cases,the module suc-cessfully identifies the correct person.We found,however,that lightning conditions may impact performance even when indexing is based on normalized colors.For example,identification accuracy (match confi-dence)was generally better under strong illumination than with muted or dim lighting.This is because color differ-ences,particularly between dark clothing and the back-ground,is more apparent in a brightly lit scene.Figure 8:two people tracking(a)(b)(c)(d)Figure 9:gesture recognitionFigure 10:human identification(a)(b)(c)(d)(a)(b)(c)(d)Additionally,other factors such as varying illumination and shadows may influence performance through color shifts.In our current system,we compensate for these problems by periodically updating the color model (histogram).8.DiscussionPerhaps the principal challenge for vision-based human-robot interaction is designing algorithms that are robust in dynamic environments.Because scene characteristics (background,illumination,etc.)may change significantly or rapidly,it is important not to rely on a single method for perception.For example,even though colorfiltering is fast and efficient,it delivers poor results when there is strong backlighting or moving shadows.This is why we have cho-sen to use a combination of colorfiltering,color indexing, and stereo vision in our work.We have found that even with limited processing power,it is possible to design effective,human-oriented tracking algorithms.A key feature of our system is that it minimizes computation by taking advantage of camera geometry, using histogramming,and taking advantage of complemen-tary perception(e.g.,color indexing to assist stereo track-ing).Moreover,because each module executes quickly,the system performs well in crowded and changing environ-ments.9.Future WorkThere are several ways in which our current system could be improved.First,although we have achieved good results with small(160x120)images,high-resolution images would allow more precise segmentation to be performed. Second,we have found that area-based stereo correlation can be noisy.Thus,a more sophisticated stereo algorithm, perhaps one which incorporates global matching criterion or color segmentation,would aid detection and localization. Finally,adding additional sensors,such as an infrared cam-era(to distinguish between humans and inert objects), would improve the overall system robustness.To date,we have only obtained anecdotal evidence that computer vision methods facilitate human-robot interac-tion.Although it seems reasonable to assume that direct, proximal communication is desirable and productive,we believe it is important to obtain quantitative proof.Thus,a natural extension to our work would be to conduct a formal user study,in which we study how the interpretation of ges-tures and movement improves(or hinders)system usability.10.AcknowledgmentsWe would like to thank Bjorn Poell for his contributions and particularly for implementing the color indexing routines. We would also like to thank Roland Siegwart for providing project administration and the EPFL Institut de production et robotique for research hardware.11.References[1] D.Beymer and K.Konolige,“Real-time tracking of multiplepeople using continuous detection”,ICCV, 1999.[2]T.Fong,et al.,“Advanced interfaces for vehicle tele-operation:collaborative control,sensor fusion displays,and remote driving tools”,Auton. Robots 11(1), July 2001. [3]S.Ghidary et al.,“Localization and approaching to thehuman by mobile home robot”, IEEE RO-MAN, 2000. [4]T.Kiriki et al.“A4-legged mobile robot control to observe ahuman behavior”, IEEE RO-MAN, 1999.[5]K.Konolige,“Small Vision Systems:Hardware andImplementation”,8th ISRR, 1997.[6] D.Kortenkamp et al.,“Recognizing and interpreting gestureson a mobile robot”,AAAI, 1996.[7]M.Soriano et al.,“Using the skin locus to cope with changingillumination conditions in color-based face tracking”,IEEE NORSIG, 2000.[8]M.Swain and D.Ballard,“Color indexing”,InternationalJournal of Computer Vision, 7(1), 1991.[9]R.Tanawongsuwan et al.,“Robust Tracking of People by aMobile Robotic Agent”,GIT-GVU-99-19,Georgia Tech University, 1999.[10]J.Triesch,and C.von der Malsburg,“A gesture interface forhuman-robot interaction”,Face and Gesture, 1998.[11]S.Waldherr et al.,“A Neural-Network Based Approach forRecognition of Pose and Motion Gestures On a Mobile Robot”,5th Brazilian Symp. on Neural Networks, 1998. [12]G.Xu et al.,“Toward Robot Guidance by Hand GesturesUsing Monocular Vision”,IEEE Hong Kong Symposium on Robotics and Control, 1999.。