Variable Pallet Pick-Up for Automatic Guided Vehicles in Industrial Environments

Variable Pallet Pick-Up for Automatic Guided Vehicles in Industrial EnvironmentsDaniel Lecking, Oliver Wulf, Bernardo WagnerInstitute for Systems Engineering, Real Time Systems GroupUniversity of HannoverHannover, Germany{lecking,wulf,wagner}@rts.uni-hannover.deAbstract - This paper presents two laser scanner based approaches to locate and pick-up pallets with the aim of automating forklift trucks. In contrast to camera based systems our approaches are independent of luminance conditions which can be highly variable in industrial environments. Whereas one approach uses pallets modified with adhesive reflectors which enables very flexible handling of pallets, the second approach detects pallets based on geometric characteristics without any modifications to the environment. Experimental results are also reported including a run of an autonomous fork-lift truck presented at the logistics fair CeMAT 2005 in Hannover.I. I NTRODUCTIONAutomatic guided vehicle systems (AGVS) to engage pallets typically require that the location of the pallets be known a priori and that the pallets are accurate positioned. To perform pallet handling also when the pallet position is not precisely known in advance, e.g. when the loads are handled and placed by a human driver, only a few solutions have been presented. Most of these approaches are based on visual features extracted by image camera data. In reference [1] an image segmentation method based on color and the geometric characteristics of the pallet is proposed. For this approach a camera calibration algorithm and a good illumination system is required. A system which does not require to maintain a strict camera calibration is presented in [2]. Therefore fiducials are used, which are artificial visual features placed on the pallets.A model-based vision algorithm without any artificial illumination support has been implemented in [3], the algorithm based on the identification of the central cavities of the pallets. In [4] a combination of range camera and a video camera are used to detect pallets.In our approach we use laser range scanner to localize and pick-up pallets in variable positions and at variable heights. In contrast to camera based systems our approach does not suffer with distortion and illumination problems or object scaling problems which can result in misdetection or false detection of important features [5]. The only other system based on a laser scanner that the authors are aware of is the so called “Pallet Finder” which is being developed at the EADS SPACE Transportation GmbH [6]. Unfortunately, there is no public information available that describes the methods employed for pallet recognition.This paper presents two approaches to localize and pick-up pallets in an industrial environment. The first approach uses reflectors to calculate the position and orientation of a pallet whereas the second approach uses only geometrical characteristics. The two approaches are described in section II. Section III gives a description of the trajectory generation as well as the tracking and docking system to fork the pallet. Experimental results are presented in section IV e.g. a working forklift truck as shown in Fig.1.Fig. 1. Autonomous fork-lift truckII. P ALLET R ECOGNITIONThe sensors that have been employed for the experiments is a SICK 2D laser scanner S3000 to recognize pallets at ground plane and in addition a SICK 2D laser scanner LMS200 to recognize pallets at variable heights additional. The main advantage of using the S3000 laser scanner based on the additive use to fulfill requirements defined in the standards for autonomous mobile systems in industrial automation and services [7]. The following two approaches are described to identify and locate a standard Euro pallet of size 1200x800mm.A. Identification based on adhesive reflectorsThe goal to localize a pallet in an industrial environment is a demanding task, especially when the location of the pallets is completely unknown. Therefore we decided to implement an effective and robust method to localize pallets with the use of reflectors placed on the pallets. The use of adhesive reflectors for industrial tasks seems maintainable as many tasks require custom made pallets anyway.For this purpose two reflectors manufactured by 3M, type Scotchlite 3000X, are fixed left and right on the short-side of the pallet (Fig. 2). In order to localize the pallet with the assistance of the reflectors and the laser scanner, the pallet with its modified side has to be in sight of the laser scanner. Thus theoretically the maximum range for identification could be the range of the laser scanner.Fig. 2. Euro pallet modified with reflectorsTo detect the reflector marks the reflected energy of the measurement beams are computed by the laser scanner. The reflectivity of reflectors are typically above 2000 per cent (high gloss finished steel approx. 200 per cent) and therefore distinguishable from other materials. To identify the position and orientation of a pallet the measured reflector positions have to be accumulated and correlated to each other. For this purpose successive measurements of the laser scan identified as reflector points are distinguished into groups and afterwards the epicenters are calculated.In this way the entire surroundings are searched for reflectors. Each found reflector is checked to discover whether a second reflector is present nearby, which together could constitute a reflector pair. Reflector pairs are constituted if the distance d r corresponds with a defined distance of the reflectors from one another with consideration given to the sensor tolerances. The defined distance in this case is the distance of the reflectors on the pallet.()()2121++−−−=i i i i r y y x x d(1)If a reflector pair is established the position and the orientation of the pallet relative to the automatic vehicle will be established by a simple geometric relationship (Fig.3). The calculated pallet position is represented in Cartesian coordinates (x p , y p ) (2) and the orientation ϕp (3).⎟⎠⎞⎜⎝⎛−+=++211i i i p x x x x⎟⎠⎞⎜⎝⎛−+=++211i i i p y y y y(2)ii ii p y y x x −−=++11arctanϕ(3)In order to keep a number of pallets apart and avoid any misinterpretations, an expectation value should be prescribed for the tracing of the pallets, which, however, according to the environment can be chosen with great leeway or under some circumstances not be required at all.()11,++i i y x pFig. 3. Pallet identification based on adhesive reflectorsB. Identification based on geometrical informationIt is not possible to place reflector marks in all fields of activity. In areas where materials are transported from factory to factory it is often not possible to fit reflectors. For this reason a geometric based recognition system is being developed as an addition to the reflector recognition system. The same sensor used for pallet recognition with reflectors is used for geometric based pallet recognition. In contrast to the first task only the distance measurements are employed. This approach for pallet recognition resets upon the well-known ICP algorithm [8]. IDC is an iterative algorithm that matches two scans in a point-to-point manner. This means it takes the scan-points, one by one, in the reference scan and tries to find a corresponding point in the current scan.In this task instead of two scans being registered for the IDC algorithm an actual 2D scan S of the environment is registered with a model of the pallet, similar to the 3D methods proposed in [9] and [10]. Thereby the position and orientation of the pallet is discovered in an actual scan. Afterwards the number of the scan points is reduced in a firstworking phase. This follows with the help of an expectation of the pallet position (x E, y E ).Around the expectation value an area of the size of the Euro pallet plus the possible position tolerance (x G, y G ) is cut out. Afterwards only the scan points, which are inside the given area, are regarded (4).⎪⎭⎪⎬⎫⎪⎩⎪⎨⎧±=∧±=∧∈=G E G E i y y y x x x S y x y x p S ),(|),()('(4)Before the scan points respective the scan S’(p i ) can be used for the ICP algorithm, a suitable model of the pallet must be found. In total three models M k are at ones disposal for pallet recognition. Each model consists of m individual line segments, which reflect the contours of a Euro pallet in a Cartesian coordinate system. A line is represented by a point (x, y) and a direction vector r =(r x , r y ).()(){}mi i y x k r r y x M ,...,1,,,==(5)As the laser scanner scans a little above the floor, the individual wooden slats of the pallet will be identified by the laser scanner, whereby according to the orientation of the sensor to the pallet not all the slats will be identified. From experiments it has been shown that three models of the pallet are sufficient in solving the problem.Fig. 4. Pallet models representing the wooden slatsWith the help of the expectancy value of the pallet position and orientation one of the models as seen in Fig. 4 is chosen. Depending on ϕ, M 1 is chosen if ϕ ≤-45 degree, M 2 if ϕ >-45 degree up to <45 degree and M 3 if ϕ ≥45 degree.In the next working phase the chosen pallet model M k is transformed to the expectancy value so that the scans points S´(p i ) and the transformed model M k ’ represented in the same coordinate system.In order to carry out the IDC algorithm corresponding points between the pallet model and the reduced scan must be found. For this purpose model M k ’ made up of the line segments must determine the next points to the scan points S´(p i ). This occurs whereby the shortest distance d i is calculated between each scan point p i (x i ,y i ) and a line segment. The receivingpoint p i ’(x i ’,y i ’) of the line segment is used as a reference point to the observed scan point (Fig. 5.).p i i (x i , y i )Fig. 5. Corresponding points between the pallet model M k ’and scan S´From the Euclidean distancesi i i p p d ′−=(6)the middle distance d is calculated.i Ii i p p I d ′−=∑∈1(7)On this basis and with the threshold valueσμ3+=d c(8)with σ as standard deviation a reduction of the number of point pairs is undertaken. This effects the filtering out of point pairs with a too great distance between them. To find the new expectation, we calculate the centroids with the reduced point pairs()∑==ni i S S S p n y x 11,μ()∑=′=ni i M M M p n y x 11,μ(9)and use the covariance matrix()()M i ni s i p p n YY YXXY XX K μμ−′−=⎟⎟⎠⎞⎜⎜⎝⎛=∑=11 (10)to determine the orientation and translation⎟⎠⎞⎜⎝⎛−−=ΔYY XX YX XY arctan ϕ(11)))sin()cos((ϕϕΔ−Δ−=S S M x y x x T))cos()sin((ϕϕΔ+Δ−=S S M y y x y T(12)(13)by minimizing E dist as described in [11]∑=Δ′−+′=Δni dist M T S R T E 12),(ϕϕ(14)were⎟⎟⎠⎞⎜⎜⎝⎛ΔΔΔ−Δ=Δ)cos()sin()sin()cos()(ϕϕϕϕϕRis the rotation matrix.On the basis of this expectation value for the position of the pallet the whole procedure will be repeated until the alternations of Δφ are smaller than a fixed threshold value. The last detected estimate value for the pallet position will then be accepted as sufficiently accurate. An example of pallet identification based on the geometrical information can be seen in Figure 6.Fig. 6. Pallet identification Scan points (red), pallet model (blue)III. S PLINE G ENERATION AND D OCKINGOnce the position of the pallet relative to the vehicle respective to the fork lift truck is obtained, a trajectory has to be generated to pick up the pallet. The trajectory planning and the ensuing vehicle control for the contact with and lifting of the pallet have been developed based on the pallet identification. The fork-lift truck in its starting position is a few meters from the pallet. However, the pallet is in sight of the fork-lift truck’s 2D laser scanner and can be recognized as described by one of the two approaches. The preeminent goal is to achieve a precise and collision-free docking of the fork-lift truck to the pallet.The purpose of the trajectory planning is of a path from starting to an ending point with regards to the cinematic constraints of the vehicle.In this work the established movement plan for docking is used to draw up suitable guiding points of the nominal path and then leads on the creation of the movement plan. Theassisting points comprise of the relative distance of the fork-lift truck from the pallet and the actual position of the fork-lift truck as well as a further guiding point, which is immediately in front of the pallet at a distance of about 1,300mm. With the help of these guiding points a nominal path will be generated from the movement plan, made up of straight and curved paths with a constant radius. As the generated nominal path cannot be driven exactly, because of the discontinuities at the line-arc-line transition points, the circular arc segments are replaced by polar-splines.The use of polar-splines provides continuous curvature of the arc segments. The continuous curvature κ is described by()222222r rr r r r ′+′′⋅−′+=κ(15)withφd dr r =′ 22φd r d r =′′(16)and)221()(2432γφγφφφ⋅+++=i i R r , γφ≤≤0 (17)R i is the radius and γ the open angle of the circular arc segment to be replaced. For further details see [12].One can see an example of a trajectory for the docking on the pallet in figure 7.Fig. 7. Trajectory for the docking on the palletAdditionally for each path segment a desired speed and driving direction is selectable. In general driving along the trajectory without a control unit is not sufficient for the precise docking of the fork-lift truck on the pallet Therefore trajectory tracking is performed by a cascade controller, combining feed forward and feedback control for high pathaccuracy as described in [13]. As the vehicle moves towards the pallet, the position of the vehicle is estimated by odometric sensors and the continual identification of the pallet’s position whereby the Kalman Filter update algorithm is used.IV. E XPERIMENTAL R ESULTSA long term test of the pallet localization system based on adhesive reflectors has been carried out with the RTS-STILL Robotic Fork-Lift during the logistics fair CeMAT 2005 in Hannover (Fig. 1.). The fork-lift truck was extended to a flexible automatic guided vehicle with additional hardware including two 2D laser scanner for pallet recognition and an embedded pc [14]. As the pallet localization based on adhesive reflectors is a straightforward implementation the system operated 5 days (8 hours a day) without any failure. The only limitation that have to be observed is that a precise differentiation between reflectors and material with high reflectivity e.g. high grade steel of shelving systems can be achieved – which should be possible with an adapted threshold.To prove the applicability of the pallet localization based on geometrical information tests with an experimental robotic platform have been undertaken at the Institute for Systems Engineering. In order to carry out the experiments the robotic platform MoRob-Kit [15] was fitted with an 2D laser scanner to identify pallets at ground level. Varying the position and the orientation of the pallet with respect to the vehicle a set of sample situations have been tested. The error due to the imprecision of the pallet localization method has been studied and shown in Table I. The actual (x R,y R,φR), the first estimation (x E,y E,φE), the calculated (x C,y C,φC) coordinates and the position error are listed for ten average samples per pallet orientation as well as the computation time and number of iterations required for ICP algorithm. All execution times are measured on a onboard Pentium III 266MHz computer. The discrepancy between the actual and the calculated pallet position has an average of 5 mm along the x-axis, 10 mm along the y-axis and 0,5 degree orientation error with a standard deviation of 5mm, 12mm and 0,3 degree. The mean and standard deviation of the y component is larger and dependent on the pallet orientation because the number of measurement points available for pallet localization is also dependent on the orientation. For pallet orientations near to 0 degree respectively ±90 degree the error is much higher because of missing measurements at the outsides of the pallet. In other orientations the v-formation of the measurement points effects the small residual error of pallet localization. Due to the reduction process of measurement points and thus small number of correspondence points the computation time of average 6ms ±1ms and number of iterations with an average of 9 is quite low.As the geometry of the pallet is symmetric the ICP algorithm converges to the correct position only if the start value is close to the actual pallet position and should be ±150mm along the x-axis and y-axis and ±15 degree of orientation. However, it should be noted that the environment surrounding the pallet was cleared of other pallets or objects that might have confused the algorithm.TABLE I.GEOMETRICAL LOCALIZATION RESULTSx Rmmy RmmφRdegrex Emmy EmmφEdegrex Cmmy CmmφCdegre20000 0 2017142 7,1 2001-18 0,7 20000 0 1993147 -1,8199716 -0,6 20000 0 194093 14,3199930 -0,7 20000 0 190093 15,9200130 -0,7 20000 0 218080 4,3 1992 6 0,1 20000 50 187580 41,0 1991 2 49,5 20000 50 2022-173 43,4 1987-5 49,0 20000 50 1795-133 50,8 1998-3 50,1 20000 50 179513 45,9 1988 3 49,4 20000 50 211567 49,6 2002 2 50,2IterationsComputation timeMsErrorΔxErrorΔyErrorΔφ5 3,31-180,77 4,6-316-0,612 6,2-130-0,714 7,4 130-0,78 4,3-860,17 6,1-92-0,58 6,1-13-5-1,013 7,2-2-30,17 6,2-123-0,66 5,3220,2V. C ONCLUSIONIn this paper we have presented two laser scanner based approaches to perform pallet handling for Automatic Guided Vehicles even if the pallet position is not precisely known in advance. One approach uses reflectors placed on the pallets and therefore enables an effective and robust localization ofthe pallets even if the position of the pallet is completely unknown. The second approach localizes unmodified standard pallets based on a ICP algorithm. Whereas the approach deals with an initial position inaccuracy of the order±150mm and ±15 degrees. Both approaches are applicable in industrial environments whereby it depends on the task witch system should be used.The extensive experimental results of the reflector based approach colleted during many hours of test in lab and on a logistic fair demonstrate the reliability of the system. The Robotic Fork-Lift presented at the logistics fair CeMAT 2005can be seen in a video available at [16].We actually plan to use both approaches in a demonstrationof a robotic fork-lift truck to provide an opportunity to compare the two approaches in industrial conditions.R EFERENCES[1] J. Pagès, X. Armangué, J. Salvi, J. Freixenet and J. Martí, “A ComputerVision System for Autonomous Forklift Vehicles in Industrial Environments” Mediterranean Conference on Control and Automation (MED), 2001.[2] M. Seelinger and J.-D. Yoder, “Automatic Pallet Engagment by aVision Guided Forklift”, International Conference on Robotics and Automation (ICRA), 2000.[3] G. Garibott, S Mascianglo, M. Ilic and P. Bassino, “ROBOLIFT: Visionbased autonomous navigation of a conventional fork-lift for pallet handling”, International Conference on Advanced Robotics (ICAR),1997.[4] J. Nygards, T. Hogstrom and A. Wernersson, “Docking to Pallets withFeedback from a Sheet-of-Light Range Camera”, International Conference on Robots and Systems (IROS), 2000.[5] R. Cucchiara, M. Piccardi and A. Prati, “Focus based feature extractionfor pallet recognition” British MachineVision Conference (BMVC), 2000.[6] …PalletFinder”, EADS SPACE Transportation GmbH Automation &Robotik, /3da/Anwendungen/FTS.html[7] European Standard “Safety of Industrial Trucks, Driverless trucks andtheir systems” prEN 1525, 2000. [8] P. Besl and N. McKay, “A method for registration of 3d shapes”, Trans.Patt. Anal. Mach. Intell. 14(2), pages 239-256, 1992.[9] T. Pajdla, L.V. Gool, “Matching of 3-D Curves using Semi-differentialInvariants”, International Conferenceon Computer Vision, 1995. [10] D. Chetverikov et al., “Robust Euclidean Alignment of 3D Point Sets”,Image and Vision Computing 23, 2005.[11] F. Lu and E E Milios. "Robot pose estimation in unknownenvironments by matching 2D range scans", Conference on Computer Vision and Pattern Recognition, 1994.[12] J. Horn “Path control of a mobile robot based on absolute localizationby fusion range image and deadreckoning data”, Fortschrittberichte VDI Reihe 8, vol 617, 1997.[13] M. Hentschel, O. Wulf and B. Wagner, “A Hybrid Feedback Controllerfor Car-Like-Robots”, submitted to the International Conference on Informatics in Control, Automation and Robotics (ICINCO), 2006. [14] B. Wagner, P. Hohmann, U. gericke, and C. Brenneke, “Technicalframework for robot platforms in education,” International Conference on Robotics and Automation (ICRA), 2004.[15] D. Lecking, O. Wulf and B. Wagner, “The RTS-STILL robotic fork-lift:Variable pallet pick-up and 3D localization in industrial environments”European Robotic Network (EURON), 2006.[16] http://www.rts.uni-hannover.de/mitarbeiter/lecking/RTS_STILL_CEMAT2005.wmv。