View independent human body pose estimation from a single perspective image

View Independent Human Body Pose Estimation from a Single PerspectiveImageVasu Parameswaran and Rama ChellappaCenter for Automation Research,University of Maryland,College Park,MD20742vasc,rama@AbstractRecovering the3D coordinates of various joints of the hu-man body from an image is a criticalfirst step for several model-based human tracking and optical motion capture systems.Unlike previous approaches that have used a re-strictive camera model or assumed a calibrated camera,our work deals with the general case of a perspective uncali-brated camera and is thus well suited for archived video. The input to the system is an image of the human body and correspondences of several body landmarks,while the out-put is the set of3D coordinates of the landmarks in a body-centric coordinate ing ideas from3D model based invariants,we set up a polynomial system of equa-tions in the unknown head pitch,yaw and roll angles.If we are able to make the often-valid assumption that torso twist is small,we show that there exists afinite number of solu-tions to the head-orientation which can be computed readily. Once the head orientation is computed,the epipolar geom-etry of the camera is recovered,leading to solutions to the 3D joint positions.Results are presented on synthetic and real images.1.IntroductionHuman body tracking and optical motion capture are two facets of the large area of research commonly referred to as human motion analysis.Good starting points for un-derstanding the applications,specific problems and solution strategies are survey papers,recent ones being[14]and[8]. Human body tracking and optical motion capture systems rely on good bootstrapping-an accurate initial estimate of the human body pose.This is a difficult problem partly due to the large number of degrees of freedom of the body,mak-ing searching for solutions computationally intensive or in-tractable.Part of the difficulty also arises because of loss of depth information and non-linearity introduced due to per-spective effects.To make the problem more tractable,re-searchers have resorted to assuming a scaled orthographic camera in the uncalibrated case or a calibrated camera in the perspective case,both of which are more restrictive than one would like in practice.In[13],Taylor uses a scaled orthographic projection model and shows that there is an infinite number of solutions parameterized by a single scaleparameter.Afterfixing arbitrarily,there is afinite num-ber of solutions to the problem because of symmetries abouta plane parallel to the image plane.Further,the methodcannot be employed in cases where strong perspective ef-fects exist.In[1]the authors recover both anthropometry and pose of a human body.However,they use scaled or-thographic projection and present a search based,iterative solution to the problem pruning the search-space using an-thropometric statistics.In[2],Bregler and Malik restrict the projection model to scaled orthographic for initializa-tion and tracking of the joint angles of a human subject.In [3]and[6],multiple calibrated cameras are used to build a 3D model of subjects while in[7]the authors work with a single camera but assume that it is calibrated.In[9]the au-thors use a learning approach to predict the2D marker posi-tion(2D pose)from a single image while in[10]they build on the work and present an approach for recovering the ap-proximate3D pose of the body from a set of uncalibrated camera views.Their work is interesting in that they do not require joint correspondences to be provided in the image. Rather,they employ a machine learning and a probabilistic approach to map the segmented human body silhouette to a set of2D pose hypotheses and recover the3D pose from them.Recently,Grauman et.al[4],reported on a prob-abilistic structure and shape model of the human body for the recovery of the3D joint positions given multiple views of the silhouettes from calibrated cameras.In[11],Smin-chisescu and Triggs report an approach for monocular3D human tracking.Model initialization is search based and camera parameters are assumed known.Working with calibrated image/video data and/or mul-tiple cameras is possible only in restricted application do-mains.Most archived videos are monocular with unknown camera parameters(intrinsic and extrinsic).Moreover,the scaled orthographic assumption may be too restrictive for many cases.We believe a full-perspective solution to the problem will increase the applicability of good tracking al-gorithms such as Bregler’s[2]because in addition to pro-viding a more accurate initial estimate,one can recover theperspective3D to2D transform of the camera,making it possible to carry out full-perspective tracking of the human body.In this work,we aim for such a solution and seek to estimate the3D positions of various body landmarks in a body-centric coordinate ing ideas from model based invariance theory,we set up a simple polynomial sys-tem of equations for which analytical solutions exist.In cases where no solutions exist,an approximate solution is calculated.Recovering the3D joint angles,which are help-ful for tracking,then becomes possible by way of inverse kinematics on the limbs.2.Problem StatementWe employ a simplified human body model of fourteen joints and four face landmarks:two feet,two knees,two hips(about which the upper-legs rotate),pelvis,upper-neck (about which the head rotates),two shoulders,two elbows, two hands,forehead,nose,chin and(right or left)ear.The hip joints constitute a rigid body.Choosing the pelvis as the origin,we can define the X axis as the line passing through the pelvis and the two hips.The line joining the base of the neck with the pelvis can be taken as the positive Y axis. The Z axis points in the forward direction.We call the XY plane the torso plane.We scale the coordinate system such that the head-to-chin distance is unity.With respect to the input and output,the problem we seek to solve in this pa-per is similar to those addressed previously(e.g.[13],[1]): Given an image with the location in the image of the body landmarks and the relative body lengths,recover their body-centric coordinates.We make use of two assumptions:1.We use the isometry approximation where all subjectsare assumed to have the same body part lengths when scaled.The allometry approximation[16]where the proportions are dependent on body size is considered to be better because the relative proportions depend upon body size:for instance,children have a propor-tionally larger head than adults.Our algorithm,how-ever,is invariant to full-body3D projective transfor-mations.2.The torso twist is small such that the shoulders take onfixed coordinates in the body-centered coordinate sys-tem.Except for the case where the subject twists the shoulder-line relative to the hip-line by a large angle, this assumption is usually applicable.Further,since our algorithm relies on human input,it is easy to tell if this assumption is violated.3.ApproachBesides the articulated pose of the human body,the un-known variables in the problem are the extrinsic and in-trinsic camera parameters.In[12],Stiller et.al.derive camera-parameter independent relationships betweenfive world points on a rigid object and their imaged coordinates for an affine camera.Weiss and Ray in[15]simplified and extended the result to the full-projective case showing that there exists one equation relating three3D invariants and four2D invariants formed six world points and their im-age coordinates.Our approach is motivated by theirs but we are able to derive a simplerfinal result involving two 3D invariants rather than three.In the following,we will first show how to recover the three angles of rotations of the head in the body-centric coordinate system,given the image locations of the body landmarks.From the recovered head orientation,we next show how the3D coordinates of the remaining joints can be recovered.Recovery of these quan-tities also allows us to determine the epipolar geometry of the camera.3.1Motivating ExampleWe review and modify the approach of[15]below.Five points(in homogenous coordinates)in3D pro-jective space cannot be linearly independent.Assuming that thefirst four points are not all coplanar,we can write the3D coordinates of thefifth point with the basis as thefirst four points:(1) The are the unknown projective scale factors and are the unknown projective coordinates of the point in the basis of thefirst four points.We would liketo model a point configuration where four points lie on the same plane.Given that we need thefirst four form a ba-sis,we can choose a labeling such that points1,2,4and5 form a plane while points3and6lie outside this plane1. In this configuration,point3doesn’t contribute to point5’s coordinates,making zero.For wehave:Figure1:Six point configuration used for analysis.Points form a plane and lie outside this plane(2) Here,,and are the basis coordinates for.If is the world to image transform such that, where is an unknown scale factor,the image coordinate for thefifth point,is given by:(3)Writing as,and doing the same algebra for point 6,we have the following two equations relating image co-ordinates:(4)We would like to eliminate the projective coordinates and the scale factors.Let denote the determinant(the notation is such that we index by the point left out from thefive-point set(). Substituting for from(1)and noting that determi-nants with two equal columns vanish we have:The projective coordinates and scale factors can be elimi-nated by taking cross ratios to obtain two3D invariants(as opposed to three in[15]):For the image coordinates,we follow the same approach of taking determinants and their ing the ‘points-left-out’notation,let denote the determinant :(6)Letting denote the determinant,we similarly obtain:(7) Obtaining expressions for the other determinants, ,,,calculating cross ratios andand equating them to and respectivelywe obtain:(8) The are known quantities,computed from image coor-dinates and we can rewrite the above equations in terms of known coefficients,as(9) (9)expresses a view-invariant relationship between solely the3D coordinates and their2D image positions for the six points shown infigure1.3.2Recovering the Head OrientationIf we choose the following labeling of points:right-hip(1),left-hip(2),left-shoulder(4),right-shoulder(5)and allow points3and6to be any two head features in(say forehead and chin),the only unknowns in the equation are the coor-dinates and.Being positions on the head,which is a rigid body that rotates about the upper-neck,in effect,there are only3scalar unknowns corresponding to a rotation ma-trix.If we use Euler angles,we can write:where and are known forehead and chin coordi-nates corresponding to a reference‘neutral’position.Ob-serving that the third elements of are zero,where is the signed area of points,a known constant and is the th row of.Similarly,When the above are substituted into(9),the scalar cancels out and we obtain(10) where,,. We now write expressions for:(11) Expanding R in terms of the Euler angles,and substituting it in the expressions for the determinants, (10)becomes a13term transcendental equation in the Eu-ler angles.Given the point correspondences of two more head features,say the nose and either ear,we will have three equations in the three unknown Euler angles.The equations depend on the neutral position of the head reflected in and.Choosing a neutral position where the head points forward with no yaw or roll,the coordinates are zero for the forehead,nose and chin and two of the equations be-come four term equations giving:(12)(13)(14) where and.Interestingly,(12)and (13)are independent of and can be solved rather trivially using and.We obtain a quadratic equation in:(15) where the can be written in terms of and.Hence there are upto four solutions for and.When these are substituted into(14),we obtain a simple equation in:(16)where the can be written in terms of,and.With ,we obtain two solutions for.Collectively, we then obtain upto eight solutions for the angles.The angle solutions represent head orientations that pro-duce the image.At this stage,we could do some rather basic anthropometricfiltering by observing that the pitch angle cannot be so large that the chin penetrates the torso. Similarly,we could also impose constraints on the roll and yaw angles.The valid solutions can then be presented to the user from which one will be selected.3.3Recovering the Epipolar Geometry Recall that projects points from the body-centered coor-dinate system to the image plane.Given the calculated head orientation,we can recover,which has eleven unknowns. From the eight point correspondences at our disposal(four head plus four torso),we have an overdetermined set of six-teen equations in the elements of which we solve for in a least squares sense using singular value decomposition. The matrix contains all information necessary to re-trieve the camera center.can be written in the formwhere is the camera center[5]. Given this,can be recovered as.3.4Recovering Body Joint Coordinates Consider any unknown world point with known image point.Inverting the relationship, we obtain a set of solutions for parametrized by the un-known.This is simply the epipolar line of the image point in the body-centered coordinate system.(17)(18) where can easily be calculated in terms of elements of and.Let represent the right elbow which is con-nected to the right shoulder with known world coordinates.We also know the upper arm length, .We then have the following constraint:(19) which is a quadratic in,representing the two points of intersection of the epipolar line with the sphere of possible right elbow positions.These two solutions for the elbow represent the unavoidable forward/backwardflipping ambi-guity inherent in the problem.Once the correct right elbow position is found,the right hand can be found in the same manner.Similarly,we can obtain the3D coordinates of all the other joints of the body.The interactivity in this solu-tion process can be eliminated by having having the user pre-specify the relative depths of the joints.In other words, before the solution process starts,each joint is assigned a boolean variable that specifies whether that joint is closer to the camera than its parent.Given that the user is specify-ing the point correspondences of body landmarks,this input imposes trivial additional burden.This idea is also used in [13].Since we have already calculated the camera center, we are able to calculate these distances readily.3.5Dealing with Unsolvable Cases Computation of the head-orientation as well as the limb 3D locations involves the solution of quadratic equations.In our experiments on real images and noisy synthetic im-ages,in several cases,there were no solutions to one or more quadratic.For the head-orientation case,we recov-ered as solutions to a constrained optimizationproblem with the objective function as the sum of squares of12and13along with the trigonometric identities as con-e of Lagrange multipliers resulted in a non-trivial system of polynomial equations in.Wetried two different approaches:(1)computing a Grobnerbasis of the polynomials so that they are reduced to triangu-lar form and(2),searching for local optima.Grobner basiscomputations were rather heavy and slowed down the algo-rithm,although the recovery of all local minima was guar-anteed.Searching for local optima(in the space wasfound to be much faster(the search space was quantized into bins)and produced a good approximatesolution most of the time.For the limb position(19)with nosolutions,we computed a scale such that the scaled limb-length(in this case)made the discriminant positive.This effectively accounted for variations in the assumed andactual limb lengths.4ResultsWe evaluated the approach on synthetic and real images,the results of which we present below.4.1Synthetic ImagesIn the synthetic case,given that the error is zero for a perfect model and perfect image correspondences,we focussed on empirical error analysis.There are two sources of error:(1) differences between the assumed model and imaged subject and(2)inaccuracies in the image correspondences.Forfive different viewpoints,and500random unknown poses per noise-level,we calculated the average error in full-body re-construction(sum of squares of the difference between real and recovered3D coordinates scaled by the head-to-foot distance)for Gaussian noise of zero mean and unit standard deviation and increasing noise intensities.The interactivity of the algorithm was eliminated by the evaluation program automatically choosing the head-orientation with minumum error among the solutions.There are three cases:noisy-model,noisy-image,and noisy-model with noisy-image. For image noise,we perturbed the image coordinates with the noise,scaled by the image dimensions which were taken to be those of the bounding box of the imaged body.For model-noise,the scale was the head-to-foot distance.Figure 2shows the dependency.An important observation is that the reconstruction is more sensitive to errors in the model than in the image point correspondences.Interestingly,the curve for noisy-model with noisy-image error is almost theFigure2:Error dependency on noisesame as the noisy-model curve.We believe that this is be-cause the model error swamps out image errors which are much smaller,especially at higher noise levels.Further, since the model and image errors are independent,errors cancel out in some cases.Nevertheless it can be seen that small errors in the model and image only produce small er-rors in thefinal reconstruction.4.2Real ImagesWe evaluated the qualitative performance of the approach on real images by using3D graphics to render the recon-structed body pose and epipolar geometry.We used a3D model derived photogrammetrically from front and side views of one subject and used the same3D model for all images.There were two important problems with real im-ages:One problem is that clothing obscures the location of the shoulders and hips,the accuracy of which affects the head orientation computation.We addressed this problem with two strategies.First,given that the shoulders,hips and upper-neck form a planar homography we compute and use it:though we do not use the upper-neck as a feature point in(12),(13)and(14),we require the user to locate it. The homography is uniquely specified by four planar points. We use thefive torso points to calculate the torso-plane-to-image homography in a least-squares sense,transform the torso-plane to the image using the homography and use the transformed points as input rather than the user-specified points.Second,rather than requiring the user to locate the true right and left hip(about which the upper legs rotate), we just require their surface locations(i.e.‘end-points’), which are easier to locate.The model stores the true centers of rotation of the legs as well as the surface locations.Another problem is due to the fact that we model the neck juction as a ball and socket joint.In reality,the skull rests on top of the cervical portion of the spinal cord and the cervical vertebrae are free to rotate(although by a smallFigure 3:Person Sitting,Front-viewFigure 4:Baseballamount and with a small radius).To compensate for this,we take the skull center of rotation to be midway between the neck-base and upper-neck.This produced a significant im-provement in the head-orientation recovery for cases where subjects lunged their head forward or backward in addition to rotating it.For some images where these two effects were signifi-cant,we had to guess the true image coordinates three or four times before the algorithm returned realistic looking results.Figure 3shows a subject sitting down and imaged from the front.Also shown in the image are user-input loca-tions of various body landmarks.Beside the image are two rendered views of the reconstructed body pose and epipo-lar lines of the body landmarks from novel viewpoints.The meeting of epipolar lines depicts the camera position.Fig-ure 4shows a baseball pitcher and the reconstruction.Inter-estingly in this case,the camera is behind the torso of the subject and this fact is recovered by the reconstruction.Fig-ure 5shows a subject sitting down with the hand pointed to-wards the camera,inducing strong perspective while figure 6shows subject skiing.The novel views of the reconstruc-tions show that the body pose is captured quite well.5ConclusionsWe presented a method to calculate the 3D positions of var-ious body landmarks in a body-centric coordinate system,given an uncalibrated perspective image and point corre-spondences in the image of the body landmarks -an impor-tant sub-problem of monocular model-based human body tracking and optical motion capture.Our small-torso-twist assumption gives us enough ground truth points on the torso and allows us to use ideas from 3D model based invariance theory to set up a simple polynomial system of equations to first recover the head orientation and with it,the epipolar geometry and all of the limb positions.While theoretically correct given the assumptions,the method encountered spe-cific problems when applied to real images,which we ad-dressed by way of strategies to reduce error in input as well as the model.We demonstrated effectiveness of the method on real images with strong perspective effects and empiri-cally characterized the influence of errors in the model and image point correspondences on the final reconstruction.Given that model accuracy has significant impact on the re-construction,we are evaluating a probabilistic approach for reconstruction using anthropometric statistics.In future,we plan to exploit the analysis by synthesis approach to render the reconstructed head on to the image plane and iteratively refine the reconstruction using color and edge cues.AcknowledgmentsThis work was supported in part by NSF Grant ECS 02-25475.Figure 5:Person Sitting,Side-viewFigure 6:Person SkiingReferences[1] C.Barron and I.A.Kakadiaris.Estimating anthropometryand pose from a single uncalibrated puter Vision and Image Understanding ,81,2001.[2] C.Bregler and J.Malik.Tracking people with twists and ex-ponential maps.Proc.IEEE Conference on Computer Vision and Pattern Recognition ,1998.[3] D.Gavrila and L Davis.3-d model-based tracking of humansin action.Proc.IEEE Conference on Computer Vision and Pattern Recognition ,pages 73–80,1996.[4]K.Grauman,G.Shakhnarovich,and T.Darrell.Inferring 3dstructure with a statistical image-based shape model.Proc.International Conference on Computer Vision ,2003.[5]R.Hartley.Chirality.International Journal of ComputerVision ,26(1):41–61,1998.[6] A.Hilton.Towards model-based capture of a person’s shape,appearance and motion.IEEE International Workshop on Modelling People ,1999.[7]H.J Lee and Z.Chen.Determination of 3d human bodyposture from a single puter Vision,Graphics and Image Processing ,30,1985.[8]T.Moeslund and E.Granum.A survey of computer visionbased human motion puter Vision and Image Understanding ,81(3),March 2001.[9]R.Rosales and S.Sclaroff.Specialized mappings and theestimation of human body pose from a single image.IEEE Workshop on Human Motion ,pages 19–24,2000.[10]R.Rosales,M.Siddiqui,J.Alon,and S.Sclaroff.Estimating3d body pose using uncalibrated cameras.Technical Report 2001-008,Dept.of Computer Science,Boston University,2001.[11] C.Sminchisescu and B Triggs.Kinematic jump processesfor monocular 3d human tracking.Proc.IEEE Conference on Computer Vision and Pattern Recognition ,2003.[12]P.F.Stiller,C.A.Asmuth,and C.S.Wan.Invariant indexingand single view recognition.Proc.DARPA Image Under-standing Workshop ,pages 1423–1428,1994.[13] C.Taylor.Reconstructions of articulated objects from pointcorrespondences in a single puter Vision and Image Understanding ,80(3),2000.[14]L.Wang,W.Hu,and T.Tan.Recent developments in humanmotion analysis.Pattern Recognition ,36(3):585–601,March 2003.[15]I.Weiss and M.Ray.Model-based recognition of 3d objectsfrom single images.IEEE Trans.on Pattern Analysis and Machine Intelligence ,23,February 2001.[16]V .M.Zatsiorsky.Kinetics of Human Motion .Human Kinet-ics,Champaign,IL,2002.。