Sequential Construction of 3-D-Based Scene Description

Sequential Construction of3-D-Based SceneDescriptionChun-Jen Tsai and Aggelos K.Katsaggelos,Fellow,IEEEAbstract—Binocular camera systems are commonly used to con-struct3-D-based scene description.However,there is a tradeoff between the length of the camera baseline and the difficulty of the matching problem and the extent of the field of view of the 3-D scene.A large baseline system provides better depth resolution than a smaller baseline system at the expense of a narrower field of view.To increase the depth resolution without increasing the diffi-culty of the matching problem and decreasing the field of view of the3-D scene,a sequential3-D-based scene description technique is proposed in this paper.Multiple small-baseline3-D scene de-scriptions from a single moving camera or an array of cameras are used to sequentially construct a large baseline3-D scene descrip-tion while maintaining the field of view of a small-baseline system.A Bayesian framework using a disparity-space image(DSI)tech-nique for disparity estimation is presented.The cost function for large baseline image matching is designed based not only on the photometric matching error,the smoothness constraint,and the ordering constraint,but also on the previous disparity estimates from smaller baseline stereo image pairs as a prior model.Texture information is registered along the scan path of the camera(s).Ex-perimental results demonstrate the effectiveness of this technique in visual communication applications.Index Terms—Disparity estimation,divide-and-conquer,se-quential3-D model reconstruction,3-D based scene description.I.I NTRODUCTIONT HE CONSTRUCTION of a3-D-based scene description is an important topic in visual communications.A3-D-based scene description consists of two parts:the3-D shape of the scene and the texture on the3-D shape.It can be used,for ex-ample,for multiview image generation for object movie[1], [2],intermediate view image generation for stereo TV[3],[4], or hybrid natural and synthetic image composition and coding [5]–[7].One of the most popular approaches to create a3-D-based scene description is to use a binocular camera system arranged in parallel-axis configuration to acquire stereo image pairs[9],[10].If the cameras are accurately calibrated such that the intrinsic and extrinsic camera parameters are available,a precise surface model of the scene can be reconstructed using triangulation[11].If the cameras are weakly calibrated(for ex-ample,arranged in a parallel-axis configuration),a disparity map can be computed to provide depth information of the scene [12].In either case,a large camera baseline is typically used in order to obtain high depth resolutions.However,stereoManuscript received March15,1999;revised September30,1999.This paper was recommended by Guest Editor M.G.Strintzis.The authors are with the Electrical and Computer Engineering Department, Northwestern University,Evanston,IL60208USA(e-mail:tsai@; aggk@).Publisher Item Identifier S1051-8215(00)04886-2.Fig.1.The single moving camera setup.image matching is a well-known hard problem when thebaseline is large.In making the problem more manageable,multiple-camera systems have been proposed[13]–[15].Evenin this case,the cameras are typically arranged in a2-Dlayout(as a triangular or rectangular array)and the baselinesbetween any two cameras are usually large.Although existingmultiple-camera systems alleviate the problem of occlusion,they do not reduce the difficulty of the matching problem.Toimmensely lessen the difficulty,a very small-baseline camerasystem should be used.In the extreme case,if the lengthof the camera baseline is close to zero,the search space ofthe matching process can be confined to a couple of pixels.Unfortunately,the depth resolution would be very coarse inthis case.To reduce the difficulty of the image matching problemwhile maintaining large depth resolution,a sequence of im-ages can be used,for which the baseline between any twoconsecutive images is small(Fig.1).In this case,the solu-tion of the image matching problem becomes easier to obtainand it is more accurate.However,the collective baseline be-tween the first and last images is large;thus,high depthresolution is maintained.In this paper,we propose a Bayesian approach which com-bines this framework with a divide-and-conquer disparity spaceimage(DSI)technique[12]for3-D-based scene descriptionconstruction.In the proposed algorithm,the matching processis based not only on the photometric similarity,the smooth-ness constraint,and the ordering constraint,but also on the pre-vious estimates from smaller baseline stereo image pairs,whichdefine the prior model.This prior model is used in both thedivide-and-conquer stage and the DSI dynamic programming(DP)stage of the disparity estimator to solve for larger baselineestimates.Texture information is also registered along the scan 1051–8215/00$10.00©2000IEEEFig.2.Camera model(the shaded part is the image plane(the CCD chip)of the camera).path of the camera.As a result,the3-D-based scene description between the first and the last images is computed recursively based on earlier estimates with smaller baselines(prior model) and the most recent image data(observations).There are some existing results which take into account the nonuniform nature of the probability density function of depth estimates.In[16]and[17],an integrated sequential image matching and3-D surface modeling technique is presented.The 3-D scene is modeled by a number of parameterized surface patches.A sequence of calibrated images is then used as the observation for the maximum a posteriori(MAP)estimation of the surface patch parameters.In[18],the problem of finding corresponding points is formulated as a hypothesis verification problem.The triangulation process is posed as a Bayesian estimation problem,taking into account the uncertainties in the observed image points and the camera parameters.In[19],a block-based disparity estimator is proposed that considers the nonuniform spatial distribution of the estimation error inside a left image block that is mapped into the right image plane using projective2-D transformations.As a consequence,the disparity estimator arranges the image block eccentrically around the pixel to be evaluated.The paper is organized as follows.In Section II,the problem of the construction of a3-D-based scene description is intro-duced.In this section,the camera model and some geometric constraints for disparity estimation are described.The image model and the formulation of the maximum-likelihood(ML) disparity estimation problem are also introduced.Section III presents the framework for the sequential construction of a3-D-based scene description.Section IV presents the actual imple-mentation of the framework of Section III using a divide-and-conquer DSI-based matching technique which incorporates the previous smaller baseline estimates as prior knowledge.Exper-iments and implementation details are presented in Section V, while conclusions are given in Section VI.II.P ROBLEM F ORMULATIONA.Camera Model and3-D-Based Scene DescriptionTo acquire a3-D-based scene description,a common setup is to use the binocular camera system,shown in Fig.2[9].If the geometry(position and orientation)and intrinsic parame-ters(focal lengths,dimensions of the CCD sensor array,and piercing points of the optical axes)of the cameras are known, the3-D model can be constructed by first solving the correspon-dence problem between the left and right image points,then computing the depth map using geometrical triangulation,and finally fitting a3-D surface model to the depth map[11].The correspondence between the scene texture and the3-D model is established during the solution of the image correspondence problem.For many visual communication applications,an exact 3-D model is not required.For example,for synthetic scene gen-eration,if the disparity field between the two video channels is obtained,virtual scenes can be synthesized by disparity-com-pensated interpolation[3],[8],[20].When the converging angle of the camera pair approaches zero,the binocular camera system becomes a parallel-axis camera system(Fig.3).In this case,all left-image based disparities are negative,and the search for corresponding image points can be confined to searching along one direction on the epipolar line.For a parallel-axis binocular camera system,the larger the camera baseline,the higher the3-D(depth)resolution of the disparity map.On the other hand,the larger the camera baseline, the harder the matching problem.This is clear since the search range required for a small-baseline setup is smaller than that of a large baseline setup.B.ML Disparity EstimationIn a binocular camera system,a video sequence consists of two channels,the left and the right channel.There are a numberFig.3.Pparallel binocular camera system.All (left image-based)disparities are negative.of sources of degradations before the 2-D projections of the scene are captured in digital format.The first major degradation comes from the lens systems of the cameras.Camera lenses in-troduce blurs,which can be modeled by point spread functions applied on the projected D sensors in the cameras bring in another source of degradation.Each pixel in the CCD sensors can be modeled by a random variable,resulting in each observed image being a realization of a 2-D random process.In the following,we show that the solution of the disparity-es-timation problem does not require knowledge of the lens degra-dation model.Letand be the true projections of a 3-D scene to the left and right image planes,respectively,and the spatially invariant point spread functions that characterize the degradation from the left and the right cameralenses,andand the additive noises introduced by the camera sensors.An observed image intensitypairand is therefore modeledby(1)whereandareassumed to be white Gaussian noises.The disparityfield(2)Assuming that the degradations from the two lenses are sim-ilar,and can be approximated by a temporally invariantfunction(3)wehave.Furthermore,assuming,wehave(5)wherethe variance of the additive noise.Therefore,the ML estimateof(6)C.Field-of-View ProblemIn a parallel-axis binocular camera system,only the right part of the left image and the left part of the right image are avail-able for 3-D scene description (Fig.4),representing the field of view of the binocular camera system.There is a tradeoff be-tween the length of the baseline and the size of the field of view.For high depth resolution,a large baseline is required.However,the larger the baseline,the smaller the field of view.To achieve high resolution without sacrificing the size of the field of view,more than two camera images are required.III.S EQUENTIAL C ONSTRUCTION OF 3-D–B ASED S CENED ESCRIPTION To solve the problems with binocular camera systems men-tioned in Section II,a framework for the sequential construc-tion of a 3-D-based scene description is proposed.As shown in Fig.1,a single camera is translated horizontally and images aretaken along its path of translation.Letdenote the disparity field between images.Four disparityfields,,,,and are shown in Fig.1.Among these fourfields,,,and are small-baseline fields with low depthresolution,andis a large baseline field with high depth res-olution.Whileis more desirable as far as depth resolution is concerned,it is considerably harder to estimate itthan,,and .Another problemwith is that the field ofview is smaller than the joint field of viewof,,and .Therefore,it would be advantageous touse,,andas prior knowledge for the estimationof .This is accomplished with a recursive Bayesian algorithm proposed later in this section.Fig.5.3-D scene description from three images E,and E,T\T,D, respectively.The setup in Fig.1can only be used for static scene analysis. When there are moving objects in the scene,an array of cam-eras should be used.The proposed framework for the sequential construction of3-D-based scene description is described next. In the beginning,only twoimages,as i.i.d.Gaussian,thatisis givenbyis estimated following the approachin[26].Let,in general,the sum of squared difference(SSD)matching-error function used for the estimation of the disparityat somepoint(10)whereand,the variancesrespectivelyofand,which we also modeledas Gaussian.Ignoring the correlationbetweenand,we havethat,whereand are,respectively,the curvatures of the parabolasfitted to the corresponding error functionsatandisgivenby(11)whereimages.Let usassume that the system has alreadyprocessed imagesandhas been computed.Whentheis estimated based on(6)and is estimated bymaximizingis approximated by a Gaussiandistribution(14)where,,andis therefore givenby(15)whereandcan be used to reduce the search range and hence eliminate falsetargets for the estimationof.Another advantage of theDSI-based technique is that it is very powerful in detecting oc-clusions[21],[23].In addition to occlusions within the field ofview,it is also important for the proposed approach to identifythe part of the left image that is invisible to the right camera andthe part of the right image which is invisible to the left camera.This can be done in a straightforward way using a DSI-basedtechnique.In a nutshell,the proposed framework is composed of twostages.The divide-and-conquer stage is used to split the fullintra-scanline matching problem into nonoverlapping smallersub-problems using feature matching.For each sub-problemwhere there is no reliable matching feature between the left andright scanlines,possibly due to occlusions or lack of textureinformation,a DSI technique is used to perform dense pixelmatching and occlusion detection.A.Divide-and-Conquer StageWith a pair of parallel-axis camera images,the correspon-dence problem becomes an intra-scanline matching problem.Furthermore,because of the ordering constraint[22],the intra-scanline matching problem can be solved using the divide-and-conquer technique[12].The divide-and-conquer stage is usedto split the intra-scanline matching problem into nonoverlap-ping smaller sub-problems using feature matching(please referto[12]for details).Two measures are used to compute the re-liability of the match.The first one is the signal-variance-to-matching-error ratio(SER)and the second one is the variancedifference(VD)between the left feature point and the right fea-ture point.They are defined,respectively,bySERis the window used for the matching processand100in the experiments),the matching pair is considered unreliable.The search for the featurepoint in the leftscanline is centeredatDSIis the matching window size.Theestimation of disparity between these two scanlines is equivalentto finding a path across a DSI starting at the lower-left cornerand ending at the upper-right corner which has the smallest cost.This minimal cost path can be found using a DP technique[23].Note that(19)is a likelihood function based on(6).Accordingto(15),the prior term is incorporated asfollows:(20)with(21)is the horizontal disparity between images onscanline.The computation is per-formed starting at the lower-left corner and terminating at theupper-right corner of the DSI.At each entry(node)in the costmatrix,certain steps that may lead to this node are considered.In Fig.6,five possible steps are considered during the cost eval-uation.There are clearly other possibilities which are not con-sidered here([25],[12]).According to Fig.6,the cost functionis definedby(22)whereare pixel coordinates.The definition of the cost termsfor stepsand at640pixels.There is no special arrangement oflights and no special treatment of shadows and background lu-minance changes from frame to frame,which are common inpractical applications.The first sequence is composed of four images.Each imageis low-pass filtered and subsampled by a factor of two to re-duce the noise from the JPEG compression built into the camerahardware.The center part(178.This is equivalent to a40mm-baselinesystem when the object is2-m away from the camera.The firstThe small-baseline disparity maps are shown in Fig.8(white pixels indicates occluded areas or areas not in the fieldof view).It is clear that when the camera baseline is small,resolution is low.The actual depth of the toy object is about40mm and the disparity difference between the backgroundand the foremost point of the toy object is only4pixels.Therefore,the average depth resolution is only10mm/pixel.The sequential reconstruction of the disparity map using(20)with scaled to the range[1.52.5]is shown in Fig.9.Ifnonlinear optimization techniques are used to minimize(11)(a)(b)(c)Fig.8.Disparity maps from consecutive small-baseline image pairs in the sequence.Since the depth resolution is low,the disparity difference between the foreground object and the background is small(around4pixels).The disparity maps are enhanced using histogram equalization.(a)D.(c)D.(b)D.Fig.10.Disparity map DDSDDD(a)(b)Fig.11.(a)First and(b)last images of the sequence.(a)(b)Fig.12.Transformed disparity maps DProc.ACM SIGGRAPH’96,1996,pp.43–54.[2]H.-C.Huang,S.-H.Nain,and Y.-P.Hung,“Generation of transitionviews for object movie,”in puter Symp.:Workshop onComputer Graphics and Virtual Reality,Dec.1998,pp.136–143.[3]J.S.McVeigh,M.W.Siegell,and A.G.Jordan,“Intermediate viewsynthesis considering occluded and ambiguously referenced image re-gions,”Signal Processing:Image Commun.,vol.9,pp.21–28,1996.[4]N.Grammalidis and M.G.Strintzis,“Disparity and occlusion estima-tion in multiocular systems and their coding for the communication of multiview image sequences,”IEEE Trans.Circuits Syst.Video Technol., vol.8,pp.328–344,June1998.[5]S.Malassiotis and M.G.Strintzis,“Object-based coding of stereo imagesequences using3D models,”IEEE Trans.Circuits Syst.Video Technol., vol.7,pp.892–906,Dec.1997.[6]K.Sengupta and J.Ohya,“Novel scene generation,merging andstitching views using the2D affine space,”Signal Processing:Image Commun.,vol.14,pp.39–53,Nov.1998.[7]T.Naemura,M.Kaneko,and H.Harashima,“3-D object based codingof multi-view images,”in Proc.Picture Coding Symp.’96,Mar.1996, pp.459–464.[8] A.Mansouri and J.Konrad,“Block-based winner-takes-all reconstruc-tion of intermediate stereoscopic images,”in Proc.SPIE Conf.Visual Communication and Image Processing,vol.3309,San Jose,CA,Jan.1998,pp.922–933.[9]J.-R.Ohm,K.Grüneberg,E.H.Hendriks,M.E.Izquierdo,D.Kalivas,M.Karl,D.Papadimatos,and A.Redert,“A realtime hardware system for stereoscopic videoconferencing with viewpoint adaptation,”Signal Processing:Image Commun.,vol.14,pp.147–171,Nov.1998. [10]S.Malassiotis and M.G.Strintzis,“Model based joint motion andstructure estimation from stereo images,”put.Vis.Image Understanding,vol.65,no.1,pp.79–94,January1997.[11]N.Ahuja and A.L.Abbott,“Active stereo:Integrating disparity,ver-gence,focus,aperture,and calibration for surface estimation,”IEEE Trans.Pattern Anal.Machine Intell.,vol.15,pp.1007–1029,1993.[12] C.-J.Tsai and A.K.Katsaggelos,“Dense disparity map estimation witha divide-and-conquer disparity-space image technique,”IEEE Trans.Multimedia,vol.1,pp.18–29,Mar.1999.[13]J.-L.Park and S.Inoue,“Acquisition of sharp depth map from multiplecameras,”Signal Processing:Image Commun.,vol.14,pp.7–19,Nov.1998.[14] F.Pedersini,P.Pigazzini,A.Sarti,and S.Tubaro,“3D area matchingwith arbitrary multiview geometry,”Signal Processing:Image Commun.,vol.14,pp.7–19,Nov.1998.[15]L.Falkenhagen,“Block-based depth estimation from image triples withunrestricted camera setup,”in Proc.IEEE Multimedia Signal Processing Workshop,Princeton,NJ,June23–25,1997.[16]Y.-P.Hung, D. B.Cooper,and B.Cernuschi-Frias,“AsymptoticBayesian surface estimation using an image sequence,”put.Vis.,vol.6,no.2,pp.105–132,1991.[17] C.-J.Tsai,Y.P.Hung,and S.C.Hsu,“Comparison of asymptoticBayesian and Kalman filter approach for surface reconstruction usinga sequence of images,”in Proc.IEEE puter Vision andPattern Recognition,New York,NY,June1993,pp.206–211.[18] A.S.Bedekar and R.M.Haralick,“A Bayesian method for triangulationand its application to finding corresponding points,”in IEEE Int.Conf.Image Processing,vol.II,1995,pp.362–365.[19]L.Falkenhagen and T.Wedi,“Improving block-based disparity estima-tion by considering the nonuniform distribution of the estimation error,”in Proc.Workshop3-D Structure from Multiple Images of Large-Scale Environments,June6–7,1998,pp.93–108.[20] B.L.Tseng and D.Anastassiou,“A theoretical study on an accuratereconstruction of multiview images based on the Viterbi algorithm,”in Proc.IEEE Int.Conf.Image Processing,vol.II,Washington,DC,Oct.1995,pp.378–381.[21]Y.Ohta and T.Kanade,“Stereo by intra-and inter-scanline search usingdynamic programming,”IEEE Trans.Pattern Anal.Machine Intell.,pp.139–154,Mar.1985.[22]S.S.Intille and A.F.Bobick,“Disparity-space images and largeocclusion stereo,”in Proc.ECCV’94,Stockholm,Sweden,1994,pp.179–186.[23]I.J.Cox,S.L.Hingorani,and S.B.Rao,“A maximum likelihoodstereo algorithm,”Comput.Vis.Image Understanding,vol.63,no.3, pp.542–567,Apr.1996.[24] E.Izquierdo,“Stereo matching for enhanced telepresence in3D-video-communications,”IEEE Trans.Circuits Syst.Video Technol.,vol.7,pp.629–643,Aug.1997.[25] A.Redert,C.-J.Tsai,E.Hendriks,and A.K.Katsaggelos,“Disparityestimation with modeling of occlusion and object orientation,”in Proc.SPIE Visual Communication and Image Processing,San Jose,CA,Jan.1998,pp.798–808.[26]L.Matthies,R.Szeliski,and T.Kanade,“Kalman filter-based algorithmsfor estimating depth from image sequences,”put.Vis.,vol.3, no.3,pp.209–238,Sept.1989.Chun-Jen Tsai received the B.S.degree in math-ematics from Fu-Jen Catholic University,Taiwan,R.O.C.,in1989,the M.S.degree in computer scienceand information engineering from National TaiwanUniversity,Taipei,Taiwan,in1992,and the Ph.D.degree in electrical engineering from NorthwesternUniversity,Evanston,IL,in1998.In1992,he became a Research Assistant inthe Institute of Information Science,AcademiaSinica,Taiwan.He was a Visiting Scientist inthe Telecommunications Institute,University of Erlangen–Nuremberg,Germany,from November1996to January1997.In 1999,he became a post-doctorial Research Fellow at the Department of Electrical and Computer Engineering,Northwestern.He is currently a senior member of Technical Staff of PacketVideo Corporation,San Diego,CA. His major research interests are in motion and disparity estimation in video sequences,surface reconstruction with passive sensors,depth-from-defocus, target tracking,multi-view image generation,compressed video restoration and enhancement,and video compression for wireless channel and embeddedsystems.Aggelos K.Katsaggelos(S’80–M’85–SM’92–F’98)received the Diploma degree in electrical and me-chanical engineering from the Aristotelian Univer-sity of Thessaloniki,Thessaloniki,Greece,in1979,and the M.S.and Ph.D.degrees,both in electrical en-gineering,from the Georgia Institute of Technology,Atlanta,GA,in1981and1985,respectively.In1985,he joined the Department of Electrical En-gineering and Computer Science,Northwestern Uni-versity,Evanston,IL,where he is currently a Pro-fessor,holding the Ameritech Chair of Information Technology.He is also the Director of the Motorola Center for Communica-tions,and a member of the Associate Staff,Department of Medicine,Evanston Hospital.During the1986–1987academic year,he was an Assistant Professor at Polytechnic University,Department of Electrical Engineering and Computer Science,Brooklyn,NY.His current research interests include image and video recovery,video compression,motion estimation,boundary encoding,compu-tational vision,and multimedia signal processing.He is the Editor of Digital Image Restoration(Heidelberg,Germany:Springer-Verlag,1991),Co-Author of Rate-Distortion Based Video Compression(Norwell,MA:Kluwer,1997), and Co-Editor of Recovery Techniques for Image and Video Compression and Transmission(Norwell,MA:Kluwer,1998).Dr.Katsaggelos is a member of the Board of Governors and the Publication Board of the IEEE Signal Processing Society,the IEEE TAB Magazine Committee,the Steering Committee of the IEEE T RANSACTIONS ON M EDICAL I MAGING,the IEEE Technical Committees on Visual Signal Processing and Communications,and Multimedia Signal Processing,and Editor-in-Chief of the IEEE Signal Processing Magazine.He has served as an Associate Editor for the IEEE T RANSACTIONS ON S IGNAL P ROCESSING(1990–1992),an Area Editor for Graphical Models and Image Processing(1992–1995),a member of the Steering Committee of the IEEE T RANSACTIONS ON I MAGE P ROCESSING (1992–1997),and the IEEE Technical Committee on Image and Multi-Dimen-sional Signal Processing(1992–1998).He has served as the General Chairman of the1994Visual Communications and Image Processing Conference,and as Technical Program Co-Chair of the1998IEEE International Conference on Image Processing.He is an Ameritech Fellow,a member of SPIE,and the recipient of the IEEE Third Millenium Medal.。