An Effective Hybrid Depth-Generation Algorithm for

An Effective Hybrid Depth-Generation Algorithm for 2D-to-3D Conversion in3D Displays Yeong-Kang Lai,Member,IEEE,Yu-Fan Lai,Student Member,IEEE,and Ying-Chang ChenAbstract—In recent years,3D display technology has been re-ceiving increasingly more attention.The most intuitive3D method is to use two temporally synchronized video streams for the left and right eyes,respectively.However,traditional2D video con-tents are captured by one camera,and in order to synthesize the left and right views as the behavior of two cameras,depth map information is required.In this paper,we propose a hybrid algo-rithm for2D-to-3D conversion in3D displays;it is a good way to solve the problem of traditional2D video contents which need to generate3D effects in3D displays.We choose three depth cues for depth estimation:motion information,linear perspective,and tex-ture characteristics.Moreover,we adopt a bilateralfilter for depth map smoothing and noise removal.From the experimental results, execution time can be reduced by25%–35%and the depth per-ception score is between75and85.Thus,the human eye cannot sense the noticeable differences from thefinal3D rendering.Fur-thermore,it is very suitable to apply our proposed hybrid algo-rithm to2D-to-3D conversion in3D displays.Index Terms—2D-to-3D conversion,3D-TV,depth map estimation.I.I NTRODUCTIONW ITH recent advances in technology,3D video tech-nology has become possible,and very attractive for display systems.There are many3D products in consumer electronics such as:3D cameras,3D projectors,3D mobile devices,and3D displays.However,there is an intense demand for converting conventional2D video contents to3D displays. Depth map estimation is a critical issue in2D-to-3D conver-sion.If the depth map is captured by a3D-depth range camera, it can produce perfect depth information[1].Unfortunately, most conventional2D video contents do not have real depth information.Thus,developing a method for estimating depth map which is close to real depth map has become of particular interest[2]–[5],[10].Depth-Image-Based Rendering(DIBR) can combine the depth map information with original2D images,and simultaneously output3D rendering for the left and right eyes.3D Display technology is mainly divided into two types[8]. One is stereoscopic display[9],requiring to wear a glasses.TheManuscript received July22,2012;accepted October04,2012.Date of pub-lication November27,2012;date of current version February11,2013.This work was supported in part by the National Science Council,Republic of China, under Grant NSC100-2220-E-005-003,and in part by the Ministry of Educa-tion,Taiwan,R.O.C.,under the ATU plan.The authors are with the Department of Electrical Engineering,National Chung Hsing University,Taichung402,Taiwan(e-mail:yklai@dragon.nchu. edu.tw;d9564504@.tw).Color versions of one or more of thefigures are available online at http:// .Digital Object Identifier10.1109/JDT.2012.2224637Fig.1.Dataflow of2D-to-3D conversion.other type is the auto-stereoscopic display,which does not re-quire to wear any glasses[11],[12].Because of the parallaxof human eyes,when the two view images are simultaneouslysent into the left and right eye,respectively,3D vision is gen-erated.This paper focuses on stereoscopic display technologywith2D-to-3D conversion which generates depth map infor-mation from2D videofirst,and then add the video contents tocreate binocular images[13]–[15].Due to the enormous numberof existing2D videos,2D-to-3D video conversion plays an im-portant role in3D content production.Fig.1shows the dataflowof2D-to-3D conversion.We adopt the human visual perceptionto visually estimate comfortable depth map information.Theproposed hybrid depth-generation algorithm can generate com-fortable and realistic depth cues,which are very close to truedepth maps.Finally,3D left/right views can be synthesized viathe DIBR method.The rest of this paper is organized as follows:the hybriddepth-generation algorithm is proposed in Section II,and theexperimental results are summarized in Section III.Finally,abrief conclusion is given in Section IV.II.P ROPOSED A LGORITHMIn order to generate more realistic depth maps,this paperuses three depth cues for depth estimation:motion information,linear perspective,and texture characteristics.Fig.2shows theblock diagram of the hybrid depth-generation algorithm.Firstly,we adopt the temporal difference of adjacent frames to distin-guish between foreground and background.Secondly,vanishinglines(VL)or vanishing points(VP)are used tofind the farthestborder or point,and then gradient depth values are assigned[16]–[18].Thirdly,different luminance information of images 1551-319X/$31.00©2012IEEEFig.2.Block diagram of hybrid depth-perception algorithm.is selected to obtain the depth values of texture properties.The detailed description and formulae are presented as follows: A.Motion InformationViewers tend tofixate on moving objects;we use the tem-poral difference method to update background information and to obtain the foreground objects.The formulas are as follows:(1) where is the current frame pixel at location,and is the pixel at location in the previous frame. The differences of frames are calculated.Then,the difference values of pixels are compared with the Threshold value:if(Threshold value)Otherwisewhere(2) where is standard deviation of the sample items(per one frame),and are the number of the total pixels and the mean of total pixel values in a frame,respectively.According to(1) and(2),we canfind the contours of moving objects andfill them in according to minimum and maximum coordinate informa-tion.The pixel value range in depth mapis from0to255.The value of0represents the farthest level(background),and the value of255represents the nearest level(foreground).Fig.3.Depth map assignment of foreground.In order to enhance the perceived depth of moving objects, we set moving objects as foreground and the others as back-ground.Fig.3shows the experimental results of the foreground assignment.B.Linear PerspectiveFirst,we adopt a Sobel operator for detecting the vanishing lines and vanishing point.Fig.4shows their horizontal and vertical matrices.Moreover,Fig.5shows the results after per-forming the Sobel operations.The formula is defined as follows:(3) where and are two images which at each point contain the horizontal and vertical derivative approximations,andis the gradient magnitude.Second,from the viewpoint of theFig.4.Sobel operators for horizontal and verticalmatrices.Fig.5.Borderdetection.Fig.6.Concept of vanishing lines and vanishing points.scene,the most distant line of the image is usually the skyline.That is,it is the farthest horizontal line of the image.We detect the VL and the VP to find the farthest line and point.However,the computational complexity of the traditional Hough trans-form algorithm [20]is too irregular to be used in hardware im-plementation.Therefore,we simplify the vanishing line algo-rithm as follows:(4)where and are coordinate values.When finding the inter-sections of horizontal and vertical lines in (4),we can accumu-late the line information and spread the gradient values from the depth map.Fig.6shows the concept of the vanishing line and the vanishing point.The value of 0represents the farthest re-gion (background),and the value of 255represents the nearest region (foreground).The detailed formulae are as follows:Case 1)If only a horizontal VL exists,the depth gradientassignment is represented as follows:(5)Case 2)If there are both horizontal VLs and vertical VLswhere the vertical VL belongs to the right-hand side,the depth gradient assignment has four re-gions,which are represented as follows:Upper Right region:(6)Upper Left region:the farthest regionLower Right region:the farthest regionLower Left region:Case 3)If there are both horizontal VLs and vertical VLswhere the vertical VL belongs to the left side,the depth gradient assignment has four regions which are represented as follows:Upper Left region:(7)Upper Right region:Fig.7.Example of linear perspective.Lower Left region:Lower Right region:From the above three cases,and represent the length and width of the image,respectively,and is the co-ordinate of the vanishing point (VP).In such a depth region,the value 0represents the farthest 3D point,and the value 255rep-resents the nearest 3D point.Fig.7shows the proven result of linear perspective.C.Texture CharacteristicsFrom some realistic examples,the texture characteristics of the object should be considered into the depth map information.Psychologists have discovered the human perception priorities of brightness,hue,and chrominance [19],[20].The human eye is sensitive to the difference between background brightness and the brightness of an object.We adopt the property of bright-ness contrast and transform color images into gray-scale images which represents the texture characteristics of the depth map.The values of the depth map are from 0to 255.Fig.8shows the experimental results.Luminance information is calculated as follows:(8)whereis luminance,andandare chrominance.D.Merging the Depth MapWe choose three depth cues for depth estimation:motioninformation,linear perspective,and texture characteristics.To convert from 2D to 3D,regions in the images are separated intoFig.8.Example of texture characteristics.two types:foreground and background.The final depth map assignment is represented as follows:Case 1:The objects which belong to the foreground.Case 2:The objects which belong to the background.where ,,)and represent weights of motion information,linear perspective,and texture characteristics,respectively.The weight values are determined by a subjective test from human visual system [21].For case 1,because moving objects attract more attention from human visual system,these objects will be regarded as foreground and set higher weight value ()in our implementation.Relatively,the weight of the linear perspective is set to lower gradient depth weight value ()for moving objects.For case 2,the static objects are considered as background information from the linear perspective and the texture information.Since the linear perspective dominates the gradient depth assignment for depth generation in static objects and background [16],[17],we set and as our experimental parameters.How-ever,these weighting factors can be adaptively adjusted.The recommended weights for merging depth are mainly related to the content analysis.E.Smooth Depth MapIn order to achieve better depth map generation,a bilateral filter is used to have smooth depth;even,there are many avail-able filters such as low pass filter,median filter,and gauss filter.The function of a bilateral filter is similar to that of a gauss filter.However,it is more accurate in proportion to the distance of rel-ative pixels.The formula of a bilateral filter is shown as follows:Fig.9.Experimental results of different sequences:(a)Poznan Street(19201088);(b)Love bird1(1024768);(c)Book Arrival(1024768);(d)Ballet (1024768);(e)Toy(960540);and(f)Akko&Kayo(640480).where is the normalized coefficient value;and are standard deviations of spatial domain and intensity domain, respectively.III.E XPERIMENTAL R ESULTSThe purpose of DIBR is to generate3D left and right virtual views from the intermediate view which comes from the orig-inal2D view image.DIBR consists of two parts:3D warping and inpainting(holefilling).The original image points at loca-tion are transferred to left points and right points as follows:and(9)where is the focal length of the camera,and is the distance between the left and right virtual cameras.Moreover,repre-sents the depth value of each pixel in the intermediate view.And it can be calculated out from the following equation.(10) where is the depth level(0–255)in the depth map,and, are the minimum and maximum depth values in a frame, respectively.According to(9)and(10),the left and right virtual views can be calculated via DIBR techniques.The above two equations are derived from[22],[23],the concept of3D warping is from the2D camera image plane to the3D coordinate plane,Fig.10.Subjective test results.TABLE IA VERAGE E XECUTION T IME P ER FRAMEand the depth cue must be considered in order to render 3D contents in 3D displays.The experimental platform is a PC with Intel Core Duo 1.73GHz and 4GB RAM.We select several test sequences such as Poznan Street (19201088),Love bird1(1024768),Book Arrival (1024768),Ballet (1024768),Toy (960540),and Akko&Kayo (640480).Table I shows the average execution time of the various test sequences with different sequences.From the experimental results,the proposed hybrid depth-gen-eration algorithm can save the execution time by 25%–35%,compared with Depth Estimation Reference Software [24].In Fig.9,in order to evaluate the performance of our proposed algorithm,most of these sequences originally have a true depth map for comparison.However,there are some sequences which do not have true depth maps.Because we adopt human visual perceptions (motion information,linear perspective,and texture characteristics)to estimate the depth map,most of the results are very similar to true depth maps.However,there are some errors in non-moving objects without any motion information such as the desk in Fig.9(c)and the coach in Fig.9(d).It is dif ficult to contour non-moving objects which are completely still,and they will be regarded as background information.Table II shows the average quality score of the depth map compared with the original (true depth map),and the scoresTABLE IIA VERAGE Q UALITY S CORE (0–100)OF D EPTH MAPFig.11.3D rendering for book arrival sequence (red-cyanglasses).Fig.12.3D rendering for Poznan street sequence (active shutter glasses).are between 75and 85(The score range is from 0to 100).The subjective test [25]–[27]is conducted to show the 3D effects by wearing red-cyan glasses or active shutter glasses.In order to evaluate the quality of 3D video,ten participants were invited to take part in this experiment.The experimental environment was setup in a dark room.The sequence was randomly selected to avoid the same content appearing continuously.Fig.10shows the subjective test results of our algorithm combined with DIBR technology;the average score of different test sequences is 83(The score range is from 0to 100).We have added the error bars in each testing image.They represent 95%con fidence interval (CI),and the corresponding values of CIare0.924,0.826,0.877,0.653,0.826,and0.716,respectively. This means that the distribution is concentrated.The depth map estimation is the most important issue in3D rendering. If the depth map is very close to the true depth,it will get a higher score.Figs.11and12show the3D rendering results while wearing red-cyan glasses or active shutter glasses after performing DIBR.From the experimental results,the proposed hybrid depth-generation algorithm demonstrates very good 2D-to-3D conversion capability.IV.C ONCLUSIONIn this paper,we proposed an efficient hybrid depth-genera-tion algorithm for2D-to-3D conversion in3D display systems. In order to generate better depth maps,this paper uses three depth cues for depth estimation:motion information,linear per-spective,and texture characteristics.From the experimental re-sults,the assignment of depth maps is very close to the true depth maps of test sequences.Moreover,the proposed algorithm can produce excellent3D stereoscopic video effect using red-cyan glasses or active shutter glasses.Therefore,our method is very effective for2D-to-3D conversion in3D displays.In the fu-ture,we will implement real hardware architecture on the FPGA platform.A CKNOWLEDGMENTThe authors thank and acknowledge the support of the re-lated EDA tools and platform by National Chip Implementation Center(CIC).R EFERENCES[1]C.Fehn and R.S.Pastoor,“Interactive3-DTV—Concepts and keytechnologies,”Proc.IEEE,vol.94,no.3,pp.524–538,Mar.2006.[2]C.H.Wu,M.McCormick,A.Aggoun,and S.Y.Kung,“Depth map-ping of integral images through viewpoint image extraction with a hy-brid disparity analysis algorithm,”J.Display Technol.,vol.4,no.1,pp.437–450,Mar.2008.[3]D.-S.Kim,S.-S.Lee,and B.-H.Choi,“A real-time stereo depth ex-traction hardware for intelligent home assistant robot,”IEEE Trans.Consum.Electron.,vol.56,no.3,pp.1782–1788,Aug.2010.[4]M.T.Pourazad,P.Nasiopoulos,and R.K.Ward,“An H.264-basedscheme for2D to3D video conversion,”IEEE Trans.Consum.Elec-tron.,vol.55,no.2,pp.742–748,May2009.[5]C.-C.Cheng,C.-T.Li,and L.-G.Chen,“A novel2D-to-3D conversionsystem using edge information,”IEEE Trans.Consum.Electron.,vol.56,no.3,pp.1739–1745,Aug.2010.[6]B.Kamolrat,W.A.C.Fernando,M.Mrak,and A.Kondoz,“3D motionestimation for depth image coding in3D video coding,”IEEE Trans.Consum.Electron.,vol.55,no.2,pp.824–830,May2009.[7]L.Zhang and W.J.Tam,“Stereoscopic image generation based ondepth images for3D TV,”IEEE Trans.Broadcast.,vol.51,no.2,pp.191–199,Jun.2005.[8]J.Geng,“V olumetric3D display for radiation therapy planning,”J.Display Technol.,vol.4,no.4,pp.437–450,Dec.2008.[9]R.Zwing,S.Weitbruch,and G.Lübcke,“Evolution of modern displaytechnologies in a3D ecosystem,”J.Display Technol.,vol.7,no.10,pp.572–578,Oct.2011.[10]Y.S.Hwang,S.-H.Hong,and B.Javidi,“Free view3-D visualizationof occluded objects by using computational synthetic aperture integralimaging,”J.Display Technol.,vol.3,no.1,pp.64–70,Mar.2007.[11]J.-C.Liou,K.Lee,and J.-F.Huang,“Low crosstalk multi-viewtracking3-D display of synchro-signal LED scanning backlightsystem,”J.Display Technol.,vol.7,no.8,pp.411–419,Aug.2011.[12]J.-Y.Son,B.Javidi,S.Yano,and K.-H.Choi,“Recent developmentsin3-D imaging technologies,”J.Display Technol.,vol.6,no.10,pp.394–403,Oct.2010.[13]Z.-W.Gao,W.-K.Lin,Y.-S.Shen,C.-Y.Lin,and W.-C.Kao,“Designof signal processing pipeline for stereoscopic cameras,”IEEE Trans.Consum.Electron.,vol.56,no.2,pp.324–331,May2010.[14]Y.Feng,J.Ren,and J.Jiang,“Object-based2D-to-3D video conver-sion for effective stereoscopic content generation in3D-TV applica-tions,”IEEE Trans.Broadcast.,vol.57,no.2,pp.500–509,Jun.2011.[15]V.V.Saveljev,J.-Y.Son,S.-H.Kim,D.-S.Kim,M.-C.Park,andY.-C.Song,“Image mixing in multiview three-dimensional imagingsystems,”J.Display Technol.,vol.4,no.3,Sep.2008.[16]A.Almansa,A.Desolneux,and S.Vamech,“Vanishing point detectionwithout any a priori information,”IEEE Trans.Pattern Anal.Mach.Intell.,vol.25,pp.502–507,Apr.2003.[17]T.Yi-Min,Y.L.Chang,and L.G.Chen,“Block-based vanishing lineand vanishing point detection for3D scene reconstruction,”in IEEEInt.Symp.on Intelligent Signal Processing and Commun.,Dec.2006,pp.586–589.[18]S.Battiato,A.Capra,S.Curti,and L.M.Cascia,“3D stereoscopicimage pairs by depth-map generation,”in IEEE Int.Symp.on3D DataProcessing,Visualization and Transmission,Sep.2004,pp.124–131.[19]K.Ghosh and S.K.Pal,“Some insights into brightness perception ofimages in the light of a new computational model offigure-ground seg-regation,”IEEE Trans.Syst.,Man,Cybern.A,Syst.,Humans,vol.40,no.4,pp.758–766,Jul.2010.[20]M.Song,D.Tao,C.Chen,X.Li,and C.W.Chen,“Color to gray:Visual cue preservation,”IEEE Trans.Pattern Anal.Mach.Intell.,vol.32,no.9,pp.1537–1552,Sep.2010.[21]A.Boev, D.Hollosi, A.Gotchev,and K.Egiazarian,“Classifi-cation and simulation of stereoscopic artifacts in mobile3DTVcontent,”in Proc.SPIE,Stereoscopic Disp.Appl.,2009,vol.7237,pp.72371F-1–72371F-12.[22]W.J.Tam and L.Zhang,“3D-TV content generation:2D-to-3D con-version,”in IEEE Int.Conf.on Multimedia and Expo,Jul.2006,pp.1869–1872.[23]Y.-R.Horng,Y.-C.Tseng,and T.-S.Chang,“VLSI architecture forreal-time HD1080p view synthesis engine,”IEEE Trans.Circuits Syst.Video Technol.,vol.21,no.9,pp.1329–1340,Sep.2011.[24]Depth Estimation Reference Software,[Online].Available:/svn/repos/MPEG-4/test/tags/3D/depth–esti-mation[25]S.-F.Tsai,C.-C.Cheng,C.-T.Li,and L.-G.Chen,“A real-time1080p2D-to-3D video conversion system,”IEEE Trans.Consum.Electron.,vol.57,no.2,pp.915–922,May2011.[26]C.-A.Chien,C.-Y.Chang,J.-S.Lee,J.-H.Chang,and J.-I.Guo,“Lowcomplexity3D depth map generation for stereo applications,”in IEEEInt.Conf.on Consumer Electron.(ICCE),Jan.2011,pp.185–186.[27]C.-L.Su,K.-N.Pang,T.-M.Chen,G.-S.Wu,C.-L.Chiang,H.-R.Wen,L.-S.Huang,Y.-H.Hsueh,and S.-Y.Tseng,“A real-timefull-HD2D-to-3D conversion system using multicore technology,”in IEEE Int.Conf.on Multimedia and Ubiquitous Eng.(MUE),Jun.2011,pp.273–276.Yeong-Kang Lai(M’94)was born in Taipei,Taiwan,in1966.He received the B.S.degree in electricalengineering from the Tamkang University,Taipei,Taiwan,in1988,and the M.S.and Ph.D.degreesfrom the National Taiwan University,in1990and1997,respectively.From1992to1993,he was with the Institute of In-formation Science,Academia Sinica,Taiwan,wherehe worked on video conference system.In1997,he was with Department of Electrical Engineering,Chang Gung University,Taoyuan,Taiwan,as an Assistant Professor.From1998to2001,he was with Department of Computer Science and Information Engineering,National Dong Hwa University,Hualien, Taiwan,as an Assistant Professor.Since2001,he has been with National Chung Hsing University,Taichung,Taiwan.Currently,he is with the Department of Electrical Engineering,National Chung Hsing University as a Professor. His research interests include3D display,3D video,video compression,DSP architecture design,video signal processor design,and VLSI signal processing.i is a member of Phi Tau Phi.In2011,he received the Outstanding Teaching Professor Award of National Chung Hsing University.In2010,he also received the Best Paper Award of the International SoC Design Conference. Currently,he is an Associate Editor for IEEE T RANSACTIONS ON C ONSUMER E LECTRONICS.He is also a member of the Technical Program Committee ofIEEE International Conference on ConsumerElectronics.Yu-Fan Lai(S’06)was born in Taichung,Taiwan,onJune14,1978.He received the B.S.degree in auto-matic control engineering from the Feng Chia Uni-versity,Taichung,Taiwan,in2000,and the M.S.de-gree in electrical engineering from the Chung HwaUniversity,Hsinchu,Taiwan,in2003.From2003to2007,he ever worked in Ritek Cor-poration,Hsinchu,Taiwan.He is currently pursuingthe Ph.D.degree in the department of electrical en-gineering at National Chung Hsing University.Hismajor research interests include3D video processing, VLSI architecture design of image and video coding,and VLSI design for dig-ital signalprocessing.Ying-Chang Chen was born in Chiayi,Taiwan,on August28,1986.He received the B.S.degree in elec-tronic engineering from the National University of Kaohsiung,Kaohsiung City,Taiwan,in2009,and the M.S.degree in electrical engineering from the Na-tional Chung Hsing University,Taiwan,in2011. His major research interests include video pro-cessing,VLSI architecture design of video coding, and image processing.。