Warping techniques for light fields

in Proc.Grafiktag2000,Gesellschaft für Informatik(GI)Warping Techniques for Light FieldsHartmut SchirmacherMax-Planck-Institut für InformatikIm Stadtwald,66123Saarbrückenschirmacher@mpi-sb.mpg.de1IntroductionA lightfield[3,1]is a four-dimensional represen-tation of the light exiting from,or incident in,a bounded region of3D ually,a lightfield is parameterized by pairs of2D coordinates on two parallel planes.At least six such pairs of planes (called slabs)are needed to represent the complete light distribution around the desired region.All rays that pass through the same point on one of the planes build up a sheared perspective image on the other plane(cf.Fig.1).This is why the two planes are often distinguished as the eye plane or plane,and the image plane,or)plane. As sketched in Fig.1,the light along an arbi-trary ray passing through both planes can be recon-structed from the nearest neighbours through quadri-linear interpolation.This method assumes that3D points represented by the rays(e.g.a sur-face that reflects light)are located in the image plane,which is not true for the general case.This results in serious blurring and ghosting when view-ing the reconstructed lightfield.In order to avoid these artifacts,the lightfield must be very densely sampled,e.g.a large number of images must be acquired and stored.The Lumigraph[1]is a lightfield representa-tion that uses some additional geometric informa-tion,usually in the form of an approximate tri-angle mesh.In contrast to the“pure”lightfield approach,the Lumigraph reconstruction algorithm can take into account the actual location of points in the scene.By intersecting the desired viewingray with the approximate scene geometry a depth correction can be performed thatfinds those sam-ples wich best represent the actual scene object that is to be reconstructed.(s,t) planeFigure1:Sketch of a two plane-parameterized lightfield.The radiance along a ray can be inter-polated from the nearest neighbours on the two planes.The set of all rays passing through one eye point represent a sheared perspective image on the plane.There are two problems with the Lumigraph ap-proach.First,it is not always trivial to obtain an approximate triangle mesh from a real-world scene.Second,it is difficult to decide on the qual-ity and resolution of the geometric information.On one hand,when too few triangles are used to rep-resent the scene,the reconstruction will produce artifacts similar those in lightfield rendering.On the other hand,too many triangles slow down the reconstruction algorithm considerably.1We try to overcome these limitations by using per-pixel depth information instead of approximate triangle meshes.Based on this representation,we proposed a number of warping-based techniques for acquring,refining,and rendering Lumigraphs with per-pixel depth information,which will be sketched briefly in the remainder of this article.2Warping-based Refinement of LumigraphsThe basic idea of Lumigraph refinement[2]is to take an initially coarse Lumigraph that contains relatively few images(e.g.)and create addi-tional images for in-between eye points.The new images are created by forward-warping,which can be seen as an inverse depth correction.The advantage of Lumigraph warping in con-trast to usual warping situations is that the pix-els are reprojected into the same plane,and that the new image is only interpolated from several nearby views,instead of being extrapolated in a direction where no further eye points can be found. As it turns out,this special kind of warping is very simple,can be performed quite efficiently, and minimizes distortions caused by changes in sampling rate.After reprojecting the original pix-els,we interpolate thefinal pixel color from all non-occluded pixels that map to the same location, thus obtain a smoothly reconstructed image.If all information about all visible surfaces is contained in the original images,the output quality is very high,making them nearly indistinguishable from the originals.Since the in-between images can be generated very quickly(at about10images/second on an sgi O2),the refinement can be done after loading the Lumigraph.This is sometimes faster than loading a higher-resolution lightfield via a LAN.The re-sulting Lumigraph can be rendered in high quality using standard lightfield rendering.3Adaptive Acquisitionof Synthetic LumigraphsAs we domonstrated in[2],it is possible to re-fine coarse Lumigraphs and obtain a high ren-dering quality if the set of images contains suffi-cient information about all visible features of the scene.Since the acquisition of Lumigraph from a synthetic scene usually involves large amounts of computational effort,it is imperative to acquire only those images that are really needed for thefi-nal reconstruction.This can be achieved using a variant of the warp-ing method presented in the previous section as an error estimator for reconstruction quality[4].First, we acquire an initial,very coarse,Lumigraph(e.g.eye points/images).We create a triangulation of the eye points and estimate the warping error for a number of reasonable candidate eye points,e.g. for the centers of the triangle edges.We choose the eye point with the largest error,acquire the corre-sponding image by ray tracing,and adapt the tri-angulation and error estimate accordingly.The al-gorithm stops after the desired number of images is acquired,or the error of the next candiate drops below a threshold.The error estimate mainly takes into account two different factors:the difference of the colors that will be interpolated to obtain thefinal pixel color, and the number of source pixels that map onto the same target pixel.This way it is relatively easy to detect holes,blending errors,and pixels for which no conservative color estimate can be derived(e.g. if there is only information from one source pixel).Our techniqie enables a fully automatic adaptive acquisition of synthetic Lumigraphs that will pro-duce the(in our sense)optimal Lumigraph for a given number of images.The method could also be extended to select the most appropriate images from a sequence,e.g.when creating Lumigraphs from video streams.24Interactive LumigraphRendering Through WarpingAs we have shown in[2],warping can efficiently produce intermediate Lumigraph images of high quality.But the Lumigraph refinement method re-sults in a very large lightfield that consumes enour-mous amounts of memory and graphics hardware resources for interactive display.Rather than warping in a preprocess,we recently introduced a technique that allows to employ a similar warping technique for viewing Lumigraphs directly at interactive frame rates[5].For every novel view,wefirst partition the output image into regions using a triangulation of the eye point plane. For each triangle,we determine from which refer-ence images we want to extract and interpolate the color information.Next,we have to determine the source region in each affected reference image that has to be repro-jected in order tofill the corresponding target re-gion(a triangle fan)in the output image.We com-pute conservative bounds on the source regions by taking into account the pixelflow into that target region.Then,we reproject only the necessary pix-els from each source image to obtain the interpo-latedfinal image.This warped image is gener-ated on the original Lumigraph image plane,and finally reprojected into the actual view by the use of hardware-supported texture mapping.The technique allows interactive viewing of rel-atively sparse Lumigraphs in very high quality, achieving frame rates of around5–7frames/second on an sgi O2.The performance of the algorithm is nearly independent of the number of reference images in the Lumigraph.Fig.2demonstrates the reconstruction quality.5Conclusions and Future Work We have presented several warping-based tech-niques for acquiring,refining,and rendering Lu-migraphs.The methods achieve results of veryhigh quality from relatively compact Lumigraphs, and also represent an interesting bridge between traditional warping-based techniques and the light field representation.There are several directions of future research,the most promoising being multi-resolution techniques,redundancy-free cod-ing,and application to real-worlddata.Figure2:Example demonstrating the high qual-ity of the warping-based interactive rendering tech-nique presented in Sec.4.References[1]S.J.Gortler,R.Grzeszczuk,R.Szeliski,and M.F.Cohen.The Lumigraph.In Proc.SIGGRAPH’96,pages43–54.Addison-Wesley,1996.[2]Wolfgang Heidrich,Hartmut Schirmacher,HendrikKück,and Hans-Peter Seidel.A warping-based refine-ment of Lumigraphs.In Vaclav Skala,editor,Proc.WSCG’99),pages102–109,Plzen,Chech Republic,February1999.University of West Bohemia.[3]M.Levoy and P.Hanrahan.Lightfield rendering.In Proc.SIGGRAPH’96,pages31–42.Addison-Wesley,1996.[4]Hartmut Schirmacher,Wolfgang Heidrich,and Hans-Peter Seidel.Adaptive acquisition of Lumigraphs fromsynthetic scenes.In Pere Brunet and Roberto Scopigno,editors,Proc.Eurographics’99,volume18of ComputerGraphics Forum,pages C–151–C–160.Blackwell,1999.[5]Hartmut Schirmacher,Wolfgang Heidrich,and Hans-Peter Seidel.High-quality interactive Lumigraph ren-dering through warping.In Sidney Fels and PierrePoulin,editors,Proc.Graphics Interface2000,Montreal,Canada,May2000.CHCCS.3。