基于图像绘制(IBR)技术综述

计算机图形学学科研讨会
基于图像绘制（IBR）技术综述（A Survey on Image-based Rendering Techniques）
石教英
浙江大学计算机学院
计算机辅助设计与图形学国家重点实验室
2005年5月
目录
一、基于图像绘制（IBR）技术定义
二、典型IBR技术演示
1、Panorama Mosaics
2、Tour-Into-Picture
3、Light Field
4、Feature-based Morphing
三、IBR技术理论基础-Plenoptic Funciton
1、An introduction
2、how to handle with plenoptic fuction
四、IBR技术发展
1、Image Matting
2、Digital PhotoMontage
3、High-Dynamic-Range Image Display
4、Plenoptic photography
五、基于图像绘制技术定义的扩展
Transform
Rasterization
(& Lighting)
¾Computer Vision methods to recover models.
Image-Based Rendering
& Modeling
Rendering
& Modeling Volume
Rendering
一、基于图像绘制（IBR）技术定义
z Problems of triangle-based graphics:•Always starts from scratch.
•Millions of sub-pixel triangles.
一、基于图像绘制（IBR）技术定义
¾Definition of IBR (by Sing Bing Kang)
image-based rendering techniques rely on interpolation using the original set of input images or pixel reprojection from source images onto the target image in order to produce a novel virtual view
一、基于图像绘制（IBR）技术定义
¾Definition of IBR (by Cha Zhang & Tsuhan Chen of CMU) Given a continuous plenoptic function that describes a scene, image-based rendering is a process of two stages: –sampling and rendering. In the sampling stage, samples are taken from the plenoptic function for representation and storage. In the rendering stage, the continuous plenoptic
function is reconstructed with the captured samples.
二、典型IBR技术演示¾Paronoma Mosaics
Demo
二、典型IBR技术演示¾Tour-Into-Picture
二、典型IBR技术演示Lightfield Video
二、典型IBR技术演示Feature-based Morphing
三、IBR技术理论基础-Plenoptic Funciton
1. An Introduction
z Two ways of describing the world:
• A source description: ——The world can be described by
geometric models, texture maps, refection models, lighting
and shading models.
•An appearance description: ——The world can be described
by the dense array of light rays filling the space, which can
be observed by posing eyes or cameras in the space. The
light rays can be represented through the Plenoptic function.
z The traditional model-based rendering approach
adopts the source description method.
z The image-based rendering approach adopots the
plenoptic function to describe the world.
三、IBR 技术理论基础-Plenoptic Funciton ¾IBR: An Old Story
z Plenoptic function z As pointed out by Adelson and Bergen (1991):The world is made of three -dimensional objects, but these objects do not communicate their properties directly to an observer. …The plenoptic function serves as the sole communication link between the physical objects and their corresponding retinal images. It is the intermediary between the world and the eye.
),,,,,,(t Z Y X f λφθ
三、IBR技术理论基础-Plenoptic Funciton ¾7D Plenoptic Function
三、IBR技术理论基础-Plenoptic Funciton
2. How to Handle Plenoptic Function?
z Two Stages
•sampling and reconstruction of sampled signals z Two directions for simplication
•Restrain the viewing space of the viewers
z View point
z Perceptual
•Introduce some source descriptions into IBR
z Geometry
z Depth
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 1: Wavelength
•Constant wavelength
•RGB
•Almost all the practical representations of IBR make this assumption.
)
,,,,,,(t Z Y X f λφθ
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 2: Air
•Air is translucent
•Radiances along a light ray through empty space remain constant.
)
,,,,,,(t Z Y X f λφθ
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 3: Time
•Static scene
•Images captured at different time and positions can be used together to render novel views.
•Too large size for dynamic scene
)
,,,,,,(t Z Y X f λφθ
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 4: Viewpoint
•The viewer is constrained to be on a surface
•Acceptable:
z
Human eyes are usually at a certain height -level z Human eyes are less sensitive to vertical parallax and lighting changes
),,,,,,(t Z Y X f λφθ
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 5: Viewpoint
•The viewer moves along a certain path.
•Reduces 2dimensions from the full plenoptic function.•Too large size for dynamic scene
)
,,,,,,(t Z Y X f λφθ
三、IBR 技术理论基础-Plenoptic Funciton ¾Restraining in View Space
z Assumption 6: Viewpoint
•The viewer has a fixed position.
•Reduces the dimension of the plenoptic function by 3.•No 3D effects can possibly be perceived.
•Similarity to regular images and videos.
)
,,,,,,(t Z Y X f λφθ
三、IBR技术理论基础-Plenoptic Funciton ¾Various Representations
三、IBR技术理论基础-Plenoptic Funciton
¾6D –The Surface Plenoptic Function z Assumption 2. As we discussed, when
radiance along a light ray through empty
space remains constant.
z6D
•Position on the surface (2D)
•Light ray direction(2D)
•Time (1D) and wavelength (1D).
三、IBR技术理论基础-Plenoptic Funciton
¾Example
z The surface light field could
be considered as
dimension-reduced version
of SPF.
D. N. Wood, D. I. Azuma, K.
Aldinger, B. Curless, T.
Duchamp, D. H. Salesin
and W. Stuetzle, “Surface
light fields for 3D
photography”, Computer
Graphics (SIGGRAPH’00),
July 2000.
三、IBR技术理论基础-Plenoptic Funciton
z5D –Plenoptic Modeling and Light field Video Plenoptic modeling, which is a 5D function
•3D for the camera position, 2D for the cylindrical
image.
L. McMillan and G. Bishop, “Plenoptic modeling: an
image-based rendering system”, Computer
Graphics (SIGGRAPH’95), August 1995,
三、IBR技术理论基础-Plenoptic Funciton
¾Plenoptic Modeling
z To render a novel view from the 5D representation, the close-by cylindrical projected images are warped to the viewing
position based on their epipolar relationship and some visibility tests.
三、IBR技术理论基础-Plenoptic Funciton
¾4D –Light field / Lumigraph
z Assumption 1, 2 and 3
z Trickily parameterize ray space to 2-plane representation.
u
f
(t s
v
),,,
三、IBR技术理论基础-Plenoptic Funciton
三、IBR技术理论基础-Plenoptic Funciton
¾3D –Concentric Mosaics
z Assumption 1, 2, 3 and 4
Tripod
z Center camera => panorama
z Off-centered cameras => motion parallax
…...
Top view
三、IBR技术理论基础-Plenoptic Funciton
¾2D –Image Mosaicing
z Composes one single mosaic with multiple
input images
z In most cases, the light rays recorded in the mosaic share the same center-of-projection
(COP) -Panoramic mosaic or panorama
三、IBR技术理论基础-Plenoptic Funciton ¾Examples
三、IBR技术理论基础-Plenoptic Funciton
¾2D –Image Mosaicing
z In the more general scenario, the cameras of the input images can move in free form and
the resultant mosaic has MCOPs
z Manifold Mosaic.
三、IBR技术理论基础-Plenoptic Funciton ¾Examples
三、IBR技术理论基础-Plenoptic Funciton
¾Quicktime VR
z Using environmental
maps
•Cylindrical
•Cubic
•spherical
z At a fixed point,
sample all the ray
directions.
z Users can look in both
horizontal and vertical
directions
三、IBR技术理论基础-Plenoptic Funciton ¾Mars Pathfinder Panorama
三、IBR技术理论基础-Plenoptic Funciton ¾IBR with Various Source Descriptions
三、IBR技术理论基础-Plenoptic Funciton ¾Correspondence between Images
三、IBR技术理论基础-Plenoptic Funciton ¾Example 1: View Morphing
(SIGGRAPH 96) Steve Seitz etc.
三、IBR技术理论基础-Plenoptic Funciton ¾Dense Depth Map
Viewing Region
Sprite with
Depth
LDI
Environment Map
三、IBR技术理论基础-Plenoptic Funciton
¾Layered Depth Images
z Idea:
•Handle disocclusion
•Store invisible geometry in depth images
三、IBR技术理论基础-Plenoptic Funciton
¾Texture Map (+ Scene Geometry) z Image based modeling
z View dependent texture mapping
三、IBR技术理论基础-Plenoptic Funciton
¾Reflection Models (+ Scene Geometry) z Image based relighting。