17-跨媒体检索介绍

Joint Graph Regularized Heterogeneous Metric
Leabharlann Baidu
Scale regularization

r(U,V) is used to control the scale of the parameters matrices and reduce overfitting.
跨媒体检索与分析
1
什么是跨媒体？从应用平台方面理解
电视机 报纸
电脑 手机
2
Ipad
什么是跨媒体？从检索研究方面理解
以文字搜文字 以图片搜图片
3
以文字搜图片
以文字搜视频
什么是跨媒体？
2010年1月《Nature》发表的“2020 Vision”论文 指出：文本、图像、语音、视频及其交互属性将紧密 混合（mix）在一起，即“跨媒体”。 2011年2月《Science》开灯“Dealing with Data” 专辑：数据的组织和使用体现跨媒体计算。
趋势：从“多媒体”研究向“跨媒体”发展！
4
什么是跨媒体？
跨媒体特性即多媒体数据之间以及用户 互动与多媒体数据之间存在着内容跨越 与语义关联。
吴飞, 庄越挺. 互联网跨媒体分析与检索:理论与算法. 计算机 辅助设计与图形学学报，Vol.22, No.1, pp.1-9, 2010.
5
跨媒体的主要研究范畴
11
Joint Graph Regularized Heterogeneous Metric

Heterogeneous Metric Learning

Given two sets of heterogeneous pairwise constraints
S {( xi , y j ) | lix l jy }
19
Joint Graph Regularized Heterogeneous Metric

Iterative optimization

Obtain orthogonal transformation matrices U and V , they minimize the following object function:

Joint graph regularization

The formulation of g(U,V) :

Minimizing g(U, V) encourages the smoothness of a mapping over the joint data graph, which is constructed from the initial label information
Joint Graph Regularized Heterogeneous Metric

Loss function

The minimization of the loss function will result in minimizing (maximizing) the distances between the media objects with the similarity (dissimilarity) constraints
Xiaohua Zhai, Yuxin Peng and Jianguo Xiao
Institute of Computer Science & technology, Peking University AAAI 2013
10
Motivation


Existing metric learning methods have previously been designed primarily for singlemedia data and cannot be directly applied to cross-media data. Make full use of the structure information of the whole heterogeneous spaces.

Obtain the analytical solution U and V as
We alternate between updates to U and V for several iterations to find a locally optimal solution. Here the iteration continues until the crossvalidation performance decreases on the training set. In practice, the iteration only repeats several rounds.
6
跨媒体分析的挑战
How to bridge both semantic-gap and heterogeneity gap?
Webpage
Japan Earthquake Video
Audio
Correlated multi-modal Data
7
From FeiWu
跨媒体的内容鸿沟
视觉特征空间
跨媒体检索：用户向计算机提交一种类型的多媒体对 象作为查询例子，系统可以自动找到其它不同类型及 语义上相似的多媒体对象。
跨媒体推理：跨媒体推理是指从一种类型的多媒体数 据，经过问题求解转向另外一种类型的多媒体数据。 （OCR等）
跨媒体存储：现有处理海量数据的检索技术主要是针 对文本信息，如 google 和百度等搜索引擎。跨媒体存 储研究高效压缩、索引和分片等方法，以及对用户行 为的个性化索引等技术。 惊涛骇浪

20
Maximize tr(X’TUVTY’) will minimize function, its singular value decomposition:
Joint Graph Regularized Heterogeneous Metric

Fix V and update U

Different Q(U,V) with respect to U and V setting it to zero, respectively:

where X’ and Y’ represent for two sets of coupled media objects from different media with the same labels. U and V define two orthogonal transformation spaces where media objects in X’ and Y’ can be projected as close to each other as possible.
arg min
U ,V
f (U , V ) g (U , V ) r (U , V )



14
f (U, V) is the loss function defined on the sets of similarity and dissimilarity constraints S and D g(U, V) and r(U, V) are regularizer defined on the target parameter matrices U, V. 0 , 0 are the balancing parameters.
zij 1, ( xi , y j ) D.
12
Joint Graph Regularized Heterogeneous Metric

They propose to learn multiple linear transformation matrices U and V , they can map the heterogeneous media data to a common output spaces. The distance measure is defined as:
L
where O represents for all of media objects in the learned metric space. denotes the normalized graph Laplacian.
18
Joint Graph Regularized Heterogeneous Metric
li lix 0 i m
li l(y i m) m i m n

where
17
Joint Graph Regularized Heterogeneous Metric

Joint graph regularization

Setting wii = 0 for 1< i ≤ m+n to avoid self-reinforcement. And the normalized graph Laplacian L is defined as:
内 容 鸿 沟
听觉特征空间
语义鸿沟 高层语义空间 爆炸、海洋、天空、鸟。。。。。。
8
基于线性变换的子空间映射算法
视觉特征空间 听觉特征空间
投影 子空 间
9
Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval
d ( xi , y j ) (U T xi V T y j )T (U T xi V T y j )

13
Joint Graph Regularized Heterogeneous Metric

Objective function

The formulation of the general regularization framework for heterogeneous distance metric learning is defined as:
Where I is an (m+n)×(m+n) identity matrix and D is an (m+n)×(m+n) diagonal matrix with dii j.wij

L is symmetric and positive semidefinite, with eigenvalue in the interval [0,2].
D {( xi , y j ) | lix l jy }


S is the set of similarity constraints and D is the set of dissimilarity constraints . Each pairwise constraints (xi,yj) indicates if two heterogeneous media objects xi and yj are relevant or irrelevant inferred from the category label. 1, ( xi , y j ) S ;
16
Joint Graph Regularized Heterogeneous Metric

Joint graph regularization


Defining a joint undirected graph, G = (V, W) on the dataset. Each element wij of the similarity matrix W = {wij}(m+n)(m+n) means the similarity between the i-th media object and j-th media object. Using label information to construct the symmetric similarity matrix: 1, li l j i j wij otherwise 0,

15
Normalize the elements of Z column by column to make sure that the sum of each column is zero to balance the influence of the similarity constraints and dissimilarity constraints.