17-跨媒体检索介绍

19
Joint Graph Regularized Heterogeneous Metric

Iterative optimization

Obtain orthogonal transformation matrices U and V , they minimize the following object function:
arg min
U ,V
f (U , V ) g (U , V ) r (U , V )



14
f (U, V) is the loss function defined on the sets of similarity and dissimilarity constraints S and D g(U, V) and r(U, V) are regularizer defined on the target parameter matrices U, V. 0 , 0 are the balancing parameters.
跨媒体检索与分析
1
什么是跨媒体？从应用平台方面理解
电视机 报纸
电脑 手机
2
Ipad
什么是跨媒体？从检索研究方面理解
以文字搜文字 以图片搜图片
3
以文字搜图片
以文字搜视频
什么是跨媒体？
2010年1月《Nature》发表的“2020 Vision”论文 指出：文本、图像、语音、视频及其交互属性将紧密 混合（mix）在一起，即“跨媒体”。 2011年2月《Science》开灯“Dealing with Data” 专辑：数据的组织和使用体现跨媒体计算。
L
where O represents for all of media objects in the learned metric space. denotes the normalized graph Laplacian.
18
Joint Graph Regularized Heterogeneous Metric
zij 1, ( xi , y j ) D.
12
Joint Graph Regularized Heterogeneous Metric

They propose to learn multiple linear transformation matrices U and V , they can map the heterogeneous media data to a common output spaces. The distance measure is defined as:
D {( xi , y j ) | lix l jy }


S is the set of similarity constraints and D is the set of dissimilarity constraints . Each pairwise constraints (xi,yj) indicates if two heterogeneous media objects xi and yj are relevant or irrelevant inferred from the category label. 1, ( xi , y j ) S ;

Joint graph regularization

The formulation of g(U,V) :

Minimizing g(U, V) encourages the smoothness of a mapping over the joint data graph, which is constructed from the initial label information
Joint Graph Regularized Heterogeneous Metric

Scale regularization

r(U,V) is used to control the scale of the parameters matrices and reduce overfitting.
11
Joint Graph Regularized Heterogeneous Metric

Heterogeneous Metric Learning

Given two sets of heterogeneous pairwise constraints
S {( xi , y j ) | lix l jy }
跨媒体检索：用户向计算机提交一种类型的多媒体对 象作为查询例子，系统可以自动找到其它不同类型及 语义上相似的多媒体对象。
跨媒体推理：跨媒体推理是指从一种类型的多媒体数 据，经过问题求解转向另外一种类型的多媒体数据。 （OCR等）
跨媒体存储：现有处理海量数据的检索技术主引和分片等方法，以及对用户行 为的个性化索引等技术。 惊涛骇浪

where X’ and Y’ represent for two sets of coupled media objects from different media with the same labels. U and V define two orthogonal transformation spaces where media objects in X’ and Y’ can be projected as close to each other as possible.
6
跨媒体分析的挑战
How to bridge both semantic-gap and heterogeneity gap?
Webpage
Japan Earthquake Video
Audio
Correlated multi-modal Data
7
From FeiWu
跨媒体的内容鸿沟
视觉特征空间

20
Maximize tr(X’TUVTY’) will minimize function, its singular value decomposition:
Joint Graph Regularized Heterogeneous Metric

Fix V and update U

Different Q(U,V) with respect to U and V setting it to zero, respectively:
内 容 鸿 沟
听觉特征空间
语义鸿沟 高层语义空间 爆炸、海洋、天空、鸟。。。。。。
8
基于线性变换的子空间映射算法
视觉特征空间 听觉特征空间
Байду номын сангаас
投影 子空 间
9
Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval
d ( xi , y j ) (U T xi V T y j )T (U T xi V T y j )

13
Joint Graph Regularized Heterogeneous Metric

Objective function

The formulation of the general regularization framework for heterogeneous distance metric learning is defined as:

Obtain the analytical solution U and V as
We alternate between updates to U and V for several iterations to find a locally optimal solution. Here the iteration continues until the crossvalidation performance decreases on the training set. In practice, the iteration only repeats several rounds.
Xiaohua Zhai, Yuxin Peng and Jianguo Xiao
Institute of Computer Science & technology, Peking University AAAI 2013
10
Motivation


Existing metric learning methods have previously been designed primarily for singlemedia data and cannot be directly applied to cross-media data. Make full use of the structure information of the whole heterogeneous spaces.
16
Joint Graph Regularized Heterogeneous Metric

Joint graph regularization


Defining a joint undirected graph, G = (V, W) on the dataset. Each element wij of the similarity matrix W = {wij}(m+n)(m+n) means the similarity between the i-th media object and j-th media object. Using label information to construct the symmetric similarity matrix: 1, li l j i j wij otherwise 0,