PREDICTING DISCRETE PROBABILITY DISTRIBUTION OF IMAGE EMOTIONS

ABSTRACT Most existing works on affective image classification tried to assign a dominant emotion category to an image. However, this is often insufficient, as the emotions that are evoked in viewers by an image are highly subjective and different. In this paper, we propose to predict the probability distribution of categorical image emotions. Firstly we extract commonly used features of different levels for each image. Then we formulize the emotion distribution prediction as a shared sparse leaning problem, which is optimized by iteratively reweighted least squares. Besides, we introduce three baseline algorithms. Experiments are carried out on a dataset of peer rated abstract paintings and the results demonstrate the superiority of our proposed method, as compared to some state-of-the-art approaches. Index Terms— Emotion Distribution Prediction, Image Emotion, Sparse Learning 1. INTRODUCTION Understanding the emotions implied in images has been widely studied recently due to its various applications in entertainment, advertisement and education [1]-[3]. Two main challenges that restrain the development of image emotion analysis are affective gap and subjective evaluation [2]. Most existing works focused on finding features that can express emotions better to bridge the affective gap. Holistic image features including Wiccest features and Gabor features are extracted to classify image emotions in [4]. Lu et al. [5] investigated the computability of emotion through shape features. Machajdik et al. [6] extracted features inspired from psychology and art theory, such as color, texture and composition. Zhao et al. [2] proposed to extract principles-of-art based emotion features, which are demonstrated to be more interpretable by humans and have stronger link to emotions than the elements-of-art based ones. Visual sentiment ontology and detectors are proposed to detect high-level adjective noun pairs based on large-scale social multimedia data [1, 3]. Besides, simple social correlation features are explored for emotion analysis of social network images [7].
This work was supported by the National Natural Science Foundation of China (No. 61472103) and Key Program (No. 61133003).
Amusement Anger Awe Contentment Disgust Excitement Fear S/15/$31.00 ©2015 IEEE
2459
ICIP 2015
mize related objective function. To our knowledge, this is the first work on emotion distribution prediction. Experiments are conducted on a dataset of peer rated abstract paintings to demonstrate the effectiveness of the proposed emotion distribution prediction method. 2. METHOD In this section, we introduce the proposed method in detail, including the extracted emotion features and the shared sparse learning algorithm. Since no algorithms to predict image emotion distribution have been developed, we first introduce three simple baselines and then present our main prediction algorithm. 2.1. Emotion Features The dataset used for evaluation is composed of abstract paintings (see Section 3.1). So we extract features inspired from art theory and low-level visual features for each image, as they are demonstrated to outperform other features for affective analysis of abstract images with few attributes, concepts or faces detected [12]. We choose GIST as low-level visual features for its powerful description ability of visual phenomena in a scene perspective [17]. We also extract features derived from elements of art, including color, texture and line [6], as special low-level features. Features inspired from principles of art, including balance, contrast, harmony, variety, gradation, and movement [2], are extracted as mid-level features. The three kinds of features are abbreviated as GIST, Elements and Principles with dimensions of 512, 48 and 165, respectively. Readers can refer to [2, 6, 12] for more details. It should be noted that our method can be easily extended to other features like attributes [17] and adjective noun pairs [1] for predicting emotion distribution of natural images. 2.2. Emotion Distribution Prediction Algorithms Suppose we have M emotion categories c1 , · · · , cM and N training images x1 , · · · , xN , which also denote related features of dimension D. Let pn = [pn1 , · · · , pnm , · · · , pnM ]T denote the emotion distribution of image xn , where pnm represents the probability that image xn conveys emotion cm (n = 1, · · · , N , m = 1, · · · , M ). For each image xn , we M have m=1 pnm = 1. Suppose x is a test image, then our goal is to predict emotion distribution p = [p1 , · · · , pM ]T , where pm = p(cm |x), for image x based on training examples (xn , pn )N n=1 . 2.2.1. Baseline A: Global Weighting The idea of global weighting (GW) is simple and direct. The emotion distributions pn (n = 1, · · · , N ) of all training images are considered as basis functions. The emotion distri-