Stochastic Search Algorithms for Optimal Content-based Sampling of Video Sequences

Stochastic Search Algorithms for OptimalContent-based Sampling of Video SequencesAnastasios D. Doulamis and Nikolaos D. DoulamisNational Technical University of Athens, Department of Electrical and Computer Engineering 9, Heroon Polytechniou, 157 73 Zografou, GreeceE-mail: adoulam@cs.ntua.grAbstractA video content representation framework is proposed in this paper, for extracting limited, but meaningful, information of video data, directly from the MPEG compressed domain. In particular, extraction of several representative shots is performed for each video sequence in a content based rate sampling framework. An approach, based on minimization of a cross-correlation criterion of the video frames has been adopted for the shot selection. For efficient implementation of the latter approach, a logarithmic search in a stochastic framework is proposed. The method always converges to the global minimum as is proven in the paper.1.IntroductionRecent progress in communication technology has led to rapid development of multimedia systems that allow users to store, retrieve and manipulate several types of visual information [1]. Many emerging applications, such as telemedicine, video on demand, distance learning, home education or even access to video libraries are expected to involve multimedia systems [2], [3], [4]. However, the traditional text-based approach, used to perform the management of these visual databases, do not work well for video applications, to which it is hard to describe the rich visual content, using only text annotation [5].Therefore, efficient indexing of visual information is required to enable fast access to the stored data, especially in cases of distributed video streams over heterogeneous platforms [6]. Traditionally, video is represented by numerous consecutive frames, each of which corresponds to a constant time interval. Moreover, access of a video sequence is linear, (sequential) and thus time consuming. On the contrary, non-linear access to video frames can present the visual content in a more compact form so that users can quickly browse through a video sequence or locate segments of interest [2], [7], [8].For this reason, a “pre-indexing” stage should be introduced to video databases, extracting a limited but meaningful information of video content. The purpose of this process is to divide video sequences into a set of meaningful and manageable “segments”, which can be used as indices for visual information. A video shot, which represents a sequence of frames, generated during a continuous action in time as well as in space, can be considered as the basic unit for such content segmentation. Then, only a small but characteristic number of shots should be extracted, which are able to give an overview of the underlying story of the video sequence and permit the user to handle the visual information more efficiently.Some works have been recently proposed in the literature towards this goal. In [9], a shot detection algorithm have been proposed, which can be used as a first stage of the proposed video representation scheme. Recently some other approaches, dealing with construction of a compact image map or image mosaics have been described in [8], [10]. Although such a representation can be very good for specific applications, it cannot be effectively implemented in real world complex shots, where background/ foreground changes or complicated camera effects may appear. A genetic algorithm used for analyzing video and building a pictorial summary for visual representation has been proposed in [7]. Extension of the previous approach to thee-dimensional (3D) video sequences has been reported in [11]. Furthermore, a simple heuristic technique, which also exploits the temporal feature vector variation has been proposed in one of our earlier works in [12]. In this work, the curvature of the feature vector trajectory is used as key-frame/shot selection.In this paper, an optimal stochastic framework is proposed for summarizing video sequences. Instead of the previous approaches, the temporal behavior of video content is taken into account and thus the redundant temporal information is reduced. This is especially evident, in cases of video conferencing sequences or documentary films, where many similar shots are repeated. On the contrary, the aforementioned summarization algorithms would fail in such sequences of "periodic content".Furthermore, a stochastic logarithmic scheme is adopted to perform the optimal classification of the temporal frame content. The concept of the proposed scheme stems form the logarithmic search method used in the MPEG coding standards. However, in our case the scheme has been greatly enhanced by using probabilistic theory, which makes the algorithm able to converge to any possible solution and thus cannot be trapped to local minimum. The convergence proof is also given in this paper.2.Video Visual Analysis StageThe first stage of the proposed scheme is i) to extract appropriate features from each video frame and ii) to detect the time instances (frame numbers) of shot changes. The latter is performed by applying the algorithm in [9] due to its efficiency and minor complexity, compared to other techniques. Furthermore, it exploits the dc coefficients of DCT transform and thus it can be applied directly to the MPEG compressed domain.As far as the features are concerned, two different types of descriptors are extracted in our case for content representation. The first type refers to global frame properties (global-based descriptors), while the second to object characteristics (object-based descriptors).As global-based descriptors, the color, texture and motion histogram, estimated over all pixels of a video frame, are considered. The histograms are evaluated directly in the MPEG compressed stream as in [7] to reduce the computational cost and storage requirements. If we denote as c gh, t gh and m gh the vectors which contain the histogram bins of global color, texture and motion properties respectively, then, the feature vector g f of global-based descriptors is given asTTmgTtgTcgg])()()[(hhhf=(1) In case object-based descriptors, the frames are first partitioned into segments by applying a segmentation algorithm.A multiresolution implementation of the Recursive Shortest Spanning Tree algorithm (RSST), called M-RSST [13], is adopted to perform the segmentation task due to the fact that it yields much faster execution time, while simultaneously keeping almost the same performance compared to the RSST [13]. For each color segment, the three color components in RGB space, the segment location and the respective area are selected as segment descriptors. Similarly, for each motionsegment, the average motion vectors and the respective location and area are extracted.In the following all color/motion segment descriptors are classified into pre-determined classes forming an object-basedcolor and motion histogram denoted as c o h and m o h . As aresult, the feature vector of object-based descriptors o f is givenasT T m o T c o o ])()[(h h f =(2)Thus, gathering the vectors g f and, the total feature vectorf of a video frame is constructed asT T oT g ][f f f = (3)The shot feature vector, say g is then formed by averagingthe feature vectors of each individual frame, composing the respective shot sequence.∑∈= 1S i i S f g(4)where )(k i f corresponds to the ith feature vector of the respective shot. and S a set containing the frame indices. The S indicates the cardinality of S , i.e., the number of S elements.3. Optimal Extraction of Characteristic ShotsHaving estimating the shot feature vectors, the next step of the algorithm is the extraction of a small but representative number of shots that are able to efficiently characterize the video content. Since each shot feature vector indicates the properties of the respective shot, we can define as the most representative shots of a video sequence, those, which are not so similar to each other. For this reason, an optimal key shot selection algorithm is introduced, based on an optimization method for locating a set of minimally correlated shot feature vectors. This is achieved by minimizing a cross-correlation criterion among all shots of a given sequence.Let us define by s N the total number of shots for a video sequence. Let us also denote by }1,,1,0{−=s N V the set containing all shot indices and by K s the number of representative shots that should be selected. Then, the correlation coefficient of feature vectors l k g g , for the two different shots is defined as)(,,l k l k l k C σσρ=, with }1,,0{,−∈s N l k (5)where )()(, g g −−=l T k l k C is the covariance of l k g g , ,while s iN i N s /10∑−==g the average shot feature vector of thesequence. The parameter i i i C ,2=σ corresponds to respective variance. Without loss of generality, it is next assumed that P s N 2= where P is an integer number, i.e., the number of shot 0frames is a power of 2. In case that the actual number does not meet this constraint, extension of the video sequence is performed by adding dummy shots whose correlation coefficients are set to infinity so that they cannot be selected as representatives.Based on correlation coefficients between pairs of feature vectors, the correlation among K s feature vectors can be defined as∑∑−=+=−==1112,1)()1(2),,()(s s j i s K i K i j x x s s K K K x x R R ρ x(6)where T K s x x x ][21 =x is an index vector, the elements of which,V x i ∈ are sorted as s K x x << 1. Thus,s K V W ⊂∈x . This is clear, since the correlation measure ofthe s K features is independent of the feature arrangement.Thus, any permutations of the elements of x will result in the same sets. Based on the above definition, searching for a set of s K minimally correlated feature vectors is equivalent to searching for an index vector x that minimizes )(x R . Finally,the set of the s K least correlated feature vectors is represented by)(min arg )ˆ,,ˆ(ˆ1x xx R x x WK s∈== (7)Unfortunately, the complexity of an exhaustive search for obtaining the minimum value of )(x R is such that a direct implementation of the method is practically unfeasible. In this paper, a new optimal algorithm is proposed, which combines the probabilistic theory with the logarithmic search method. The scheme is able to find any possible solution and thus is cannot be trapped to local minima as it is proved in section 3.2 of the paper.3.1.The Stochastic Search AlgorithmThe concept of the proposed method is to assign a probability to every neighbor point of the current examined point )(n x . The index n has been added to indicate the nth iteration of the algorithm Let us denote as},)()(:{))(),((s K G n n W n n N ∈+=∈=p p x y y x δδ(8)the neighbors of the examined points )(n x . The G ={-1,0,1}while corresponds to an integer indicating the step size of the neighborhood region. The step size is initialized as22)0(−=P δ so that the algorithm covers all possible points.At each iteration, the following steps are repeated. (i) to select the neighbor of )(n x ii) to divide the step size by two.The selection of the neighboring point is performed using a probability, which is inversely proportional to the respective correlation measure of the neighbor.The search procedure is repeated several times, so that in effect multiple logarithmic search experiments take place in a random way. Due to the stochastic behavior of the algorithm,different neighbors are selected in every new experiment,resulting in the generation several random paths.Let us denote by )(n m x the index vector at the n th iteration step for the m th experiment, and by i y , i =1,…,N its neighbors, where N is the set cardinality. Vector )(n m x is the same as vector )(n x , apart from index m , which corresponds to the mth experiment. Then, a probability value is assigned to each iy N i R R p N j j i i ,,1,)()(11=−=∑=y y(9)A cumulative probability function is then constructed for all i y , i.e., ∑==i j j i p c 1, i =1,…,N , with 00=c . Using a given random number r , uniformly distributed in the range [0,1], the next index vector )1(+n m x is chosen among the neighbors i y as follows}:))(),(({)1(1i i m i m q r q n n N n ≤<∈=+−δx y x (10)The iteration is repeated n =0,2,…,P -2, as in the case of the logarithmic search algorithm, and the result of the m th experiment is the index vector))((min arg ˆ1,,0i R m M i m x x −== (11)corresponding to the minimum correlation measure along thepath of the experiment. The final result is the index vectorcorresponding to the minimum correlation measure of all vectors in all experiments. After J experiments, the optimal solution)ˆ(min arg ˆ,,1m Jm R x x== (12)is selected, containing the indices of the K s key frames.3.2.The Convergence of the Stochastic Search AlgorithmIn this section, we prove that the aforementioned algorithmalways converges to the optimal solution (global minimum).This is accomplished by modeling the scheme using Markov chains.In particular, each state of the Markov chain corresponds to a possible solution of the optimization problem, i.e., to an index vector x . Let us denote by C the set that contains all possible states of the chain, i.e., all possible values of index vector x . Let us also denote as j i p , the transition probability from a state C i ∈ to another state C j ∈. Gathering all transition probabilities, a matrix )(,j i p =P is formed.Matrix P is further decomposed asA L P ⋅=(13)where matrix L corresponds to the logarithmic search mechanism while matrix A to the stochastic mechanism.Matrix L is stochastic, that is its elements ij l satisfies the property 1=∑i ij l . This is due to the fact that the proposed search mechanism maps one state to another uniquely (with probability 1).On the other hand, matrix A is positive since it contains transition probabilities (positive numbers) estimated for all possible transitions. However, if L is stochastic and matrix A is positive, then the transition matrix A L P ⋅= of the Markov chain is primitive (i.e., ∃k >0: P k is positive). In this case, the Markov chain is ergodic , which means that starting from a random state, the process can visit any other existing state in finite number of transitions (finite time). As a result, the algorithm always converges to the optimum solution after a finite number of steps.4. Experimental ResultsThe proposed algorithms were integrated using real life video sequences. Figures 1-3 show the results obtained from a documentary film coded using the MPEG-2 algorithm. The film deals with space technology. Applying the shot detection algorithm described in section 2, 14 shots have been extracted.The most characteristic shots (key-shots) extracted using thee proposed stochastic logarithmic method, are illustrated in Figure 1. As can be seen, the shots are able to efficiently represent the video visual content. This is due to the fact that although a small amount of shots is retained, it is clear that one can visualize the content of the sequence by just examining the three selected shots. Consequently, the selected shots give a meaningful representation of the content of the video content.In order to estimate the performance of the algorithms in terms of the obtained correlation measure R (x ), a test of 100,000random index vectors is first performed, and a histogram of R (x ) is constructed, as depicted in Figure 2(a). The optimal values of R (x ) obtained through the proposed stochastic logarithmic search approach is shown in Figure 2(b). In the same figure, we have also depicted the minimum value of the conventional logarithmic search without applying the stochastic framework at the same number of cycles iterations. As can be seen, the minimum value of the proposed technique is lowerthan the other two, verifying the fact that the proposed method outperforms the other two. The most characteristic shots are illustrated in Figure 3.Another example is presented in Figures 4-6, for a shot of the aforementioned sequence. One shot every 10 is depicted in Figure 4 for presentations purposes. The most characteristics frames extracted using the stochastic search algorithm are shown in Figure 5, while comparison with a random search approach and the minimum value obtained by the conventional algorithm is shown in Figure 6.5. References[1] Y. Deng and B. S. Manjunath, “NeTra-V: Toward anObject-Based Video Representation,” IEEE Trans. on Circuits and Syst. for Video Techn., vol. 8, no. 5, pp. 616-627, Sept. 1998.[2] F. M. Idris and S. Panchanathan, “Spatio-TemporalIndexing of Vector Quantized Video Sequences,” IEEE Trans. on Circuits & Syst. For Video Techn., vol. 7, no. 5,Oct. 1997.[3] N. Doulamis, A. Doulamis, D. Kalogeras and S. Kollias,“Very Low Bit-Rate Coding of Image Sequences Using Adaptive Regions of Interest,” IEEE Trans. Circuits and Systems for Video Technology , vol. 8, no. 8, pp. 928-934.[4] N. Doulamis, A. Doulamis and S. Kollias, “Improving thePerformance of MPEG Compatible Encoding at Low Bit Rates Using Adaptive Neural Networks,” J. of Real Time Imaging , Academic Press, Vol. 6, pp. 327- 345, Oct. 2000.[5] Y. Riu, T. Hunag, M. Ortega and S. Mehrotra, “RelevanceFeedback: A Power Tool for Interactive Content-based Image Retrieval,” IEEE Trans. on Circ. Systs for Video Techn., vol. 8, pp. 644-656.[6] M. Yeung and Boon-Lock Yeo, “Video Visualization forCompact Presentation and Fast Browsing of Pictorial Content,” IEEE Trans. on Circuits and Systems for Video Techn., vol. 7, no. 5, pp. 771- 785, Oct. 1997.[7] Y. Avrithis, A. Doulamis, N. Doulamis and S. Kollias, “AStochastic Framework for Optimal Key Frame Extraction from MPEG Video Databases,” Journal of Computer Vision and Image Understanding, Academic Press, Vol.75, Nos 1/2, pp. 3-24, July/August 1999.[8] Michal Irani and P.Anandan,” Video Indexing Based onMosaic Representation,” Proceedings of the IEEE , vol. 86,no. 5., pp. 805-921, May 1998.[9] B. L. Yeo and B. Liu, “Rapid Scene Analysis onCompressed Videos,” IEEE Trans. Circuits and Systems for Video Technology , vol. 5, pp. 533- 544, Dec. 1995.[10] N. Vasconcelos and Andrew Lippman, “A SpatiotemporalMotion Model for Video Summarization,” Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 361- 366, Santa Barbara, CA, June 1998.[11] N. Doulamis, A. Doulamis, Y. Avrithis, K. Ntalianis and S.Kollias, "Efficient Summarization of Stereoscopic Video Sequences," IEEE Trans. on CSVT , Vol. 10, pp. 501-517,June 2000.[12] $ 'RXODPLV 'RXODPLV DQG 6 Kollias, "Non-SequentialVideo Content Representation Using Temporal Variation Of Feature Vectors," IEEE Trans. on Consumer Electronics , Vol. 46, No. 3, pp. 758-768, August 2000.[13] A, Doulamis, Y. Avrithis, N. Doualmis and S. Kollias,“Interactive Content-Based Retrieval in Video Databases Using Fuzzy Classification and Relevance Feedback”,IEEE Inter. Comf. On Multimedia & Computer Systems,Florence Italy June 1999.Figure 1. Shots of a video sequence.0.550.60.650.70.750.80.850.90.95100.010.020.030.040.050.060.07Correlation Measure R(x)Prob a b i l i t y Dens it y10203040500.30.350.40.450.50.550.60.650.7Number of CyclesC o r r e l a t i o n C o e f f i c i e n t J (z )Figure 2. (a) Histogram of correlation measure R (x ) together with optimal values, (dashed line, logarithmic solid line, stochastic logarithmic search) for the above space sequence. (b) The convergence of correlation measure R (x ) versus the number of cycles of the stochastic logarithmic algorithmic for the same sequence.Figure 3. The three most representive shots selected using the proposed optimal scheme.#0#20#40#60#80#100#120#140#160#180#200#220#240#260#280#300#320Figure 4. A video shot about space consisting of 334 frames, shown with one frame every 10.#22#143#228#255#261Figure 5. The six key-frames selected using the proposed stochastic scheme.0.20.30.40.50.60.70.80.9100.010.020.030.040.050.06Correlation Measure R(x)P r o b a b i l i t y D e n s i t y )Figure 6. Histogram of correlation measure R (x ) together with optimal values, (dashed line, logarithmic solid line, stochastic logarithmic search) for the shot of Figure 4.。