Recognizing Continuous Grammatical Marker Facial Gestures in Sign Language Video

Recognizing Continuous Grammatical Marker Facial Gestures in Sign Language VideoTan Dat Nguyen1and Surendra Ranganath21Dept.of Electrical&Computer EngineeringNational University of Singapore,Singapore117576ntdat@.sg2Indian Institute of Technology–Gandhinagar,India382424surendra@iitgn.ac.inAbstract.In American Sign Language(ASL)the structure of signedsentences is conveyed by grammatical markers which are represented byfacial feature movements and head motions.Without recovering gram-matical markers,a sign language recognition system cannot fully re-construct a signed sentence.However,this problem has been largelyneglected in the literature.In this paper,we propose to use a2-layerConditional Random Field model for recognizing continuously signedgrammatical markers in ASL.This recognition requires identifying bothfacial feature movements and head motions while dealing with uncer-tainty introduced by movement epenthesis and other effects.We usedvideos of the signers’faces,recorded while they signed simple sentencescontaining multiple grammatical markers.In our experiments,the pro-posed classifier yielded a precision rate of93.76%and a recall rate of85.54%.1IntroductionInterpreting sign language not only requires recognition of hand gestures/signs, but also other non-manual signs.As pointed out in[1],non-manual signs convey important grammatical information.Without these grammatical markers,the same sequence of hand gestures can be interpreted differently.For example,with the hand signs for BOOK and WHERE,a couple of sentences can be framed as –[BOOK]T P[W HERE]W H→Where is the book?–[BOOK]T P[W HERE]RH→I know where the book is!In the notation of the above example,the left hand side of the arrows represent signs in American Sign Language(ASL).The subscripts TP,WH and RH on the words BOOK and WHERE indicate grammatical markers conveyed by facial feature movements and head motions.The facial gesture for Topic(TP)is used to convey that BOOK is the topic of the sentence.The word WHERE accompanied by a WH facial gesture,signals a“where?”.The hand sign for WHERE made concurrently with the facial gesture for RH indicates the rhetorical nature of the second sentence.R.Kimmel,R.Klette,and A.Sugimoto(Eds.):ACCV2010,Part IV,LNCS6495,pp.665–676,2011.Springer-Verlag Berlin Heidelberg2011666T.D.Nguyen and S.RanganathThus,recognition of non-manual signs is required for building a complete sign language understanding system.However,review[2]of sign language recognition indicates that the dominant interest in sign language recognition has been insign recognition has only recently startedPrevious works on recognizing facial expressions were reviewed in[5]and[6]. These surveys showed that many works focused on recognizing the six isolated universal expressions(Anger,Disgust,Fear,Happiness,Sadness,and Surprise) with minimal head motion.The latter simplification of the problem makes these methods inapplicable for recognizing facial gestures in sign language,where facial expressions are defined concurrently with head motion to define grammatical markers.There are also many works on analyzing head pose and head motion[7]. However,there are few works in the literature that address recognizing facial expressions coupled with concurrent head motion.Black and Yacoob’s work[8]is a pioneering work in recognizing continuous facial expressions with head motion based on features from dense opticalflow and rule-based discriminative models.They obtained an average recognition rate of88%and73%on laboratory data and data from TV programs,respectively. De la Torre et al.[9]proposed to detect rare facial gestures made during an interview based on Personalized Active Appearance Model[10].However,quan-titative assessment of the detection was not reported.Cohen et al.[11]used a piecewise3D wire frame model-based approach for tracking16facial features and estimated their3D motions.These were used in a multi-level HMM scheme for classifying the six universal expressions and the neutral expression in video sequences containing multiple expressions.They reported82.46%and58.63% accuracy for person dependent and person independent tests,respectively,on their database of5persons.As generative models,HMMs suffer from two weaknesses:the statistical inde-pendence assumption of observations and the difficulty in modeling their compli-cated underlying distributions.On the other hand,Conditional Random Fields (CRF)proposed by Lafferty et al.[12]is a discriminative model which avoids these weaknesses.Kanaujia and Metaxas[13]used the CRF to recognize the six universal expressions and obtained promising results.Quattoni et al.[14]pro-posed Hidden-state CRF(HCRF)models and obtained an accuracy of85.25% for recognizing head shakes and head nods.Chang et al.[15]proposed a modified HCRF called Partially-Observed HCRF(PO-HCRF).The PO-HCRF achieved an accuracy of80.1%with9.18%false alarm rate for recognizing the six“con-tinuous”universal facial expressions in simulated sequences created by concate-nating sequences of isolated expressions.Neidle et al.[4]proposed to detect the presence of WH and NEG grammatical markers in ASL signed sentences.An ASM-based tracking scheme proposed was used to track face and facial feature movements,and provide head pose(pitch,yaw,and tilt)in each frame.Each video frame was classified as either WH or not-WH,and a video sequence was labeled based on majority voting of frames.A multiple-SVM classifier was usedRecognizing Continuous Grammatical Marker Facial Gestures667 to label each frame.The recognition accuracies were100%and95%for WH and NEG,respectively.In this paper,we consider recognizing continuous facial gestures in sign lan-guage,particularly grammatical markers inconsidered in this paper are summarized inhead movements.We propose to use a layered Conditional Random Field(CRF) model[12]for this purpose.The classifier includes two CRF layers,thefirst layer to model head motions and the second to model grammatical markers.The sepa-rate head motion layer helps to reduce the ambiguity in recognizing grammatical markers in the second layer.For each video sequence,probabilities of different head motions are evaluated by thefirst layer,and these are input to the sec-ond layer together with other features for labeling the grammatical marker for each frame.Manually annotated labels of head motions and grammatical mark-ers were used for training the classifier and assessing performance.The classifier yielded precision and recall rates of95.24%and85.54%,respectively.2Recognizing Continuous Facial Gestures in Sign Language2.1ChallengesFacial gestures in ASL are identified from head motion and facial feature move-ment.In this paper we consider recognition of six grammatical markers listed and described in Table1,through their head gestures comprising,eye,eyebrow and head movements.In previous work[16],we have considered recognition of isolated facial gestures.Here,we extend our work to recognition of continuous facial gestures as would occur in sign language discourse,and consider four types of facial gesture sequences(Table2)composed of these grammatical markers. Examples of these facial gesture chains are shown in Table3.There are several aspects to the continuous facial gesture recognition problem which make it challenging,more so than isolated recognition.Movement epenthe-sis is the extra motion required by the head(and facial features),due to physical constraints,to transit from the end of the previous gesture to the beginning of Table1.Simplified description of the six ASL grammatical markers(Exp.)considered: Assertion(AS),Negation(NEG),Rhetorical(RH),Topic(TP),Wh question(WH),and Yes/No question(YN).Nil denotes unspecified facial feature movements.Exp.Brow Eye HeadAS Raise Nil NodNEG Knit Nil ShakeRH Raise Widen Tilt(left/right)TP Raise Widen Move upwardWH Knit Squint Move ForwardYN Raise Widen Move Forward668T.D.Nguyen and S.RanganathTable2.Types of grammatical marker sequences considered Sequence English sentence ASL signsTP AS I really want the book![BOOK]T P[WANT]AS TP NEG I don’t want the book.[BOOK]T P[WANT]NEGTP RH AS I know where the game is!It’s inSingapore.[GAME]T P[WHERE]RH [SINGAPORE]ASTP WH YN Where is the game?Is it in NewYork?[GAME]T P[WHERE]W H [NEW YORK]Y NUnidentified(Neutral)Topic Undefined RhetoricalFig.1.When the Rhetorical gesture is performed after a Topic gesture,the head will move from backward position to neutral position before tilting forward(movement epenthesis)while the brow still held raisedthe next;this is difficult to model due to its variability.Coarticulation refers to the appearance of a head gesture being influenced by adjacent gestures.There can also be asynchronization between head motion and facial feature movement. Movement epenthesis between grammatical markers is shown in Fig.1.Table3 shows examples of grammatical marker chains;any facial gesture video frame that does not contain one of the six grammatical marker classes is labeled as Unidentified.This is a generic class which includes gestures between two gram-matical markers,and also the neutral expression,which is usually present at the beginning of a sequence.Visually,the beginning and ending of an expression can be considered to co-incide with the beginning and ending of the head motion corresponding to that expression.However,while signing,movements of facial features like brows and eyes are independent and may evolve asynchronously with the head motion. This asynchronization adds to the uncertainty in identifying a facial gesture by using a combination of features from head motions and facial feature move-ments.An effective strategy to deal with this problem is to use multi-channel frameworks[17],where the classifier learns the correlations between the channels through supervised training.Movement epenthesis between grammatical markers also introduces additional variability.This is manifested through the head tending to move back to the neutral position before comfortably starting the next motion.Besides,if expre-sions have similar eye/brow movements,some subjects tend to hold the state established at one expression into the next expression,while others do not.This phenomenon will alter the temporal patterns of eye/brow movements and affect algorithm performance.The movements of the eyes and brows can be further affected by factors that are not related to facial gestures of interest:natural eyeRecognizing Continuous Grammatical Marker Facial Gestures669 Table3.Examples of four types of grammatical marker chains.The neutral expression shown in thefirst frame is considered to be an unidentified expression.An unidentified facial gesture can also be present between any two grammatical markers and can vary greatly depending on nearby grammaticalmarkers.Unidentified Topic UnidentifiedAssertionUnidentified Topic UnidentifiedNegationUnidentified Topic RhetoricalAssertionUnidentified Topic Wh question Unidentified Yes/No questionblinks,hand signs for adjectives such as HUNGRY or FAST involving added facial expressions.Moreover,unidentified gestures between facial gestures of interest are highly varied due to combinations of movement epenthesis and other effects.Thus,it will be ineffective to model the sequences using generative models like HMMs.A discriminative model may be more suited for this scenario,and we propose to use a2-layer CRF model to handle head motion and facial expression towards recognizing continuous grammatical markers.The use of a2-layer model is also motivated by the experimental data that we gathered,which showed that in spite of movement epentheses,head motions are more consistent than corresponding facial feature movements.2.2Layered Conditional Random Field ModelThe CRF is a discriminative probabilistic model proposed by Lafferty et al.[12] which can be trained to assign a sequence of predefined labels to a sequence of observations.Its evaluation function is composed of weighted potential functions which can utilize not only features extracted from the observations but also their interactions and temporal dependencies.In the linear-chain model,the probability of a label sequence y given an observation sequence x is computed as:p(y|x)=1Z(x)expTt=1⎛⎝Ni=1λi f i(y t,x)+Mj=1μj g j(y t,y t−1,x)⎞⎠(1)670T.D.Nguyen and S.RanganathTable4.Head labels used to train the CRF at thefirst layerbel Meaning1Neutral(Neu)Head at normal position2Forward(Fw)Head moves forward3Back from Forward(BfF)Head moves from forward position to neutral position 4Backward(Bw)Head moves backward5Back from Backward (BfB)Head moves from backward position to neutral posi-tion6Turn left(TL)Head turns left,usually a part of head shake 7Back from Turn left(BfTL)Head pose changes from leftward to frontal8Turn right(TR)Head turns right,usually a part of head shake 9Back from Turn right(BfTR)Head pose changes from rightward to frontal 10Move down(MD)Head moves down,usually a part of head nod11Back from Move down (BfMD)Head pose changes from downward to frontal,usually a part of head nod12Still Head is kept still13Forward left(FL)Head moves forward and slightly turns left14Back from Forward left (BfFL)Head pose changes from leftward to frontal and head moves from forward to neutral position15Forward right(FR)Head moves forward and slightly turns right16Back from Forward right (BfFR)Head pose changes from rightward to frontal and head moves from forward to neutral positionwhere f i and g j are potential functions that evaluate the interaction and tempo-ral dependencies among features,respectively.λi andμj are weights estimated from training data,and Z(x)is a normalization factor.It was shown[12]that the right hand side of Eq.1is a convex function parameterized byλi andμj,whose global optimum can be obtained by using iterative scaling algorithms or gradient-based methods.CRFs,which avoid the assumption of statistical independence of observations, have shown better performance than HMMs in many applications[12][14].We use a layered model of the chain CRF(Fig.2)to recognize continuous facial gestures in ASL.The probabilities of head motion labels are evaluated by a CRF in thefirst layer.These probabilities are passed to the second layer where other facial feature channels are also integrated.The second layer CRF is trained on these integrated features,to provide grammatical marker labels for frames in the test video sequences.Our observations show that the transition from one type of head motion to another mainly include movement epenthesis.Thus we choose to model move-ment epentheses explicitly,together with meaningful head motions.Currently, we have used16labels of head motions(both meaningful head motion and their movement epentheses)as described in Table4for all combinations of head mo-tions which occur in conjunction with the six grammatical markers of interest.Recognizing Continuous Grammatical Marker Facial Gestures 671matical markers in sign language Fig.3.Feature points of interestFig.4.Distance features used In manually annotating the frames,besides the head motion label,each video frame in the data set is also labeled with one of seven facial gestures:AS ,NEG ,RH ,TP ,RH ,WH ,YN ,and Und .The label Und is assigned to frames with unidentified expressions.As shown in Table 4,head motions with labels such as “Back from X”are defined to explicitly model movement epentheses.Exceptional cases are labels 7,9,and 11which are constituents of multi-part head motions:head shake and head nod.The N eutral label appears mostly at the beginning of the video sequences.During facial gestures,the head does move past the neutral position but does not stop.The frames in which the head is temporarily at the neutral position is also annotated with the N eutral label.The label S till plays an important role in segmenting meaningful head motions and their movement epentheses (Back from X)because there is usually a short pause (or even long pause)between the meaningful head motion and its “Back from”movement.Motion of the head and facial features are obtained from the tracked feature points (shown in Fig.3)using an enhanced version of the robust tracking al-gorithm developed by the authors [16].The feature points are placed at both rigid and non-rigid facial locations,and distances between them are extracted and used for recognition.These distances (shown in Fig.4)are,(a)five eyebrow parameters:Left inner brow height (B IL ),Right inner brow height (B IR ),Left middle brow height (B ML ),Right middle brow height (B MR ),Distance between brows (B B );and (b)two eye parameters:Left eye height (summation of E BL and E T L ),Right eye height (summation of E BR and E T R ).A reference line is defined as the line passing through the two inner eye corners,and the height pa-rameters are the perpendicular distances of the feature points from this line.All distance parameters are normalized with respect to their corresponding values in the first frame to remove scaling effects across video sequences.672T.D.Nguyen and S.Ranganath1102030405060708090100 Fig.5.Frames in a test sequence containing the facial gesture chain TP RH AS.The frame index is shown below each image.Blue dots at facial features of interest are our tracking results.To recognize head motions,tracks of non-deformable facial feature locations, namely,the two inner eye corners(E L3,E R3)and the middle of the nose(N2), are used to define three features;S M(the area of the triangle formed by the above three locations in each frame),and C M x,C M y(components of the2D motion vector1C M of the center of gravity of the triangle).S M and C M are normalized by the distance E M0between the two inner eye corners in thefirstframe:C n M t=C M tE M0and S n M t=S M tE M02.These three features form the feature vec-tor(at each frame)for thefirst CRF layer to evaluate probabilities of different head motions.The feature vector(at each frame)of the second CRF layer for recognizing continuous grammatical markers thus has23elements:16probabil-ities of head motions and7distance ratios computed from the eyes and brows’tracked features.3Experiments and ResultsVideos of natural sign language facial gestures of interest were recorded by pro-viding deaf signers(from the Deaf and Hard-of-Hearing Foundation of Singa-pore)with appropriate signing scripts for sentences.Each English sentence in the script was signed in ASL with hand signs and corresponding facial gestures. These sentences were created or adapted from ASL resources(e.g.[1]).A sub-ject signed each sentence ten times.As mentioned in Section2,the data includes four types of grammatical marker chains described in Table2.All six grammatical markers listed in Table1are present in the data set together with the16types of head motion described in Table4.For evaluating the feasibility of our proposed recognition method,data from three subjects was used for experiments.The data set included a total of129video sequences divided into93video sequences for training(an average of seven sequences per subject for each of the four grammatical marker chains)and36for testing(about 3sequences per subject per chain).Each video frame was manually transcribed to have two labels,one for head motion,and the other for grammatical marker, both identified based on visual observation and the signing script.The training set was used to train both CRF layers of the model:head motion layer and grammatical marker layer.Recognition accuracy for grammatical markers was measured by two methods: frame based and label-aligned.In the frame-based method,the label assigned 1Motion vector vt+1=(x t+1,y t+1)−(x t,y t).Recognizing Continuous Grammatical Marker Facial Gestures673 for each frame is compared with the corresponding human annotated label.In the label-aligned method,the frame labels of each sequence are reduced such that consecutive frames with the same label are replaced by a single label.The two reduced sequences of labels are aligned using the Needleman-Wunsch al-gorithm[18].The number of matches,insertions,deletions,and changed labels are then obtained.Insertions are labels output by the classifier,which do not appear in the corresponding annotated data.Deletions are labels which are not recognized by the classifier while they appear in the annotated data.An experiment was conducted to evaluate the performance of the proposed model.Thefirst CRF layer for head motion was trainedfirst.The head motion probabilities output by this trained CRF was used as a part of the training vector for the CRF at the second layer.The two CRF layers were trained using the scaled conjugate gradient algorithm with the CRF Toolbox[19].Frames from a video sequence in the test set are shown in Fig.5,where the sequence of facial gestures corresponds to TP RH AS.Fig.6shows the probability output of thefirst layer for the16head motion labels described in Table4.As mentioned in Section2,the head tends to move past the neutral position before starting a new motion.In the last10frames in Fig.6,there is confusion due to ambiguous head motions at the end of the signed sentence. Fig.7shows the probability for the grammatical markers output by the2-layer CRF classifier.Seven probabilities including six for grammatical markers and one for unidentified expression are obtained at each frame.Fig.7shows that the second CRF layer,which is trained with output from thefirst layer,can tolerate the ambiguity of head motions in recognizing continuous grammatical markers.The average frame-based grammatical marker recognition rate using the com-plete2-layer CRF model was80.82%.The corresponding confusion matrix is shown in Table5which shows that most of the confusions are between any grammatical marker and the unidentified expression.Particularly,frame-based label confusions occur at the boundary between facial gestures where ambigu-ous head motions and asynchronous movements of facial features are present. This makes even manual annotation of consecutive frames into different facial gestures difficult.The label-aligned method of computing accuracy reveals more about the ca-pability of the layered CRF for recognizing continuous grammatical markers by discounting unavoidable confusions during transitions between facial gestures. Table5can be augmented with insertion and deletion entries to obtain the ex-tended confusion matrix C from which precision and recall rates are computedas:P recision=MatchMatch+Change+Insert and Recall=MatchMatch+Change+Delete,wherefor marker i,Match rate=C(i,i),Change rate=j/∈{i,Insert,Delete}C(i,j),In-sertion rate=C(i,Insert),Deletion rate=C(i,Delete),and C(i,j)is the value at row i and column j of the extended confusion matrix.Label-aligned results were93.76%for precision and84.54%for recall.The extended confusion matrix for this evaluation is shown in Table.6.The precision rate appears quite reasonable given the complexity of the problem.However,674T.D.Nguyen and S.RanganathFig.6.The probability outputs of thefirst layer CRF trained to recognize16types of head motion.The color bar at the top is the human annotated head motion label for this video sequence.The curve and barwith the same color are associated with the same head bels for last10frames are ambigu-ous due to ambiguous head motions at the end of the signed sentence.Fig.7.The probabilities of the gram-matical markers,output by the second CRF layer trained using head motion probability output(shown in Fig.6) from thefirst layerthe lower recall rate hints that the layered CRF is less sensitive to change of facial gestures in video sequences.This may be improved with more descriptive features for head motion and facial feature movements.As a comparison,the results obtained in this experiment were quite close to the results we obtained in another experiment where the head motion labels were assumed known(the human annotated labels)and were input to the second layer CRF(rather than using thefirst layer outputs).In this experiment,precision rate of94.54%and recall rate of90.78%were obtained for recognizing grammatical markers.Our recent results show that the layered-CRF model outperforms the linear chain CRF and the layered HMM models.Table5.Confusion matrix for labeling grammatical markers with the proposed model. The average frame-based recognition rate is80.82%.Und AS NEG RH TP WH YN59.627.60 3.09 6.659.2612.11 1.669.6287.460 2.920000.98097.070 1.950010.780089.220003.06 1.31 1.17 3.3591.1079005.619.3500084.580.4627.8410.31000 5.1656.70Recognizing Continuous Grammatical Marker Facial Gestures675 Table6.Extended confusion matrix for label-based facial gesture recognition result (%)using2-layer CRFUN AS NEG RH TP WH YN Insert Delete Precision Recall UN68.970.000.000.000.000.000.00 3.4527.5995.2471.43AS 5.2684.210.00 5.260.000.000.000.00 5.2688.8984.21 NEG0.000.001000.000.000.000.000.000.00100100RH0.000.000.001000.000.000.000.000.00100100TP0.000.000.000.0091.670.000.000.008.3310091.67WH0.0011.110.000.000.0088.890.000.000.0088.8988.89YN0.0011.110.000.000.000.0055.560.0033.3383.3355.56 Average93.7684.544ConclusionIn this paper,we addressed the problem of recognizing continuous facial ges-tures in sign language video.A2-layer CRF was proposed for recognizing six common grammatical markers in ASL sentences.Thefirst layer was trained for evaluating head motions and the second layer was trained for segmenting and recognizing facial gestures using the output from thefirst layer and measure-ments of facial feature movements.Data was collected using an experimental set up for capturing natural facial gestures without a forced“neutral”state between gestures.The performance of the complete2-layer CRF model yielded precision rate of93.76%,and recall rate of85.54%for recognizing the six types of con-tinuously signed grammatical markers.These encouraging results show that the proposed2-layer model is a viable scheme for recognizing facial gestures in sign language.In the near future,we propose to enhance the robustness of the model by incorporating more descriptive features for identifying head motions.We will also conduct more evaluations and comparisons with other methods.Other non-manual signals will be considered for further development of the system. Acknowledgement.This work is partially support by project grant NRF2007IDM-IDM002-069on“Life Spaces”from the IDM Project Office, Media Development Authority of Singapore.References1.Baker,C.,Cokely,D.:American Sign Language:A teacher’s Resource Text onGrammar and Culture.Clerc Books,Gallaudet University Press,Wasington D.C.(1980)2.Ong,S.,Ranganath,S.:Automatic Sign Language Analysis:A Survey and theFuture Beyond Lexical Meaning.IEEE Transactions on Pattern Analysis and Machine Intelligence27,873–891(2005)3.Vogler,C.,Goldenstein,S.:Facial movement analysis in ASL.Journal on UniversalAccess in the Information Society6,363–374(2008)676T.D.Nguyen and S.Ranganath4.Neidle,C.,Nash,J.,Michael,N.,Metaxas,D.:A Method for Recognition of Gram-matically Significant Head Movements and Facial Expressions,Developed Through Use of a Linguistically Annotated Video Corpus.In:Proceedings of the Language and Logic Workshop,Formal Approaches to Sign Languages,European Summer School in Logic,Language,and Information(ESSLLI2009),Bordeaux,France (2009)5.Pantic,M.,Rothkrantz,L.J.:Automatic Analysis of Facial Expressions:The Stateof the Art.IEEE Transactions on Pattern Analysis and Machine Intelligence22, 1424–1445(2000)6.Fasel,B.,Luettin,J.:Automatic facial expression analysis:a survey.Pattern Recog-nition36,259–275(2003)7.Murphy-Chutorian,E.,Trivedi,M.M.:Head pose estimation in computer vision:Asurvey.IEEE Transactions on Pattern Analysis and Machine Intelligence31(2009) 8.Black,M.,Yacoob,Y.:Recognizing facial expressions in image sequences usinglocal parameterized models of image motion.International Journal of Computer Vision25,23–48(1997) Torre,F.D.,Campoy,J.,Ambadar,Z.,Cohn,J.F.:Temporal Segmentation ofFacial Behavior.In:International Conference on Computer Vision(2007)10.Matthews,I.,Baker,S.:Active AppearanceModels Revisited.International Journalof Computer Vision60,135–164(2004)11.Cohen,I.,Sebe,N.,Garg,A.,Chen,L.S.,Huang,T.S.:Facial expression recognitionfrom video sequences:temporal and static puter Vision and Image Understanding91,160–187(2003);Special Issue on Face Recognitionfferty,J.,McCallum,A.,Pereira,F.:Conditional randomfields:Probabilisticmodels for segmenting and labelling sequence data.In:International Conference on Machine Learning(2001)13.Kanaujia,A.,Metaxas,D.:Recognizing Facial Expressions by Tracking FeatureShapes.In:International Conference on Pattern Recognition,Hong Kong,China (2006)14.Quattoni, A.,Wang,S.B.,Morency,L.P.,Collins,M.,Darrell,T.:HiddenConditional Random Fields.IEEE Transactions on Pattern Analysis and Machine Intelligence29,1848–1852(2007)15.Chang,K.Y.,Liu,T.L.,Lai,S.H.:Learning partially-observed hidden conditionalrandomfields for facial expression recognition.In:IEEE Conference on Computer Vision and Pattern Recognition,pp.533–540(2009)16.Nguyen,T.D.,Ranganath,S.:Tracking facial features under occlusions andrecognizing facial expressions in sign language.In:IEEE International Confer-ence on Automatic Face&Gesture Recognition,Amsterdam,Netherlands,pp.1–7(2008)17.Oliver,N.,Horvitz,E.,Garg,A.:Layered representations for learning and infer-ring office activity from multiple sensory puter Vision and Image Understanding96,163–180(2004)18.Needleman,S.B.,Wunsch,C.D.:A general method applicable to the search forsimilarities in the amino acid sequence of two proteins.Journal of Molecular Biology48,443–453(1970)19.Schmidt,M.,Swersky,K.:Conditional Random Field Toolbox for Matlab,http://www.cs.ubc.ca/~murphyk/Software/CRF/crf.html。