Human Facial Modeling and Animation

Simulation of Facial Muscle Actions Based onRational Free Form DeformationsPrem Kalra1, Angelo Mangili2, Nadia Magnenat Thalmann1 and Daniel Thalmann21 MIRALab, University of GenevaSwitzerland2 Computer Graphics LabSwiss Federal Institute of TechnologyLausanne, SwitzerlandABSTRACTThis paper describes interactive facilities for simulating abstract muscle actions using Rational Free Form Deformations (RFFD). The particular muscle action is simulated as the displacement of the control points of the control-unit for an RFFD defined on a region of interest. One or several simulated muscle actions constitute a Minimum Perceptible Action (MPA), which is defined as the atomic action unit, similar to Action Unit (AU) of the Facial Action Coding System (FACS), to build an expression.Keywords : Facial Animation, Expressions, Rational Free Form Deformations.1. IntroductionHuman Facial Modeling and AnimationThe face is a small part of a human, but it plays an essential role in communication. People look at faces for clues to emotions or even to read lips. It is a particular challenge to imitate these few details. An ultimate objective therefore is to model human facial anatomy exactly including its movements to satisfy both structural and functional aspects of simulation. However, this involves many problems to be solved concurrently. The human face is a very irregular structure, varying from person to person. The problem is further compounded with interior details such as muscles, bones, tissues, and motion which involves complex interactions and deformations of different facial features.The Basic Facial Animation ModelsThere has been extensive research done on basic facial animation and several models have been proposed. Early models proposed by Parke (1975, 1982), used a combination of digitized expressions and linearinterpolation of features such as eyelids, eyebrows and jaw-rotations. Motions are described as a pair of numeric tuples which identify the initial frame, final frame, and interpolation.Platt and Badler (1981) based their facial animation system on the Facial Action Coding System (FACS) developed by Ekman and Friesen (1975). In FACS, facial expression is specified in terms of Action Units (AU) which are single muscles or clusters of muscles. In their model, skin is the outermost level represented by a set of 3D points defining a surface which can move. Bones represent innermost level which cannot be moved. Muscles are groups of points with elastic arcs between the two levels.In the model of Waters (1987), muscles are geometric deformation operators which the user places on the face in order to simulate the contraction of real muscles. Two types of muscles are created: linear/parallel muscles that pull, and sphincter muscles that squeeze. These muscles are independent of the underlying bone structure, which makes the muscle model independent of specific face topology. The control parameters are based on FACS.Nahas et al. (1988) proposed a method based on B-splines. A digitizing system is used to register the position of certain facial points which are organized in a matrix. The matrix is used as a set of control points for a 5-dimensional bicubic B-spline surface. Motion of the face is obtained by moving these control points. Magnenat-Thalmann et al. (1988) provided another approach to simulate a muscle action by using a procedure called an Abstract Muscle Action (AMA) procedure. Each AMA has an associated procedure with a set of parameters which can be used to control the motion of vertices composing the face. By combining the facial parameters obtained by the AMA procedures in different ways, complex entities corresponding to the concept of facial expressions, can be constructed.Terzopoulos and Waters (1990) extended the Waters model, using three layered deformable lattice structures for facial tissues. The three layers correspond to the skin, the subcutaneous fatty tissue, and the muscles. The bottom surface of the muscle layer is attached to the underlying bone. The model uses a physically-based technique.Parke (1991) reviews different parameterization mechanism used in different previously proposed models and introduces the future guidelines for ideal control parameterization and interface. Ideal parameterization is in fact a universal parameterization which would enable all possible individual faces with all possible expressions and expression transitions.Recently several authors have provided a new facial animation technique which is based on the information derived from human performances (deGraf 1989; Williams 1990; Terzopoulos and Waters 1991). The information extracted is used for controlling the facial animation. These performance driven techniques provide a very realistic rendering and motion of the face. Kurihara and Arai (1991) introduced a new transformation method for modeling and animating the face using photographs of an individual face. The transformation method enables the movement of points in the skin mesh to be determined by the movement of some selected control points. Texture mapping is used to render the final image.In this paper, we describe another approach to deform the facial skin surface using free form deformations. It is evident that the skin being supported by bone and multiple layers of muscle produces an enormous combination of movements. The effort here is not to exactly simulate the neurons, veins, muscles and bone joints but to design a model with a few dynamic parameters that emulate the basic movements.Speech, Emotion and SynchronizationMost of the facial movements result from either speech or the display of emotions; each of these has its own complexity. However, both speech and emotions need a higher level specification of the controlling parameters. The second level parameterization used in speech animation is usually in terms of phonemes. Aphoneme is a particular position of the mouth during a sound emission. These phonemes in turn control the lower level parameters for the actual deformations. Similarly, emotion is a sequence of expressions, and each expression is a particular position of the face at a given time. In order to have natural manipulation of speech and emotion there is a need of some synchronization mechanism.Efforts for lip synchronization and speech automation initiated with the first study of Lewis and Parke (1987). In their approach, the desired speech is spoken and recorded. The recording is then sampled and analyzed to produce a timed sequence of pauses and phonemes. Hill et al. (1988) have introduced an automatic approach to animate speech using speech synthesized by rules. Magnenat-Thalmann et al. (1987) have used lip synchronization based on AMA procedures. A collection of multiple tracks is used, where each track is a chronological sequence of keyframes for a given facial parameter. Tracks are independent but can be mixed in the same way as sound is mixed in a sound studio. This approach was used in the film Rendez-vous àMontréal (1987). However, the process of synchronization is manual and must be performed by the animator. In another system, Kalra et al. (1991) introduce a multi-layer approach where, at each level, the degree of abstraction increases. This results in a system where complexity is relatively low from the point of view of the animator. The defined entities correspond to intuitive concepts such as phonemes, expressions, words, emotions, sentences and eye motion, which make them natural to manipulate. A manipulation language, HLSS, is provided to ensure synchronization while manipulating these entities.In this paper we present an interface for building up expressions which may be pre-defined and saved for further use in a sequence with phonemes and emotions.2. Simulation of Muscle ActionsRational Free Form DeformationsFree form deformation (FFD) is a technique for deforming solid geometric models in a free form manner (Sederberg and Parry 1986). It can deform surface primitives of any type or degree, for example, planes, quadrics, parametric surface patches or implicitly defined surfaces. Physically, FFD corresponds to deformations applied to an imaginary parallelepiped of clear, flexible plastic in which are embedded the object(s) to be deformed. The objects are also considered to be flexible so that they are deformed along with the plastic that surrounds them. FFD involves a mapping from R3 to R3 through a trivariate tensor product Bernstein polynomial. Mathematically, imposing a local coordinate system (S,T,U) on a parallelepiped region with origin at X0, a point X has (s,t,u) coordinates in this system such thatX = X0 + sS + tT + uU(1)A grid of control points P ijk (i=0 to l, j=0 to m, k=0 to n) is imposed on the parallelepiped. The location of these points is defined asP ijk = X0 + ilS +jmT +knU (2)The (s,t,u) coordinates of X can be found by the following equations:s = TxU.(X-X 0)TxU.S , t = SxU.(X-X 0)SxU.T , u = SxT.(X-X 0)SxT.U (3)For any point interior to the parallelepiped, 0<s<1, 0<t<1, 0<u<1, the deformation is specified by moving the control point(s) from their undisplaced latticial position. The deformed position X' of a point X is computed from the following equation :X' = ∑i=0l ∑j=0m ∑k=0n P ijk B l i (s) B m j (t) B n k (u)(4)where B l i (s) , B m j (t) , B n k (u) are the Bernstein polynomials defined asB l i (s) = (l i ) (1-s)l-i s i ,B m j (t) = (m j ) (1-t)m-j t j ,B n k (u) = (n k ) (1-u)n-k u k (5)As an extension to the basic FFDs, we provide the option of including rational basis functions in the formulation of deformation. The rational basis functions allow incorporation of weights defined for each control point (W ijk ) in the parallelepiped grid. With new formulation the equation (4) changes as follows :X' = ∑i=0l ∑j=0m ∑k=0n P ijk W ijk B l i (s)B m j (t)B n k (u) ∑i=0l ∑j=0m ∑k=0n W ijk B l i (s)B m j (t)B n k (u)(6)The advantage of using rational FFDs (RFFDs), is that it provides one more degree of freedom of manipulating the deformations by changing the weights at the control points. When the weights at each control point are unity, the deformations are equivalent to the basic FFD. Figure 1 shows an illustration where the deformations are accomplished by changing the position of the control point (Figure 1(b)) and by changing the weight at the control point (Figure 1(c)). It is possible to combine the two types. Figure 2shows a few examples of facial deformations.Figure 1 : Rational Free Form Deformations. a. Undeformed surface. b. Displacement of control point. c.Changing the weight of control pointFigure 2 : Examples of Facial deformationsRegion Based ApproachTo simulate the muscle action on the skin surface of human face, we define regions on the face mesh which correspond to the anatomical description of the facial region on which a muscle action is desired. A parallelepiped control unit then can be defined on the region of interest. The deformations which are obtained by actuating muscles to stretch, squash, expand and compress the inside volume of the facial geometry, are simulated by displacing the control point and by changing the weights of the control points of the control-unit. The region inside the control-unit deforms like a flexible volume, corresponding to the displacement and the weights of the control points.Displacing a control point is analogous to adding a muscle vector to the control-unit. As opposed to the Waters model where the animator has to specify the muscle vectors in the form of some geometric operators, here the displacement of the control points gives rise to similar effects. Specifying the displacement of the control point is, however, more intuitive and simpler to simulate the muscle vectors. In addition, the result matches the natural notion of muscles acting on that region. For example, a depressor muscle would need to squash the control point inside the control-unit, and a pulling muscle would pull the control points away from the control-unit.In order to propagate the deformations of regions to the adjoining regions, linear interpolation can be used to decide the deformation of the boundary points. Although higher order interpolation schemes can also be used, movements in the face are rather small and arising higher ordered discontinuities may not affect the visual aspects so much.Physical Characteristics of SkinPhysical properties of the skin surface such as mass, stiffness and elasticity can also be incorporated when applying the deformations. The deformed point in that case can be given as :X'' = (X' - X)xp + X;whereX''is the final deformed position of the point,X'is the position obtained after the FFD,X is the undeformed position,p factor for the surface properties.Association of physical properties provides a means of further controlling the deformations in a more natural way.Interactive FacilitiesThe construction of the face is based on one of the techniques described by Paouri et al. (1991). The surface of a human face, an irregular structure, is considered a polygonal mesh. Leblanc et al. (1991) have introduced a methodology for interactive sculpting using a six degree of freedom interactive input device called the Spaceball. When used in conjunction with a common 2D mouse, full three dimensional user interaction is achieved, with the Spaceball in one hand and the mouse in the other. In this way, the user not only views the object from different angles but can also perform various operations from every angle interactively. We use this methodology for our system of facial expressions.The building up of regions is accomplished by interactive selection of polygons. These regions on the human face generally correspond to their anatomical descriptions, such as nose, lips, eyes etc. The selection process can be further hastened by attaching the color attributes of the polygons. The magnitude and direction of the muscle pulled is interactively adjusted by changing the position and weight of the control points on the control-unit of a region.Certain points in a defined region can be "anchored" meaning that they will not deform when the deformations are applied on the region. The physical characteristics for the points of the facial mesh can also be set interactively.3. The Expression EditorMinimum Perceptible ActionsA minimal perceptible action is a basic facial motion parameter. The range of this motion is normalized between 0 and 1 or -1 and 1. An instance of the minimal action in the general form is given as follows :<MPA name> <frame number> <intensity>Each MPA has a corresponding set of visible facial features such as movement of eyebrows, movements of jaw, or mouth and others which occur as a result of contracting and pulling of muscles associated with the region. MPA's also include non-facial muscle actions such as nods and turns of the head and movement of the eyes. An MPA can be considered as an atomic action unit similar to the AU of the FACS, execution of which results in a visible and percetible variation of a face.Table 1 gives the list of the presently available MPA with their range of intensity.Table 1. Minimum Perceptible ActionsMPA name IntensityRaise_eyebrow-1 to 1Squeeze_eyebrow 0 to 1Move_horizontal_eye-1 to 1Move_vertical_eye-1 to 1Close_upper_eyelids-1 to 1Close_lower_eyelids-1 to 1Stretch_nose-1 to 1Raise_nose 0 to 1Raise_upper_lip 0 to 1Pull_upper_lip 0 to 1Lower_lower_lip 0 to 1Pull_lower_lip 0 to 1Raise_corner_lip 0 to 1Lower_corner_lip 0 to 1Stretch_corner_lip 0 to 1Mouth_beak 0 to 1Zygomatic_cheek 0 to 1Puff_cheek-1 to 1Nod_head-1 to 1Turn_head-1 to 1Roll_head-1 to 1Expressions and PhonemesExpressions and phonemes in our system are considered facial snapshots: a particular position of the face at a given time. For phonemes, only the lips are considered during the emission of sound. A facial snapshot is made up of one or more MPAs. The set of MPAs included is general enough to account for most possible facial expressions. The basic MPAs can be combined to give rise to the higher level emotions corresponding to familiar expressions such as anger, fear, and surprise. A generic expression can be represented as follows : [expression <name>[mpa <name1> intensity <i1>][mpa <name2> intensity <i2>]--]Figure 3 shows some of the expressions created. One can create natural expressions as well as some unnatural and ideosyncratic expressions using the editor. The intensity of an expression is directly influenced by the intensity of contained MPAs. A strong and feeble expression can be created by appropriately changing the intensities of the associated MPAs.Figure 3 : Some expressions created using Expression Editor.By design, the Expression Editor is independent of the low level realization of muscles and their actions. With this independence the low level implementation of an expression could be as simple as rotation or scale of a region or as complex as a 3D finite element model of the region. Using this scheme it would even be possible to use entirely different simulation models for each MPA separately without effecting the high level control provided by the Expression Editor.Interactive EditingFacial snapshots representing expressions and phonemes can be built up interactively. Users can construct and save static expressions, and thus build up a library of pre-defined expressions. There are options such as Select, Add, and Save. For example, Add enables a user to start interactively and build an expression from the same range of MPAs used in generating animation sequence. When saved, the expression becomes available for selection as part of the pre-defined expression database.Having built-up an expression sequence it is possible to read the script file, and when the script is running the snapshot of the expressive face is saved in a file. The sequence of these files then can be seen in the preview program outside this software. The entries inside the script file are of the form :[mpa <name><frame_no> <intensity> <comments>-- -- --][mpa <name><frame_no> <intensity> <comments>-- -- --]--Figure 4 shows an interactive session.Figure 4 : Interactive session of Expression Editor.4. RenderingUsual shading techniques tend to give a rather artificial and plastic look to a face. For realistic rendering, we employ texture mapping using 2D photographs. Texture mapping is a relatively efficient way to create the appearance of complex surface detail without having to go through the tedium of modeling and rendering every detail of a 3D surface.An interactive tool is provided for matching the 3D facial features on a given photograph. Only a few control points of the 3D facial model are selected. These points are then projected onto a plane or cylinder. The viewing parameters for the 3D model can be interactively adjusted to correspond with the given photograph. Delaunny triangulation is then used to connect these points in a triangular manner. These points can be moved and displaced interactively on the picture. Once the desired correspondence for these feature points is established, an interpolation scheme using barycentric coordinates in triangular domain is used to acquire the texture coordinates (u,v) for all the points of the 3D model. Attributes such as "boundary" and "control" are attached to the points to provide some constrains to compute the texture coordinates of these marked points. For example, the points coming from two different groups of control points and marked "boundary" share the same texture coordinates. In addition it allows the use of different pictures for different groups of points if needed. To blend two views of a picture we use linear interpolation for a given band width. By using texture mapping the quality of rendering improves considerably. Figure 5 and 6 show some of the resulting images.5. ImplementationThe present facial animation system is part of a new system for the intelligent animation of human characters in their environment, developed in the C language on IRIS Silicon Graphics networks at the Swiss Federal Institute of Technology in Lausanne and the University of Geneva. The system uses the fifth Dimension Toolkit (Turner et al. 1991) for building up the user interface.6. Future DevelopmentsThe diversity of facial forms in terms of sex, age, and race is enormous. It is these forms that allow us to recognize individuals and send complex non-verbal signals to one another. Our present system will be extended to define some global conformation shapes of faces and make a database for such shapes. These shapes will act as a much better starting point to convert a generic face into a specific face by changing the soft features of the face.We plan to use a direct phoneme interface using a synthesizer keyboard in conjunction with a NeXT computer in order to automate the speech production.7. ConclusionThe facial expression simulation system described here is designed to simplify the task of specifying, and thus controlling, facial expressions. Simplifying expression control requires minimizing the number of parameters which have to be specified and making those parameters intuitively easy to control. The Expression Editor allows the user to specify expressions at a higher level without having detailed knowledge of the anatomy of the human face.8. AcknowledgmentsThe research was supported by le Fonds National Suisse pour la Recherche Scientifique. The authors are grateful to Agnes Daldegan-Balduino for the creation of a few pictures and to Hans Martin Werner for reviewing text.9. ReferencesdeGraf B (1989) in State of the Art in Facial Animation, SIGGRAPH '89 Course Notes No. 26, pp. 10-20.Ekman P, Friesen WV (1975), Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, Printice-HallHill DR, Pearce A, Wyvill B (1988), Animating Speech: An Automated Approach Using Speech Synthesised by Rules, The Visual Computer, Vol. 3, No. 5, pp. 277-289.Kalra P, Mangili A, Magnenat-Thalmann N, Thalmann D (1991) SMILE : A Multilayered Facial Animation System, Proc IFIP WG 5.10, Tokyo, Japan (Ed Kunii Tosiyasu L) pp. 189-198.Kurihara T, Arai K (1991), A Transformation Method for Modeling and Animation of the Human Face from Photographs, Proc. Computer Animation '91 Geneva, Switzerland, (Eds Magnenat-Thalmann N and Thalmann D), pp. 45-57.LeBlanc A, Kalra P, Magnenat-Thalmann N, Thalmann D (1991), Sculpting with the "Ball & Mouse"Metaphor, Proc. Graphics Interface '91, Calgary, Canada, pp. 152-159.Lewis JP, Parke FI (1987), Automated Lipsynch and Speech Synthesis for Character Animation, Proc. CHI '87 and Graphics Interface '87, Toronto, pp. 143-147.Magnenat-Thalmann N, Primeau E, Thalmann D (1988), Abstract Muscle Action Procedures for Human Face Animation, The Visual Computer, Vol. 3, No. 5, pp. 290-297.Magnenat-Thalmann N, Thalmann D (1987), The Direction of Synthetic Actors in the film Rendez-vous àMontréal, IEEE Computer Graphics and Applications, Vol. 7, No. 12, pp. 9-19.Nahas M, Huitric H, Saintourens M (1988), Animation of a B-Spline Figure, The Visual Computer, Vol. 3, No. 5, pp. 272-276.Paouri A, Magnenat-Thalmann N, Thalmann D (1991), Creating Realistic Three-Dimensional Human Shape Characters for Computer-Generated Films, Proc. Computer Animation '91, Geneva, Switzerland, (Eds Magnenat-Thalmann N and Thalmann D), pp. 89-99.Parke FI (1975), A Model for Human Faces that allows Speech Synchronized Animation, Computer and Graphics, Pregamon Press, Vol. 1, No. 1, pp. 1-4.Parke FI (1982), Parametrized Models for Facial Animation, IEEE Computer Graphics and Applications, Vol. 2, No. 9, pp. 61-68.Parke FI (1991), Control Parameterization for Facial Animation, Proc. Computer Animation '91, Geneva, Switzerland, (Eds Magnenat-Thalmann N and Thalmann D), pp. 3-13.Platt S, Badler N (1981), Animating Facial Expressions, Proc SIGGRAPH '81, pp. 245-252.Sederberg TW, Parry SR (1986), Free Form Deformation of Solid Geometric Models, Proc. SIGGRAPH '86, pp. 151-160.Terzopoulos D, Waters K (1990) Physically Based Facial Modeling, Analysis, and Animation, Visualization and Computer Animation, Vol. 1, No. 2, pp. 73-80.Turner R, Gobbetti E, Balaguer F, Mangili A, Thalmann D, Magnenat-Thalmann N, An Object-Oriented Methodology Using Dynamic Variables for Animation and Scientific Visualization, CG International '90, Springer Verlag, pp. 317-328.Waters K (1987), A Muscle Model for Animating Three Dimensional Facial Expression, Proc SIGGRAPH '87, Vol. 21, No. 4, pp. 17-24.Williams L (1990), Performance Driven Facial Animation, Proc SIGGRAPH '90, pp. 235-242.Captions for color pictures Figure 5 : Facial expression after texture mapping. Figure 6 :Facial expression after texture mapping。