Computer facial animation

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Computer facial animation" – news · newspapers · books · scholar · JSTOR (February 2010) (Learn how and when to remove this template message)

Computer facial animation is primarily an area of computer graphics that encapsulates methods and techniques for generating and animating images or models of a character face. The character can be a human, a humanoid, an animal, a legendary creature or character, etc. Due to its subject and output type, it is also related to many other scientific and artistic fields from psychology to traditional animation. The importance of human faces in verbal and non-verbal communication and advances in computer graphics hardware and software have caused considerable scientific, technological, and artistic interests in computer facial animation.

Although development of computer graphics methods for facial animation started in the early-1970s, major achievements in this field are more recent and happened since the late 1980s.

The body of work around computer facial animation can be divided into two main areas: techniques to generate animation data, and methods to apply such data to a character. Techniques such as

History

Human facial expression has been the subject of scientific investigation for more than one hundred years. Study of facial movements and expressions started from a biological point of view. After some older investigations, for example by John Bulwer in the late 1640s, Charles Darwin's book The Expression of the Emotions in Men and Animals can be considered a major departure for modern research in behavioural biology.

Computer based facial expression modelling and

One of the most important attempts to describe facial movements was Facial Action Coding System (FACS). Originally developed by Carl-Herman Hjortsjö^[1] in the 1960s and updated by Ekman and Friesen in 1978, FACS defines 46 basic facial Action Units (AUs). A major group of these Action Units represent primitive movements of facial muscles in actions such as raising brows, winking, and talking. Eight AU's are for rigid three-dimensional head movements, (i.e. turning and tilting left and right and going up, down, forward and backward). FACS has been successfully used for describing desired movements of synthetic faces and also in tracking facial activities.

The early-1980s saw the development of the first physically based muscle-controlled face model by Platt and the development of techniques for facial caricatures by Brennan. In 1985, the animated short film Tony de Peltrie was a landmark for facial animation. This marked the first time computer facial expression and speech animation were a fundamental part of telling the story.

The late-1980s saw the development of a new muscle-based model by

• Polar Express by Imageworks where hundreds of motion points were captured. This film was very accomplished and while it attempted to recreate realism, it was criticized for having fallen in the 'uncanny valley
  ', the realm where animation realism is sufficient for human recognition and to convey the emotional message but where the characters fail to be perceived as realistic. The main difficulties of motion capture are the quality of the data which may include vibration as well as the retargeting of the geometry of the points.
• Markerless motion capture aims at simplifying the motion capture process by avoiding encumbering the performer with markers. Several techniques came out recently leveraging different sensors, among which standard video cameras, Kinect and depth sensors or other structured-light based devices. Systems based on structured light may achieve real-time performance without the use of any markers using a high speed structured light scanner. The system is based on a robust offline face tracking stage which trains the system with different facial expressions. The matched sequences are used to build a person-specific linear face model that is subsequently used for online face tracking and expression transfer.
• Audio-driven techniques are particularly well fitted for speech animation. Speech is usually treated in a different way to the animation of facial expressions, this is because simple
  neural nets
  to transform audio parameters into a stream of control parameters for a facial model. The advantage of this method is the capability of voice context handling, the natural rhythm, tempo, emotional and dynamics handling without complex approximation algorithms. The training database is not needed to be labeled since there are no phonemes or visemes needed; the only needed data is the voice and the animation parameters.
• keyframe animation process a control rig is used by the animation. The control rig represents a higher level of abstraction that can act on multiple morph targets
  coefficients or bones at the same time. For example, a "smile" control can act simultaneously on the mouth shape curving up and the eyes squinting.

Applying facial animation to a character

The main techniques used to apply facial animation to a character are: 1.) morph targets animation, 2.) bone driven animation, 3.) texture-based animation (2D or 3D), and 4.) physiological models.

• Morph targets (also called "blendshapes") based systems offer a fast playback as well as a high degree of fidelity of expressions. The technique involves modeling portions of the face mesh to approximate expressions and visemes and then blending the different sub meshes, known as morph targets or blendshapes. Perhaps the most accomplished character using this technique was Gollum, from The Lord of the Rings. Drawbacks of this technique are that they involve intensive manual labor and are specific to each character. Recently, new concepts in 3D modeling have started to emerge. Recently, a new technology departing from the traditional techniques starts to emerge, such as Curve Controlled Modeling^[3] that emphasizes the modeling of the movement of a 3D object instead of the traditional modeling of the static shape.
• Bone driven animation is very broadly used in games. The bones setup can vary between few bones to close to a hundred to allow all subtle facial expressions. The main advantages of bone driven animation is that the same animation can be used for different characters as long as the morphology of their faces is similar, and secondly they do not require loading in memory all the Morph targets data. Bone driven animation is most widely supported by 3D game engines. Bone driven animation can be used for both 2D and 3D animation. For example, it is possible to rig and animate using bones a 2D character using Adobe Flash.

• Texture-based animation uses pixel color to create the animation on the character face. 2D facial animation is commonly based upon the transformation of images, including both images from still photography and sequences of video. Image morphing is a technique which allows in-between transitional images to be generated between a pair of target still images or between frames from sequences of video. These morphing techniques usually consist of a combination of a geometric deformation technique, which aligns the target images, and a cross-fade which creates the smooth transition in the image texture. An early example of image morphing can be seen in Michael Jackson's video for "Black Or White". In 3D animation texture based animation can be achieved by animating the texture itself or the UV mapping. In the latter case a texture map of all the facial expression is created and the UV map animation is used to transition from one expression to the next.
• tissues, and skin
  are simulated to provide a realistic appearance (e.g. spring-like elasticity). Such methods can be very powerful for creating realism but the complexity of facial structures make them computationally expensive, and difficult to create. Considering the effectiveness of parameterized models for communicative purposes (as explained in the next section), it may be argued that physically based models are not a very efficient choice in many applications. This does not deny the advantages of physically based models and the fact that they can even be used within the context of parameterized models to provide local details when needed.

Face animation languages

Many face animation languages are used to describe the content of facial animation. They can be input to a compatible "player" software which then creates the requested actions. Face animation languages are closely related to other multimedia presentation languages such as SMIL and VRML. Due to the popularity and effectiveness of XML as a data representation mechanism, most face animation languages are XML-based. For instance, this is a sample from Virtual Human Markup Language (VHML):

 <vhml>
   <person disposition="angry">
     First I speak with an angry voice and look very angry,
     <surprised intensity="50">
       but suddenly I change to look more surprised.
     </surprised>
   </person>
 </vhml>

More advanced languages allow decision-making, event handling, and parallel and sequential actions. The Face Modeling Language (FML) is an

event handling, and typical programming constructs such as loops. It is part of the iFACE system.^[5]

The following is an example from FML:

 <fml>
   <act>
     <par>
 	<hdmv type="yaw" value="15" begin="0" end="2000" />
 	<expr type="joy" value="-60" begin="0" end="2000" />
     </par>
     <excl event_name="kbd" event_value="" repeat="kbd;F3_up" >
 	<hdmv type="yaw" value="40" begin="0" end="2000" event_value="F1_up" />
 	<hdmv type="yaw" value="-40" begin="0" end="2000" event_value="F2_up" />
     </excl>
   </act>
 </fml>

See also

• Animation
• Caricature
• Computer animation
• Computer graphics
• Deepfake
• Facial expression
• Facial motion capture
• Interactive online characters
• Morphing
• Parametric surface
• Texture mapping

References

1. ^ Hjortsjö, CH (1969). Man's face and mimic language Archived 2022-08-06 at the Wayback Machine.
2. ^ Learning Audio-Driven Viseme Dynamics for 3D Face Animation
3. doi:10.1016/S0097-8493(03)00033-5
   .
4. PMID 10573899
   .
5. ^ ^{a b} "iFACE". Carleton University. 6 June 2007. Archived from the original on 6 June 2007. Retrieved 16 June 2019.

Further reading

• Computer Facial Animation by Frederic I. Parke, Keith Waters 2008
  ISBN 1-56881-448-8
• Data-driven 3D facial animation by Zhigang Deng, Ulrich Neumann 2007
  ISBN 1-84628-906-8
• Handbook of Virtual Humans by Nadia Magnenat-Thalmann and Daniel Thalmann, 2004
  ISBN 0-470-02316-3
• Osipa, Jason (2005). Stop Staring: Facial Modeling and Animation Done Right (2nd ed.). John Wiley & Sons.
  ISBN 978-0-471-78920-8
  .

External links

• Face/Off: Live Facial Puppetry - Realtime markerless facial animation technology developed at ETH Zurich
• The "Artificial Actors" Project - Institute of Animation
• iFACE
• Animated Baldi
• download of Carl-Herman Hjortsjö, Man's face and mimic language" Archived 2022-08-06 at the Wayback Machine (the original Swedish title of the book is: "Människans ansikte och mimiska språket". The correct translation would be: "Man's face and facial language")