Client: CGI Face: The Face Project

Need: Hyper-Realistic CGI Facial Animation

Author: Chris McMahon


Faceware’s capture solutions enable true-to-life performances across the spectrum of media, but Jim Su and his collaborators are hoping to take things a step further: pushing CGI faces to the next level of realism. Faceware Analyzer and Retargeter are on hand to help them advance the industry’s gold standard for faces.

Faceware was built specifically to ensure that creators can transfer natural facial performances into their movies, TV shows, games, and other media with a much greater degree of efficiency.

However, creating a convincing digital face is like walking a tightrope; there is a fine line between immersing viewers in a realistic vision and leading them down the uncanny valley. The “uncanny valley” refers to Masahiro Mori’s hypothesis that human replicas can often elicit feelings of eeriness and revulsion among observers. As video games grow increasingly realistic, for instance, the more players notice imperfections.

A small team in Vancouver is attempting to advance the quality standards for CGI animated faces, however – free from the constraints of a typical production schedule. Using Faceware’s Analyzer and Retargeter software, veteran film industry rigger Jim Su has teamed up with Pixel Light Effects and Waterproof Studios, combining their collective talents to develop a hyper-realistic visage. The aim is for this CGI face to mimic complex lip-sync for very fast, difficult phrases, all without spoiling the illusion that the face is truly human.

At the time of writing Jim Su’s project is still unfinished, but the results so far are nothing short of mesmerizing.

Face scanned using photogrammetry as well as a series of face poses based on FACS (Facial Action Coding System)

Actor filmed performing challenging lines of dialogue. Faceware used to retarget the performance to the facial rig.

Masters of their craft

Su’s career dates to 1999, when he started out as a modeler and rigger before moving on to positions as a rigging supervisor and character/creature supervisor. His credits include 300: Rise of an Empire and Heart of the Sea while employed at Scanline VFX, and more recently, he helped deliver the Teenaxi aliens in Star Trek: Beyond at Double Negative.

This new project is one borne purely of passion. It’s taking place on Su’s own time, enabling him to pair his CG knowledge with the unique skills of his collaborators.

Pixel Light Effects is providing photogrammetry scans via its one-of-a-kind mobile capture setup, which features a 144-camera DSLR array permanently installed inside a cargo truck. The company can transport its technology around Vancouver for the many productions taking place there, implementing an ideal means for capturing the real-life actor that shares a face with Su’s digital recreation.

Using the setup, Pixel Light Effects delivers the base mesh, facial expression scans, and textures for this project, while Su handles the facial rig, blendshapes, animation, and the facial motion capture.

Waterproof Studios, which specializes in video game cinematics for titles like XCOM 2 and Battleborn, is meanwhile handling look development, lighting, rendering, and compositing. Original look development was handled by Hernan Melzi, who is well known for his work on Captain America: Civil War.

“The dream goal was to create the most realistic CGI face that we were capable of,” begins Su. “We had some pragmatic goals too. These days, visual effects and video games demand compelling lip-sync. While there have been some impressive CGI faces that have surfaced in the last couple of years—particularly in the study of microstructure details like pores—they haven’t tackled lip-sync in the way Digital Emily did. That was one of the goals of our project; to create compelling lip-sync with three very challenging lines of dialogue.”

Indeed, the lines (pulled from memorable moments in films Point Break and Snatch) are tense word jumbles that not only catch your ear, but require a highly animated face to deliver in a realistic fashion.

To help pin the targeted footage onto their facial rig and then start fine-tuning the results, Su and team turned to Faceware’s Analyzer and Retargeter.

Original Performance Capture

Finished CGI Using Faceware Analyzer and Retargeter

Retargeting realism

Alongside capturing full 3D scans of the actor in Pixel Light Effects’ mobile rig, the team captured video of the actor performing lines in front of a still camera. From there, Faceware’s Analyzer was used to recognize the actor’s face and generate the motion capture data needed to transport his performance into the digital realm.

“We didn’t have the benefit of having a headset,” explains Su, who had encountered Faceware’s tech on previous projects, notably used the Analyzer and Retargeter on Star Trek: Beyond.

“However, a quality video camera with a high frame rate and well-lit conditions certainly made up for it. I’ve also used video captured straight from a webcam while using Faceware, which makes it incredibly accessible for an animator to do their own character studies.”

From there, the Retargeter was used in Autodesk Maya to affix Analyzer’s data to the facial rig. This initial result was up and running with ease, and provided a strong starting base to work from.

“I stayed with Retargeter due to its flexibility in adding and editing keyframes,” explains Su. “A hand keyframe pass was done, and some shot sculpting was done directly to the rig. Then the rig was alembic geocached and handed off to Waterproof for rendering.”

Su maintains that it would have been impossible to complete this project without the ease of Analyzer and Retargeter on hand. Although he is not an animator himself, Su could work easily with the results generated by Faceware’s one-two software punch.

“There’s no way that I could have animated this on my own,” he says. “We would have had to find an extremely highly-skilled animator to do it. With Analyzer and Retargeter we didn’t have to—we had everything we needed to generate a robust first pass right there.”

A finely tuned performance

 For Su, the time-saving benefits offered by Faceware are invaluable, making passion projects like this a realistic option.

“Suveying some veteran animators, it generally takes anywhere from 1-3 days to do a quality first pass on 200-300 frames of dialogue,” he says. “For me, using Faceware’s software, it took around 2-3 hours to go from raw video to delivering animation curves for the animators with the same level of quality. And this was without even taking advantage of the batch functionality.

“It provides a great starting point to hand off to a talented animator,” he continues. “Even better would be to teach an animator how to use Retargeter as an animation tool. I think they could harness better quality results than a technician could.”

Waterproof Studios knows this well: the studio now uses the Analyzer and Retargeter for its video game productions for various clients, and has already seen immense savings in time and effort, along with a huge leap in facial animation quality.

“The addition of Faceware’s Analyzer and Retargeter software to our facial pipeline has already paid dividends in our pre-rendered and game engine productions,” explains Animation Supervisor Aaron Zacher. “We now achieve an overall higher fidelity in our initial facial animation passes, and matched with the benefits of the decreased turnaround time for internal reviews and client revisions, we are able to process and quickly deliver a much higher benchmark of facial animation to our clients.

“Faceware allows our artists to spend more time fine-tuning great performances, and allows the studio to deliver an improved product to our clients with much more flexibility in our overall workflow,” he adds. “We are big fans, to say the least.”

Face forward

The CGI Face project commenced a year ago. Su and Waterproof Studios are now focused on making small yet important refinements to deliver on their ambitious aims, enhancing elements like pores and wrinkles, as well as adding details like the eye meniscus. “It’s the near-imperceptible elements that really recall realism in a digital character,” claims Su.

Achieving the project’s goal of true-to-life realism will ultimately showcase Su and team’s respective skills, not to mention the potential of using photogrammetry, motion capture, and other techniques to deliver peak facial realism. The process will also enable them to deliver greater results to clients in the future. There may be other uses, as well: Su says he has been contacted by companies with scientific aims, who focus on cosmetic surgery and dental implants, to see how the technology could aid their own fields.

While they still have work to do, Su is keen to stress that it’s Faceware that has got them so far, and it will remain a part of his pipeline on any similar projects going forwards.

“Faceware is like recorded reality,” concludes Su. “If we didn’t have that, then you’d probably just see just a rendered face going through the facial range of motion and not really communicating anything. Faceware definitely takes things to the next level.”

Faceware and delivering total reality

Today’s digital experiences have more visual fidelity than ever before—audiences are coming to expect ever more realistic faces, and they’re primed to notice flaws when they’re there. That’s why Faceware’s Analyzer and Retargeter  software is designed to help achieve true-to-life capture at the pace of production, as Peter Busch explains…

How does Analyzer and Retargeter work to deliver a realistic final result?

Analyzer and Retargeter are our markerless facial tracking and animation tools. Once the facial motion capture video is shot, Analyzer is used to convert the video of the facial performance into facial motion files for use in Retargeter. Using the facial motion capture data from Analyzer, Retargeter applies the data onto any facial rig through a plug-in for Autodesk Maya 3ds Max and MotionBuilder. By combining the facial poses and high-quality movement data, Retargeter gives animators the control and flexibility needed for the creation of convincing facial animation.

What kind of nuances are these tools able to capture in an actor’s performance?

Our facial recognition technology is able to capture a great deal of nuances from the brows, to the eyes, nose, cheeks, and mouth by tracking textures and features in order to create highly detailed output. This enables our software to handle anything from stylized to photo-real characters. Together with other features like timecode support, capture, trimming, pose libraries, and a technological core drawing from over 10,000 minutes of facial performance data and users in almost 50 countries, Analyzer and Retargeter are the most powerful tools available for facial animation.

What subtleties are incredibly important in a facial performance when it comes to delivering reality?

In facial animation a realistic rendering of the entire face is important, but the eyes help bring out the emotion and believability in any performance. Our advanced pixel-tracking technology provides extremely accurate eye and gaze tracking, further reducing the barrier between your performance actor and their digital counterpart, providing the most realistic animation possible.

How fast are these tools? Can they deliver efficient results at a fast turnaround?

All of our tools are designed to work at the speed of production. We recently worked with EA to improve their batch processing and create an automated pipeline, enabling them to produce an average of 30-45,000 seconds of animation per month.

What are normally the biggest challenges you face when building a realistic CGI face?

A major challenge for facial motion capture and animation is consistency among a large team of animators. The final result should be fluid and look as if it were created by one skilled animator. To manage the realities of production environments, we’ve created features in our Retargeter 5.0 Studio Plus software that enable your team to work as one. Animators can create centralized Shared Pose libraries to ensure that every pose is created the exact same way, regardless of the animator. We’ve also enabled users to access our entire library of commands, allowing technical artists to create entirely automated pipelines. This allows you to maximize facial animation output and take advantage of your computer’s processing time, not your artists’ time.  

Are Faceware tools easy for animators to learn, navigate and use?

Our software is designed by animators for animators. It’s simple and easy to understand, yet fully featured and capable of performing extremely complex, believable facial animation. No other software offers region-based control with facial movements, meaning each emoting area (the eyes, the brows, the mouth, etc.) are all separate entities. This independent control provides an unparalleled degree of precision, delivering a higher level of detail and realism.

What are some of the most useful features in Faceware when constructing a true-to-life face?

Our software takes the stress out of the facial animation process. With Retargeter, manual keyframe changes and polishing are things of the past, and data is applied quickly so iterations are fast and efficient. Intelligent keyframe reduction (pruning), animation smoothing, a master timing tool, and several other features make working with the data very easy. Our workflow and software was designed by animators in real-world production so that the software is easy to use, intuitive, and guaranteed to work every time.

How important is using a headset when capturing facial movements? How does this affect the applications of Faceware?

Using a quality, professional headcam is incredibly important. If the facial motion capture data is low quality, then all resulting facial animation will also be of lower quality. As such, capturing crystal clear video is ideal for both animator reference and for facial motion capture tracking with Faceware Analyzer. With this in mind, Faceware Headcam Systems are designed using over a decade of industry experience to capture and render high-quality facial movement data.