When husband and wife team Solomon and Nerissa Jagwe formed art, animation and digital production outfit Sowl Studios, their mission was to create content learning opportunities for their two children, other children and future generations by tapping into a rich and diverse African heritage—and to do it just about all from their own home.

The result is Sowl’s The Adventures of Nkoza and Nankya, an animated series which follows two characters that can currently be seen via a YouTube channel and in an associated app. Geography, counting and languages—such as Luganda—and other important cultural aspects are central components of the app and videos.

Incredibly, the facial animation of the characters has all been crafted with only limited resources using a range of accessible tools. This includes facial motion capture enabled with Faceware’s hardware and software made up of the Faceware Mark IV helmet mounted camera (HMC) coupled with an Xsens motion capture suit, and other tools.

Before diving into 3D character animation, Solomon and Nerissa had already been developing an African language app parents could use to help children learn more about their culture, but they wanted to take it a step further. That’s where the 3D characters Nkoza and Nankya came in.

“We embarked on expanding our language app to help parents with an educational tool that contains engaging content,” says Solomon, who adds that the animation is being done mostly from their “living room.”

Indeed, in order to make the content for the app and YouTube channel, they needed fast and efficient ways to complete the facial motion capture. The small team behind the work is based on three continents, with one of the key voice actors located in the UK (Solomon and Nerissa are in Virginia).

As part of his exploration into facial animation for Nkoza and Nankya, Solomon says he had also been “on a quest to find a head-mounted camera system that was durable and light enough for facial performance capture.” He even made his “own homemade system to help capture my own performances, but it was too heavy and I certainly couldn’t trust it enough to use it with other voice actors.”

Then, Faceware sent him the Mark IV system. “I was on cloud 9 with excitement. The helmet fits so comfortably and is lightweight. It offers a wireless solution that makes it easier to move around while doing the performance, without being tethered to a computer.”

Then there’s the refactored LED lighting system on the Mark IV HMC. Solomon advises that he had previous tried out various lighting options on a home-made head mounted camera system but struggled with an over-lit face resulting in overblown videos.

“I used the Mark IV system to record myself performing sessions of a poem, and the story from The Adventures of Nkoza and Nankya,” adds Solomon. “There are some scenes for our kid actors that needed me to add some extra animation, so I loaded the audio and lipsynched to match my son’s and daughter’s voice. I was able to use the video from the Mark IV system, inside of Faceware Studio as a video input, and with the professional headcam as the face tracking model, and streamed that into Reallusion iClone.”

What Solomon found worked well with the Mark IV system was its flexibility on the wearable side and its lighting. For example, there are three helmet sizes available, each with additional foam padding for interiors for a snug fit.

“This is such a thoughtful option because I have two kids who voice our Nkoza and Nankya characters,” says Solomon. “So, if you have child actors playing the parts of kids in your film or children’s series, there is an option for them to wear one of the helmets for their facial performance capture. Also, the portability of the Mark IV system is awesome, to have the rechargeable battery on the belt harness, makes it easier to work for extended periods of time while doing the motion capture session with the Xsens suit.”

The Mark IV HMC also features a newly designed set of click-in-place bar mounts that the cameras sit on. Solomon followed the illustrated manual for step-by-step instructions to set the helmet mounts. “The new design makes it easier to adjust the angle of the camera for new sessions and for follow-up sessions. The camera is set at just the right distance on the adjustable bars, which eliminates the fisheye lens look you often see in videos captured with head-mounted cameras.”

For a complete capture process, Solomon has been making use of a Xsens Link suit for body capture. “I’ve used other inertial solutions in the past, but I was completely blown away by the accuracy of the Xsens Link suit data. Saved me a lot of time in clean up.” Also, the easy integration of Xsens and the Mark IV HMC for facial capture means capture can happen fully wireless. “The seamless working together of the two systems,” details Solomon, “is made possible by the Shepherd software from Faceware, which records the video from the Mark IV system camera, via the KiPro Rack, in sync with the Xsens MVN animate Pro software.”

Also added to the mix were Manus Prime II Xsens gloves, which allowed the capture of finger tracking together with the full-body data to create clips of the kids dancing. “This saved us so much time because we would have had to keyframe the fingers and the body animation had we not had the Xsens suit and Manus gloves.”

The finger and body data was streamed from MVN Animate Pro into iClone using the Xsens Motion Live profile, and Solomon applauds how it all worked seamlessly with Faceware hardware via Faceware’s Shepherd platform. “This makes it possible to record the mocap data simultaneously with the video data from the camera on the Mark IV HMC system.”

Here’s how Sowl Studios tackles facial motion capture and animation for Nkoza and Nankya from their “living room,” from go to whoa. They start by sending a script and storyboard to their voice actor, detailing specs on the environment recording requirements, emphasizing the need for stable footage and noise-free audio.

The voice actor then sends back a video recorded with audio. It’s a workflow that would not be possible with several other cheaper or make-shift solutions such as smart phone video capture. Faceware allows for significant flexibility in the capture process.

“We load that footage in Faceware Studio as a media input type,” continues Solomon, “with a stationary camera selected as the face tracking model. We then enable streaming and load our custom character inside of iClone.”

Earlier, Maya and 3ds Max are used for modeling and rigging Nkoza and Nankya, with iClone 3dxchange Expression Editor relied upon to prepare the characters for facial and body animation support in iClone.

Facial animation is then made possible with the Motion LIVE plug-in for iClone that connects with Faceware Studio to capture the actor’s facial animation.

The project is a family affair, with each of them donning the Mark IV HMC to carry out the motion-captured facial performances. “My wife plays a major part as a performance actor, and so does our daughter, who is the voice of little Nankya, and our son who is the voice actor of Nkoza,” states Solomon, who, as noted, is also particularly thankful for some of the new options available in Faceware Studio to help enable the animation workflow.

“I love the addition of video as one of the options for the media input type,” he says. “Previously there were two options one could use to read the performance of a voice actor; via webcam and via a sequence of images. In the current build, it’s now possible to add a video as the source to drive the facial animation of a 3D character.”

“This makes it possible to use video from my teammates who are overseas, especially when it comes to the character of Maama Nankya (Nkoza and Nankya’s mom) played by Diana Lwanga, who is based in the UK. We were able to use videos that she recorded remotely, to bring a number of shots to life using the video source feature in Faceware Studio.”

Another favorite Faceware Studio feature of Solomon’s is ‘animation tuning and motion effects’, one that he feels has become indispensable in Sowl Studios’ facial animation workflow.

“Being able to fine-tune the facial animation in real-time before recording the final takes is invaluable and has made our facial animation process so much faster. We are able to adjust facial elements like the head, mouth, eyes, and brows and see the results right away in iClone. Voice actors vary in their mannerisms, so for each take, we can save a profile for a specific voice actor and reload that whenever we have a new facial performance video take from them.”

The need to carry out production on Nkoza and Nankya with limited resources and to do so remotely, owing to the pandemic, is one aspect of Sowl Studios’ production that has benefited greatly from using Faceware Studio, notes Solomon.

He says, though, that this has meant that careful planning of voice recording and facial animation is crucial. Asked what advice he might give fellow creators in that regard, Solomon centers on the preparation of the voice actors in advance.

“The lighting and facial orientation of the voice actor are very important. We do our best to over-communicate about lighting and the position of the camera the voice actor is using.

Solomon says he is excited to continue to use Faceware’s Mark IV HMC and Faceware Studio, along with the Xsens and other workflows, to process all the voice actor videos for new characters in the Nkoza and Nankya universe, and try out new workflows to speed up animation renders, such as working in Unreal Engine via iClone’s Unreal Live Link—all the while doing this with an indie set-up in mind.

Sowl Studios is constantly generating new Nkoza and Nankya animated content, and expanding the different languages, games and animated video available in the app.

Being able to achieve this remotely, nimbly and with so few people is key.

“I am super grateful to our small but mighty team,” remarks Solomon, “without whom we wouldn’t have made progress and created a product that is making a difference in these trying times.”