Have you ever been curious about how facial animation and motion capture can be used in a live production setting? Well our friends at Iralta were gracious enough to give us some insight on how they were able to create Mapi for Spain’s public broadcast network TVE1. Get a behind-the-scenes look at how Mapi was created and the workflows involved in creating a show like no other on TV today!

Mapi is more than a quiz show with questions and answers, Mapi is an Augmented Reality 3D character. Using Multi-Camera Virtual Production technology, the show and character were created by Iralta Virtual Production in collaboration with Mediacrest and RTVE. Mapi is broadcast during “access prime time” on TVE1 and is an adaptation of “Chikos’s Challenge”, the revolutionary and successful program from Japan’s public channel NHK. “Chiko’s Challenge” is a weekly program, so adapting it and its workflow to the demands of a daily format has been a technological and artistic challenge.

The requirements were for a 3D character that could perform the same as an actor of flesh and blood: a hybrid half human and half 3D character who can interact with other actors, contestants and real-life objects.

The 3D model is positioned in space using Mo-Sys fitted on the cameras and the actress. By using this system, we are able to locate the character in both real and virtual spaces.

The facial capture workflow used Faceware’s Mark IV headcam and Faceware Studio to translate the actress’ gestures to the 3D model in real time. The use of “machine learning” helps the facial capture software to improve the expressivity of Mapi over time as well.

Extra animations were developed in Unreal Engine to add to Mapi, such as smoke, rain or hearts.

Using Unreal Engine, a real time render engine, we could integrate 3D live on four cameras. To help composite and augment the realism, we also used the same illumination as the set and lighting from the video panels.

The integration of the AR 3D character was developed to deliver four live feeds to the vision mixer, thus allowing the director to mix live.

The workflow we designed for this Virtual Production produced over 5,400 minutes (90 hours) of AR content in real time without the need to do any post-production in 3D.

This new way of producing 3D, VFX and animation drastically changes the established pattern of shoot then post-production. Virtual Production allows for animation and “post-production” to be created in real time parallel to the shoot now. Actors are now able to interact live with 3D elements of the set as well as virtual actors and objects.

The production of Mapi proved to be a difficult and creatively challenging feat. How could we use similar technology as Avatar, Mandalorian or Fortnite to produce an entire live TV show? Augmented Reality is used by many broadcasters to add graphic content or data, but this is the first time a hybrid 3D character acted in real time for multi cameras.

The design of Mapi took into consideration “Chiko’s Challenge” and how it originally fit into Japanese culture and television. For a more western appeal, inspiration was taken from characters such as Clea and Cuquin from “Family Telerin” or Vanellope from “Wreck-It Ralph”.

The character design was undertaken by Marcos Raya with the Iralta VP team directed by Angel Urbina and Dave Burton. Ultimately, Mapi is a humorous little girl with a questioning mind. On a deeper level, Map is the queen of knowledge and full of emotion. The 3D model of Map is a girl’s face capable of expressing an array of emotions from laughter to anger with the ability to improvise with her co-characters.

Augmented Reality using Virtual Production allows us to get the best performance from an actor and translate it to a 3D character in real time. However, the expressions wouldn’t be the same without the great actress behind Mapi: Carla Pulpón. Every Day on set Carla gave life and voice to the virtual character she was portraying. She did this while wearing a dummy head for reference and the Faceware Mark IV head mounted camera system that synchronizes her movements with those of Mapi.

Mapi is half virtual and half human. The costume for the human portion of Mapi was taken on by Esther R. Luaces and Reyes Pe from Producciones Mutis. The designs approved by TVE were made in record time to coincide with the pilots and technical tests for the show. The colors and textures of the fabrics had to work and integrate with the virtual digital character. Mapi’s outfit needed to be visually appealing while also very functional as her backpack had to hide the hardware for facial and motion tracking. The dummy head helped Carla achieve the correct movement and gave us real reflections and shadows to work with.

Seven powerful computers were used to bring life to the virtual part of Mapi and to integrate the different data (e.g., video, position and facial recognition): analysis, animation and rendering. A whole series of digital props were also developed to enhance the emotions and movements of Mapi.

During the months leading up to recording, the Iralta VP crew were involved with R+D and other tests to bring Mapi to life. Many long days were spent optimizing the facial and camera tracking systems that continued to evolve in every program. Over various weeks, the crew pushed the computer systems and software to the limits to attain the best real time result.

The team’s directors, Ángel Urbina, Dave Burton and César Urbina, wish to thank all the technical and artistic crew at IraltaVP: Peter Gagnon, Xavi Calvo, Gonzalo Fuentes, Alberto Sanchez, Illary Vasquez, Ramón Verdugo, Iván Quesada, Sandra Vazquez, Fran Carbajo and Claudio Garcia.

We at Iralta would like to thank RTVE and Mediacrest for commissioning us to create Mapi, an AR content piece using pioneering Virtual Production at a global level. 

We would also like to thank the entire team at Iralta VP and our technology partners: Epic Games, Mo-Sys and Faceware.