Real-time face tracking + 3d projection mapping

Nobumichi Asai [WOW]　Creative / Technical Director

OMOTE

[Omote] is a Japanese word for face, or a mask.

NOBUMICHI ASAI [OMOTE] (PRODUCER / TECHNICAL DIRECTOR – PICS)/ YKBX (DIRECTOR)
HIROTO KUWAHARA [OMOTE] (MAKEUP ARTIST / FACE DIRECTOR) / EVALA (MUSIC) / PAUL LACROIX [OMOTE] (TECHNICAL DIRECTOR / SYSTEM ARCHITECT – TRANSIT DIGITAL WORKS) / TOSHIHIKO SAKATA (CG PRODUCER – FILM) / KAZUHIRO KOBAYASHI (CG DIRECTOR- FILM) / KENTARO KOYAKU (CG DESIGNER – PICTREGO) / SHINTARO NODA (CG DESIGNER) / NAOFUMI YAMAMOTO (CG DESIGNER)

Face is considered as [mirror] that reflects human soul, a separation between Omote (exterior) and Ura  (interior), and in Nogaku, Japan’s classical musical plays, performers use Omtoe masks to express multitude of dramatic emotions.

As we spent more time on the project, we became more aware of its similarities with Nogaku’s Omote masks, and explored further possibilities through Integration of latest technology and classical Japanese art.

I always considered [face] to be the most delicate yet powerful medium for art.

Make-up artist Hiroto Kuwahara said, ‘Face evolved with a sole purpose of conveying emotions, and it is the only body part that effectively communicates and reacts to the most subtle changes and conditions.  It’s not an exaggeration to say that face is one of the most intriguing features.’  I totally agree with him, and got interested in the possibility of face mapping, its potential in expressing something extraordinary.

Face-mapping technology already exists, often used in marketing initiatives.  But I wanted to focus on the expression of [beauty] and [make-up art] instead of technological gimmicks and impacts.  But this demands a higher level of technological precision.  A subtle mismatch in projection would ruin the effect.

The team also wanted to express Japanese beauty to the world.  Kuwahara said, ‘When imagining global definition of Japanese aesthetics, philosophy, and virtue onto a face, I thought about the cosmetics commercial featuring Sayako Yamaguchi, which I saw in my childhood.  In my mind, Japanese women have bewitching, ethereal, and sophisticated qualities that are unique.  To communicate Japanese beauty, I particularly focused on these attributes and chose Ms. Sekimizu as our model, instead of someone else with more structured features and big double-eyelid eyes.’

When I look around nowadays, it is quite rare to see women with traditional Japanese aesthetics.  My intention is not to communicate [make-up] as influenced by other cultures, but to express [kesho (Japanese word for make-up)], Japan’s unique set of aesthetic ideals that have been passed on through generations.  I feel that conveying this message is important for the traditional and cultural aesthetic ideals of [kesho] to thrive.

Additionally, [Japanimation] is also an art form, a subculture that is unique to Japan.  I personally like manga and animation, and looked for particular motifs from them.  ‘Ghost in the Shell’, ‘Battle Angel Alita’, ‘Space Cobra’ and ‘Jo Jo’s Bizarre Adventure’ are some examples.

The overall system was created and developed by Paul Lacroix and Nobumichi Asai. Technically, it’ s very difficult to project with real-time tracking.

Nobody has done it before, so we had to start from scratch. The process was complicated, capturing with sensors, processing the data, creating CG renderings, and using projectors. Getting the timing right was also challenging, fixing delays and making sure the projection technology had the maximum precision to get the face and visuals matched.
For this project, we used OptiTrack sensors for speedy response, and developed original program C++. To reduce minimal

latency (delay), we devised ways to integrate the whole process into one program. This creates amazing effects.

What’s most important, is thinking about what message to convey through rich facial expression and power of communication.  Nothing is set in stone, but I hope to express message of ‘love & peace’ in my next work.

The real essence of technology art exists in the coexistence of 2D and 3D realms, like projection mapping.  For example, holograms, 3D printers and AKB48.  They all share spatial experiences that is physical and real. Generally, I think people are attracted to and have the desire to create this type of spatial existence that is both real and physical.

Additionally, this field attracts media artists who are more focused on technology, but I feel too much emphasis on technology potentially weakens [expression].  My background in science and my extensive work in [expressive] form give me the right balance. New values are created when foreign or diverse things merge.

As business, I hope to continue creating work that is enjoyable, beautiful, and entertaining for everyone.  As for my personal work, I hope to explore ways to express universal peace.  I believe that technology will give me new and unlimited possibilities.