Disney’s projection tech turns actors’ faces into nightmare fuel

Makeup Lamps tracks an actor’s movements without using the facial markers common in motion capture, then it displays any color or texture the actor wants by adjusting the lighting. It can make someone appear older by creating “wrinkles” on their face, for example, or it can paint their face in creepy clown makeup, à la Heath Ledger in The Dark Knight. And all of it is done in real-time.

A similar technology was used earlier this year during Lady Gaga’s performance at the Superbowl. Nobumichi Asai, creative director of Japanese visual studio WOW, was brought in to create a red lightning bolt on Gaga’s face during her David Bowie tribute. The attention that performance received has helped the technology become more mainstream.

Latency — the time between generating an image that matches the actor’s pose and when the image is displayed — is a big challenge to live augmentation, of course. Large amounts of it will cause the projection and the actor’s face to appear out of sync. Disney’s research team combated this problem by limiting the complexity of its algorithms and employing a method called Kalman filtering, which uses measurements over time to make predictions and minor adjustments.

“We’ve seen astounding advances in recent years in capturing facial performances of actors and transferring those expressions to virtual characters,” said Markus Gross, vice president at Disney Research. “Leveraging these technologies to augment the appearance of live actors is the next step and could result in amazing transformations before our eyes of stage actors in theaters or other venues.”