Google’s mixed reality tech shows faces behind VR headsets
YouTube has already perfected the art of mixed reality videos that show the VR world and the real world at the same time. A great example is the video of Conan O’Brien’s virtual trip to outer space taken when he visited YouTube’s VR Lab. The video platform’s technique still can’t capture the whole picture, however, since VR headsets get in the way. Google Research and Daydream Labs had to team up to solve the problem, and they did just that with the help of machine learning, 3D computer vision and advanced rendering techniques.
In order to “remove” the VR headset in a video, the user’s face has to be scanned with a camera and reconstructed as a dynamic 3D model that blinks and looks in different directions. Next, the VR headset — Google used an HTC Vive for its tests — has to be outfitted with eye-tracking tech in order to detect where the user is looking. Google’s and Daydream’s technique superimposes the 3D model on the face of the real person. It doesn’t completely remove the VR headset, though: it just replaces it with a transparent version to prevent the uncanny valley effect.
As you can guess, the technique will benefit YouTube creators the most at this point in time. In fact, Google is exploring ways to make it accessible to YouTubers. It could become a lot more useful when VR becomes more widespread, though. For instance, it could be adapted to show people’s faces in VR video conferences or the faces of friends playing VR games together.
(34)
