Apple Vision Pro’s hand tracking: a big mistake, or the future of gaming?

Apple’s Vision Pro won’t be available until 2024, but one of the headset’s key features is already dividing developers: The Vision Pro will be shipping without controllers, as it relies solely on eye tracking and hand gestures to navigate 3D environments.

That’s a noted departure from the way other headsets work. Devices such as Meta’s Quest 2 and Sony’s PlayStation VR use custom hand controllers for navigation and gameplay. Apple isn’t marketing the Vision Pro as a direct competitor to these devices, but instead as the first in a line of new spatial-computing devices that the company hopes could one day be as successful as the iPhone.

Andrew Eiche, for one, can’t wait for a controller-free future. Eiche is the CEO of Owlchemy Labs, a Google-owned VR studio best known for its Job Simulator and Vacation Simulator games. Owlchemy has been experimenting with hand tracking as an alternative to controllers, and Eiche believes that this approach is the only way to bring VR and spatial computing to the masses.

“To be perfectly blunt,” Eiche says, “it’s the future of the medium.”

However, turning it into a mass-market product may take some time: Apple reportedly expects to make fewer than 400,000 units next year. Many VR game developers are worried that they won’t be able to bring their existing titles to Apple’s headset, which could further depress interest in the $3,500 device. Some wonder whether a headset without controllers will even be usable for gaming.

But Eiche believes that most headset manufacturers will eventually follow Apple’s example and ship their devices without controllers. He likens this transition to a controller-free future to the jump from early smartphones like the BlackBerry, with their integrated keyboards, to the touchscreen-only iPhone. “Controllers are a huge impediment if your goal is to get VR out to the masses,” he argues. “The second you remove controllers, it just removes a barrier.”

That’s especially true for newcomers who are unfamiliar with the way the typical pair of VR controllers work. “There are a lot of people with whom handing them basically a controller split in half, putting a blindfold on their face, and then telling them to press X is just not going to work,” Eiche says.

The counterargument: Even beyond VR, controllers are familiar to tens of millions of console and PC gamers. “There is a comfort in understanding the topological landscape of a controller,” Cloudhead Games CEO Denny Unger wrote in June. “Say what you will about which genres remain popular in this $450 billion industry, but it does strain belief to think we’ll all be playing with finger guns in the latest and greatest shooter.”

Eiche would actually like nothing more than to move beyond those kinds of games. “I’m really excited for a market where the primary interaction isn’t [to] shoot [a] gun,” he says.

Owlchemy Labs has been making VR games that are anything but your typical shooter since 2016. The studio’s first title, Job Simulator, puts players into a future world in which all work has been automated, and humans get to relive the joy of menial work in a museum run by robots.

The studio revealed last year that it was working on a new game that relies solely on hand tracking, and it showed off some of the progress it has made on hand tracking with a short demo at the Game Developer Conference in San Francisco earlier this year. The demo puts players into a Job Simulator-like futuristic coffee shop, where they have to serve beverages to robot customers. Players can manipulate a range of buttons on the espresso machine, and even flip a coin with their fingers. “That’s not something your controller can replicate,” Eiche says.

Eiche doesn’t discount controllers entirely; he believes that they could become an optional third-party accessory for future headsets. Whether these will include Apple products remains to be seen: The company did announce that Vision Pro owners will be able to use existing console game controllers for traditional gameplay, but reportedly has no intention of adding support for VR game controllers to its headset.

The Vision Pro headset has yet to land in Eiche’s hands, but he is looking forward to trying out ways the device utilizes hand tracking for mixed reality, which combines a view of the real world with virtual objects. One difficult technical challenge will be something known in industry circles as occlusion: When a Vision Pro user grabs a virtual can of Coke with their real hand, will the headset know to hide the parts of the can covered by their fingers? Eiche says he’s interested to see “if that’s something Apple can pull off.”