Apple’s ‘RealityOS’ may be a baby step into a new computing paradigm

By Mark Sullivan

February 10, 2022

Apple’s augmented reality work is getting more real.

The company has kept its work on augmented reality glasses fairly well hidden, but the media and analyst reports of the existence of the new wearable device are getting more frequent. The current thinking is that Apple is working on an AR headset product (than may also support VR) for release next year. It’s also working on a more svelte pair of AR glasses to be released sometime after that.

On Tuesday the first mention of the (possible) OS–called “RealityOS”–was spotted in the code in pre-release iOS 13 builds, and in App Store upload logs.

The augmented reality technology that Apple is reportedly working on involves placing two transparent displays within the lenses of a headset or glasses. Rather than being confined to the borders of a phone or laptop screen, the OS and apps would be superimposed upon, or interspersed with, the view of the real world that users see in front of them. That’s why the name “RealityOS” may fit–because the OS blends your computer interface with your reality.

While Apple executives have voiced their excitement about AR technology, the company has not confirmed that it’s developing a pair of AR glasses or an OS for AR. The reliable Apple analyst  Ming-Chi Kuo has said the first AR device, which will feature two powerful Apple-designed chips, is coming next year. In the meantime, outsiders have been busy creating their own fantasy previews of what Apple is building.

This mockup by the designer Dominik Hofacker provides a plausible suggestion of how Apple might present its new approach to personal computing to the world. “What would you do if the world were your display?” begins the mock promo.

 

Hofacker’s mockup depicts a virtual keyboard that you type on with your fingers (presumably hand tracking sensors on the front of the glasses track their movement). There’s a virtual trackpad that you can run your fingers over (also tracked by sensors). Then there are a few different types and sizes of displays floating in the user’s view. Whether or not it ends up presaging Apple’s actual approach to AR, this is all pretty cool, because it virtualizes and replaces a collection Apple tablets, phones, and computers on your desk.

Baby steps

The mock OS takes a series of user interfaces that we’re already used to and making them virtual. That might be a good thing–at first. People might benefit from the familiarity of that old design language as they try to navigate the jump from 2D graphics and physical controllers to 3D graphics and virtual controls in AR.

Judging by past Apple platforms such as the iPhone, iPad, and Apple Watch, the company is perfectly willing to introduce new features little by little, so that the product gradually fulfills the promise of the technology over time. The same will likely be true with Apple’s first foray into AR. For example AR glasses may be a common way to access the metaverse, a virtual space that may eventually present the internet as an immersive 3D experience. But Apple’s device may not extend to that wider experience, at least at first.

Ultimately “RealityOS” will be something altogether different. It will inhabit a completely different canvas than iOS or macOS–a larger canvas that moves with the user and senses depth. Apple will have to create an OS that integrates graphics into spatial environments in useful and meaningful ways. The controls and apps within the OS may need to change dynamically according to the real world setting in front of the user. Eventually RealityOS may look as different from iOS as Windows did from DOS—or Apple’s own Mac did from the Apple II.

Exploiting AR

Apple will also have to envision new use cases and new apps that exploit the 3D holographic imagery AR is capable of, and exploit the powerful computer vision AI running in the background.

Some of the first instances of 3D holograms might appear in a gaming context. In 2017, Niantic released an “AR+” version of its Pokémon Go game that placed holograms within the real world as seen through a smartphone, for example. It’s easy to envision a “RealityOS” version of that game released alongside the eventual AR headset or glasses. It’s also not hard to imagine a version of FaceTime where your friends appear to be sitting in a (virtual) room with you.

It’ll be up to developers to find AR’s killer apps. The App Store upload logs that were discovered made reference to a “RealityOS” simulator, which could be used by developers to test glasses-based AR apps. Apple released its AR development framework, ARKit, to developers in 2017.

Apple’s AR chance, challenge

Many in tech expect mixed reality wearable devices to eventually replace the smartphone as our go-to personal computing device. If that is indeed the future, it would be strange if Apple didn’t release a spatial computing device.

Other big tech companies such as Facebook and Google may release AR glasses first, but Apple is well-positioned to play its traditional “fast-follower” role in AR. Judging from its history in smartphones, smartwatches, and other categories, it has a chance to define the AR glasses category and send the technology toward the mainstream.

Apple’s genius is the user interface–the environment we see in front of us—on our phone, Mac, or iPad—that mediates between our mind and our computing device (yes, the hardware is beautiful too, and it plays an important supporting role). The UX is the reason for all of Apple’s big successes, from the iPod to the iPhone. So developing the UX for the next major computing platform–something like “RealityOS”–is within the company’s sweet spot.

Apple also has an edge in terms of privacy. Because AR glasses will be closer to our bodies, and filled with more sensors and AI muscle than any consumer device in history, they will know far more about us than even our phones do. That’s why Apple’s activist position on data privacy is so important. Consumers may see the company as the safe choice to protect all the data generated by this most personal of all computing devices.

Finally, AR glasses may be an important test for Apple. The Apple Watch notwithstanding, the company has become a $3 trillion company largely on the strength of older platforms such as the iPhone, which debuted 15 years ago. The iPhone—and services and apps for it—remain Apple’s primary revenue source. Now a new paradigm in computing approaches. How Apple responds may say a lot about whether it still has the vision and engineering chops to deliver world-changing devices.

 
Fast Company , Read Full Story

(22)