Why Siri Needs to Get Smarter Faster

Apple is the most successful company in the world. But future success for Apple depends to a surprising degree on artificial intelligence.

When you think of Silicon Valley’s AI leaders, Apple may not immediately come to mind. But the company’s initiatives around AI are “amazing.” At least that’s what the company’s newest senior employee said.

Carnegie Mellon researcher Russ Salakhutdinov announced on Twitter and Google+ this week that he’s been hired by Apple and that his duties include building and managing the team that will develop future AI for the company. He will also continue to work as a professor at Carnegie Mellon. (An Apple spokesperson confirmed his hire.)

On Google+, Salakhutdinov wrote that “Lots of truly amazing AI and Machine Learning projects are taking place all over Apple” and that he and Apple will build “a team of top-notch Machine Learning and Deep Learning researchers, working on challenging R&D projects as well as conducting fundamental research to advance the state-of-the-art in AI!” He didn’t respond to my request for comment.

Salakhutdinov has worked as an associate professor in the Machine Learning Department of the School of Computer Science at Carnegie Mellon University since February. Before that, he did similar research at MIT and the University of Toronto. Besides machine learning and deep learning, he specializes in applying AI to finding patterns buried in huge data sets, such as millions of photos or gigabytes of text.

Apple wouldn’t tell me how many people within the company are working on AI. But based on the company’s job listings and acqui-hires, it needs more. (In the past year alone, Apple has acquired at least five AI-related startups: Turi, Tuplejump, Emotient, Perceptio, and Vocal IQ.)

Apple AI? You’re Soaking In It

Apple uses big AI for small things that enhance user experiences. For example, Apple uses AI to improve the iPhone’s battery life, identify people and objects in Apple Photos, and predict which apps you’ll want to open next in the “Siri App Suggestions” widget on the Today view.

Apple uses machine learning for the iPad Pro’s “palm rejection,” which enables Apple Pencil users to rest their hand on the screen while writing or drawing without triggering touch responses.

When you’re watching a movie on an Apple TV and miss some dialogue, you can ask: “What did she say?” Apple uses AI to figure out who “she” is, and also which line you’re asking about.

These are just a few of the hundreds of subtle applications of AI sprinkled throughout Apple products.

One unique feature of Apple’s approach to AI: In the interest of privacy, it handles personal data locally on the phone, rather than hoovering it all up and storing it in the cloud, as Google and Amazon do. This is a challenge, because normally user data is harvested and stored in data centers, then used to train neural nets.

Open The Pod Bay Door, Siri

Siri is the most obvious place for more noticeable AI at Apple, and the company is actively pursuing the enhancement of Siri with ever-more advanced AI, according to an Apple spokesperson.

Siri is already AI-based. Two years ago, Apple flipped a switch and Siri gained a range of advanced machine learning techniques. Apple has claimed that the AI has dramatically improved Siri’s speech recognition and the accuracy of results. Users don’t seem to have noticed. Complaints about Siri are still rife.

Veteran tech journalist Walt Mossberg wrote a scathing takedown of Siri this month for Recode and The Verge headlined, “Why does Siri seem so dumb?” Mossberg wrote that “in its current incarnation, Siri is too limited and unreliable to be an effective weapon for Apple in the coming AI wars.”

Today’s Siri fits solidly within the pantheon of first-generation virtual assistants that includes Google Now, Microsoft’s Cortana, Amazon’s Alexa, and others, at least from the user perspective. These services create the illusion that the assistant can understand what users say, then return an audible voice response in colloquial speech. They all do a pretty good job under specific circumstances and for specific queries. But they lack more advanced AI, and cannot “figure things out” as well as many users would like them to do.

Refining today’s virtual assistants with tomorrow’s AI may be an imperative for Silicon Valley’s tech giants. It’s easy to imagine a future where virtual assistants replace most Google searches, most direct interaction with calendars and to do lists, and most remote controls, buttons, and dials for smart home appliances. We’ll just talk, and virtual assistants will do all our bidding.

The company that can deliver the most compelling virtual assistant experience will have an enormous advantage. An AI virtual assistant may become the ultimate gateway drug for a company’s other products. For example, if users come to rely upon a future Google AI virtual assistant, they may be incentivized to buy Google and Android phones and Chromebook laptops, use a Google browser, and buy a TV that uses Android TV. And that’s a problem for Apple.

The race is on, and Google is probably winning. Google has made big strides in applying AI to identifying people, objects, and places in Google Photos, as well as in the human-like task of choosing the best pictures to be clustered into collections of “highlights” and “memories.” Google’s Smart Reply, available in Google Inbox and the Allo messaging app, offers suggested replies based on shockingly advanced neural networking technology.

Tech giants have been gobbling up AI startups like breakfast cereal. Apple, Google, Microsoft, Intel, IBM, Yahoo, eBay, Amazon, AOL, Nokia, Facebook, Twitter, Salesforce, and Oracle have collectively grabbed 140 AI-related companies since 2011, according to research firm CB Insights.

But Google leads the pack in the acquisition of AI-related startups both quantitatively (11 startups in the past five years) and, arguably, qualitatively: Google acquired DeepMind two years ago. The DeepMind system is capable of learning all on its own from its own memory without human input.

Tech pundits had high hopes for Google’s new Assistant product, which promised to bring more advanced AI into the virtual assistant world. Google Assistant is available in Google’s newish Allo messaging app, its Home virtual assistant appliance, and in the new line of Google Pixel smartphones. The company intends to add Assistant to Android Wear watches, the Chrome browser, and even car dashboards.

So far, however, Assistant has not lived up to expectations. Today, most queries return little more than the equivalent of the “I’m Feeling Lucky” option from a Google Voice Search. Assistant does, however, add a conversation engine, so you can go back and forth on the same topic to refine a question. The Assistant even intervenes in chats to offer additional information or recommendations.

Most of the AI in Assistant comes in the form of pre-existing AI-based services, such as the ability for Assistant to respond to the request “show me my cat photos” with your pictures of cats from Google Photos. Other AI reveals itself in the Assistant’s conversational abilities. Users can pepper Assistant with follow-up questions, and third-party apps will be able to grill users to get more information.

Google plans to open in December its “Actions on Google” and Embedded Google Assistant SDK for software and hardware makers to support Assistant, which should eventually extend what Assistant can do.

Another promising venture is a company called Viv Labs, headed by Siri founders Dag Kittlaus, Adam Cheyer, and Chris Brigham. Viv was developed in stealth mode for four years and unveiled in May.

As with Google Assistant, Viv leverages machine learning, runs third-party integrations, remembers user actions and preferences, and offers agency, such as the ability to make restaurant reservations and call an Uber.

Samsung announced earlier this month that it would acquire Viv.

Samsung sells a wide range of devices that could run Viv, from smartphones to TVs to refrigerators. The company also demonstrated a prototype virtual assistant appliance in April called Otto, which could compete against Amazon Echo, Google Home, and a recently announced “smart speaker” device from Boombotix that uses SoundHound’s Houndify platform for the virtual assistant.

The tech news site The Information reported in May that Apple is working on a Siri-based virtual assistant appliance that would compete with Echo and Home. Apple declined to comment on the report.

Siri In The Age Of Mixed Reality

Rumors are flying around the possibility that Apple is already prototyping a version of Siri with more explicitly advanced AI, chatter inspired in part by a Facebook post by tech evangelist and author Robert Scoble. In that post, Scoble claimed a source told him that a future iPhone is “a clear piece of glass” with the electronics and presumably a battery packed “in a strip at the bottom.” In a subsequent email conversation, Scoble told me he now has three sources who described the same iPhone project and have seen demonstrations of the device.

Specifically, the iPhone could be snapped into a headset for a HoloLens-like experience, where the “reality” in mixed reality is viewed through clear glass, rather than through the phone’s camera. (Apple was granted patents for an iPhone-compatible augmented-, mixed-, or virtual reality headset, most recently in August.)

Scoble said he personally believes the clear iPhone will be one option, and would not represent the entire line. He claims that Apple demonstrated “the new Siri for mixed reality” to the Viv team, and its quality prompted Viv to sell to Samsung. Apple declined to comment on future plans for iPhone, Siri, or headsets.

Given Apple’s historically conservative approach to launching new product lines based on bleeding-edge technology, however, it’s unlikely that we’ll get see-through AI iPhones and mixed-reality Apple headsets anytime soon.

But a new version of Siri with radically better AI? That sounds more likely—and more urgent for Apple.

There’s no denying that the coming revolution in AI-based virtual assistants will change how people use phones, computers, and home appliances. And when people change how they use technology, they also tend to change what they use.

Siri is based on AI now. But it needs to get a lot smarter a lot faster—or Apple could be left behind.