Your Photo Of A Burrito Is Now Worth A Thousand Words

That burrito in your hands—so warm, so gooey, the richness cut by cilantro and red-hot spice. Before you take a bite, you’d better take a picture.

Multiply that impulse by tens of thousands and you get Yelp’s database of images, drawn from burrito joints, cocktail bars, and more. Until recently, Yelp was dependent on users to tag their images with search-friendly metadata. But now, using the kind of deep learning techniques that are transforming the field of AI, Yelp is starting to see the business benefits of using software intelligence to power its listing pages and user recommendations.

News coverage of AI, or artificial intelligence, has focused on the efforts of deep-pocketed Silicon Valley giants Amazon, Apple, Google, and Facebook, which are embroiled in an arms race for developer talent. They are battling to hire the world’s top PhDs—and ultimately, to anticipate their users’ every want and need.

Smaller technology companies have just started to test the waters with more modest AI initiatives, thanks to the increasing availability of sophisticated software libraries. But unlike the complex and grandiose projects underway at Facebook and its peers, deep learning implementations at companies like Yelp are perhaps more likely to deliver near-term value to customers and to the company.

“It’s game-changing over the next five years,” says Michael Stoppelman, Yelp’s senior vice president of engineering and brother of CEO Jeremy Stoppelman. After launching a competition this past spring that called for coders to build models that could tag restaurants with labels based on user-submitted photos, the company is now able to predict attributes like “good for kids” and “ambiance is classy” with 83% accuracy. That’s all thanks to the winning set of deep learning techniques.

“You used to have to ask users,” says Stoppelman. Now, just a year after getting an AI team up and running, software can do the job. With a bit more work, Stoppelman expects to be able to auto-caption images, improve search recommendations, and better select an assortment of images to feature on businesses’ listing pages. “We’re quickly moving into a land where this deep learning is allowing us to unlock this semantic information in photos.”

The software will prove especially useful for businesses lacking reviews. “Users will be able to search that business regardless of whether they have review text,” he says. “In the future we’ll be able to detect, ‘Oh, there are burritos,’ which I think is really exciting.”

Yelp software engineer Wei-Hong Chuang was initially skeptical of deep learning, a form of machine learning that involves training neural networks to solve problems using large sets of data. Through simpler image analysis techniques like color detection, he and his team had been able to predict many of the business attributes that the spring competition highlighted—natural light, for example, suggests outdoor seating. “You can tell a lot about the business by reading the color distribution,” Chuang says.

But the competition opened his eyes to new possibilities like texture and shapes. “Although color definitely helps a lot, deep learning is better,” he says, adding that this is especially true when it comes to extrapolating results to better understand a business category or a type of user.

For Yelp, that could translate into better search recommendations and eventually better user engagement and retention. As for determining which restaurant’s burritos reign supreme, experts say it will take another AI breakthrough to answer that critical question.