Pinterest Is Using Machine Learning To Help You Find What You’ll Pin Next

With 100 million users active on its platform every month, Pinterest is increasingly relying on machine learning to help guide the company to new online discoveries.

People come to Pinterest to explore, save, and share images and posts from around the internet. Finding content they like naturally keeps them engrossed in the platform: The company says 30% of engagement and 25% of in-Pinterest purchases are driven by the platform’s recommendations of related content. To get those recommendations right, the company relies on cutting-edge, data-driven techniques and lots of experimentation.

“A lot of what I”m doing here is trying to shape what direction we go in approaching the discovery problem,” says Pinterest’s lead discovery science engineer Mohammad Shahangian. “We launch hundreds of experiments that actually make small changes to our algorithms, and every single one of these changes has places where it helps, and places where it hurts.”

One advantage is that the platform is explicitly built around recording people’s interests, as users save products, posts, and images from around the web to virtual pinboards. That means Pinterest doesn’t have to guess what users find interesting from, say, click patterns or time spent on particular pages, as other social networks might. And it means its algorithms can guess which of the 75 billion pinned items in its database are related to each other, since they’re more likely to be pinned to the same boards.

“A lot of companies are trying to infer what interests users have off of inputs or signals,” Shahangian says. “At Pinterest, users are explicitly giving that signal, saying this is what I’m interested in.”

Pinterest visitors are essentially contributing to an ever-growing, three-part social graph, with billions of connections between users, the items they pin and the boards to which they pin them. And all that data lets Pinterest populate users’ home feeds, search results, and related pin recommendations with a greater degree of nuance: Simply showing users’ recommendations based on who they follow is less than ideal—think of a case where a user is planning a wedding and pinning dresses, while her followers are not—and just suggesting similar items can get repetitive, according to Shahangian.

“If you pinned a kitchen sink, do we want to send you 10,000 more kitchen sinks, or inspiration for how you could design your kitchen overall?” he asks.

To make those kinds of decisions, the company’s engineers have experimented with a variety of machine learning algorithms. They’ve studied how those different formulas perform on test sets of similar and dissimilar pins and, ultimately, how they impact the engagement of real-world users.

“We do have live experiments, but there are cases where we actually know a lot before exposing it to users,” Shahangian says.

Of course, there’s no way, short of actual testing, to know for a fact whether a given user will prefer a new set of recommendations. “I can’t pay somebody money to tell me whether or not Jack or Susie is going to like this pin,” Shahangian says. But looking at whether the algorithms accurately recommend content that human testers agree is related to a particular pin has proven to be a decent approximation.

Moving to an algorithmically generated feed, instead of a purely chronological display of followed users’ posts, has boosted engagement by a factor of five or 10, with additional boosts as the algorithms have gotten better.

“We’ve seen a lot of gains throughout its history,” Shahangian says. “Personalization has been one of the biggest levers for boosting user engagement.”

At the same time, the company has also been working on improving visual searches, helping users find pinned images that are similar to other pictures. Pinterest engineers have worked with researchers from the University of California Berkeley Vision and Learning Center to develop the technology, which as of earlier this year can automatically detect objects in images using deep learning techniques. Then, users can tap those objects to find similar examples across Pinterest’s library of saved content.

“It’s not quite like a classification task where we try to figure out, is this a cat or a dog,” says Dmitry Kislyuk, a lead visual search engineer at Pinterest. “We’re actually trying to find some visual similarity between every single image, and we want to do this in real time.”

The visual search tool works particularly well for finding home decor and fashion products saved to the site, he says. And in the future, the company hopes to improve its ability to map objects to categories, making it more useful for other types of searches. One example might be helping users find new recipes that are alike in ways beyond having similar photos of food.

“I think our models can become more semantic,” says lead visual search engineer Andrew Zhai, referring to the idea of using deep learning to effectively map images to more conceptual categories. “We can eventually get better at those types of pins.”

In the meantime, Pinterest’s engineers have focused on perfecting object detection and search, with an eye toward potentially developing an app that would let smartphone users take pictures of objects in the real world, then get recommendations of related pins on the platform.

“It’s just such an exciting time in the deep learning, computer vision field—everything moves so quickly,” Kislyuk says. “The state of the art changes every couple of months.”