To get skin tones right, Pinterest’s AI went way beyond facial recognition

Among the many ways people use Pinterest is as a visual search engine for beauty and style tips. Since 2018, the social sharing platform has let users filter searches by skin tone, so they can find examples of looks that match their complexions.

“Before we launched skin tone ranges in 2018, nearly sixty percent of the top 100 search terms for skin-related searches involved a tone, such as dark, pale, and olive tones, showing people wanted a way to customize their searching,” wrote Nadia Fawaz, Pinterest’s technical lead for fairness in AI, in an email to Fast Company.

Now, the company has unveiled a new version of its technology designed to more accurately detect skin tones across a broader class of images, quadrupling the number of beauty and fashion pinned items where its algorithms can spot a skin tone.

Pinterest’s skin-tone AI is used in a range of features letting people use the platform to find custom looks for themselves, including a recently unveiled augmented-reality Try On tool that lets people use their smartphone’s camera to see how various lip colors look on them. That’s good for Pinterest users who are using the platform to shop—and, of course, for Pinterest advertisers participating in the Try On program. The company claims millions of users come each month seeking beauty ideas.

Rethinking the challenge

For the new version of the skin-tone technology, Pinterest engineers moved away from their earlier reliance on facial recognition AI. Avoiding it meant that Pinterest’s skin-tone algorithms wouldn’t incorporate any biases from algorithms for spotting faces, such as if they had a harder time handling faces with darker skin, an issue that’s been found with AI algorithms across the industry. It also means Pinterest’s skin-tone systems aren’t limited to images that feature faces, so they can be used for close-up pictures of clothing, accessories, or manicured nails that include skin.

The engineers worked with designers to label tens of thousands of diverse images, including some with faces obscured by accessories such as sunglasses, ones with facial hair, images diverse in gender, pictures of body parts besides faces, and photos captured after the coronavirus pandemic erupted.

“As the world changed, so did images on Pinterest,” wrote Fawaz. “For instance, the pandemic caused a surge in face mask images to which the later [AI] models had to adapt.”

Pinterest’s team also purposely included images the initial version of the skin-tone system couldn’t handle, so they could be successfully processed by the new algorithm. And the company ran internal tests, capturing feedback from internal users about the search results, then ran tests with external users.

One indicator that the new technology was working was seeing comparable boosts in adoption rates on the skin tone filtering features in the U.S. and other English-speaking countries, a sign that the algorithm could, in fact, handle diverse hues. Moving forward, Fawaz writes that the company is exploring other ways to let users find looks that accurately reflect them without having to painstakingly craft text queries to find what they’re looking for.

“People shouldn’t have to work extra hard by adding additional search terms to feel represented, to feel beautiful,” she writes. “We believe that it is easier to find inspiration when you feel included and represented within the product.”