Facebook downplays “ambient audio” tech that can eavesdrop on you

Amid persistent rumors that Facebook is spying on you through your phone’s microphone–rumors that the company has denied since at least 2016–reports emerged this week that the company has applied for a patent on technology that listens to “ambient audio” to detect what you’re watching on TV.

Essentially, your phone or another device would detect, via Bluetooth or other signals, that it’s near your turned-on TV and would record short snippets of sound. TV shows and ads could include high-pitched signals identifying precisely what you’re watching, and that data could be beamed back to a central server to “update user profiles of the individuals” in your household in order to better target users with personalized content, according to the patent application filed in late 2016.

This isn’t the company’s only technology for capturing and identifying audio signals in a user’s environment. In another patent application filed in 2015, Facebook describes a system that can capture data about what a user is watching by using “acoustic fingerprinting” to analyze ambient audio, or even by analyzing electrical signals emitted from an HDMI cable or a television. Neither patents mention listening in on users’ voices or conversations.

When reached by Fast Company, Facebook said it will “never” actually put the patent-pending technology to use. According to an emailed statement from Allen Lo, VP, deputy general counsel and head of intellectual property at Facebook:

It is common practice to file patents to prevent aggression from other companies. Because of this, patents tend to focus on future-looking technology that is often speculative in nature and could be commercialized by other companies. The technology in this patent has not been included in any of our products, and never will be. As we’ve said before, we often seek patents for technology we never implement, and patent applications should not be taken as an indication of future product plans.

And when asked during his April U.S. Senate testimony whether Facebook used “audio obtained from mobile devices to enrich personal information about its users,” Zuckerberg answered, “No.”

Of course, the company stores audio and video deliberately uploaded and shared by its users, and its terms of service give it fairly wide latitude to analyze and process that data. But the company says it has no plans to use ambient sounds to shape ads or users’ feeds.

“Facebook has never used your phone’s microphone to inform ads or to change what you see in News Feed, and we have no plans to do so in the future,” said Facebook’s vice president of ads Rob Goldman in a statement shared with Fast Company.

Meanwhile, rumors suggest that Facebook will soon launch a smart speaker product similar to Amazon Echo and Google Home, which would likely revive theories of spying, even unintentionally. Amazon devices have been known to misinterpret conversations as commands to record and transmit audio, and the devices generally work by constantly listening for cues in people’s conversations, even if they don’t retain that incidental audio or send it to the cloud.

It’s already happening

Capturing audio to figure out what you’re watching might sound outlandish, but it’s actually being done. A company called Alphonso does just that, using modules in apps such as games to listen to TV audio and identify shows, the New York Times reported last year. The company says it doesn’t pick up human voices, and users of apps with the code can opt out at any time.

In 2016, the Federal Trade Commission (FTC) wrote to developers warning them that if they used another TV-observing app module by data firm Silverpush and didn’t properly notify users, they could be violating the law. At the time, Silverpush said its technology wasn’t in use in the United States, according to the FTC, but vowed that it would kill the software anyway due to a “business decision.”

A year later, however, researchers found over 200 Android apps still using Silverpush’s inaudible sound “beacon” technology to spy on what TV commercials users were watching. The apps, including some used overseas and connected with large companies like McDonald’s and Krispy Kreme, had collectively been downloaded millions of times from the official Google Play market, the researchers wrote in a May 2017 paper.

Many smart TVs already do phone home with information about what you’re watching, provided you give consent in the often-dense menus you see when you first get the device home and plug it in. The TVs generally don’t need to listen to external audio to learn what you’re watching, since they already have that data. By gathering your IP address, broadcasters and advertisers can also use that information to match your viewing habits with other information known about your online behavior.

For instance, 4C, an analytics company and early Facebook and Twitter marketing partner, uses various techniques to help determine users’ offline activity and pair digital ads with live TV moments and commercials. Before it collapsed, disgraced elections firm Cambridge Analytica was touting similar techniques.

Even if marketers aren’t literally listening in, hackers and stalkers have been known to take over webcams and microphones to spy on innocent people. (Even Facebook CEO Mark Zuckerberg reportedly covers the camera and mic on his laptop.) Last year, a leak of CIA cyberweapons revealed a tool for capturing audio recorded by Samsung’s Smart TVs. And some companies offer spying apps that you can install on smartphones (often targeted at employers and parents) to monitor use, including turning on the microphone to eavesdrop as you see fit.

Related: Hackers Could Use A Pop Song To “Watch” You Through Your Smart Speaker

To prevent clandestine audio monitoring by apps, users can use their smartphone privacy settings to disable an app’s access to the microphone. This, however, can also prevent a number of useful features, like recording video with sound.

The bottom line is that advertisers have found they like the wealth of data they can get about your online habits, and there’s likely to be a continual push to find ways to get similar data about what you’re watching, listening to, and even discussing in the physical world. Even if Facebook isn’t listening, that doesn’t mean it technically can’t or that it won’t some time in the future.