Last week, Apple previewed a number of updates meant to beef up child safety features on its devices. Among them: a new technology that can scan the photos on users’ devices in order to detect child sexual abuse material (CSAM). Though the change was widely praised by some lawmakers and child safety advocates, it prompted immediate pushback from many security and privacy experts, who say the update amounts to Apple walking back its commitment to putting user privacy above all else.

Apple has disputed that characterization, saying that its approach balances both privacy and the need to do more to protect children by preventing some of the most abhorrent content from spreading more widely.

What did Apple announce?

Apple announced three separate updates, all of which fall under the umbrella of “child safety.” The most significant — and the one that’s gotten the bulk of the attention — is a feature that will scan iCloud Photos for known CSAM. The feature, which is built into iCloud Photos, compares a user’s photos against a database of previously identified material. If a certain number of those images is detected, it triggers a review process. If the images are verified by human reviewers, Apple will suspend that iCloud account and report it to the National Center for Missing and Exploited Children (NCMEC).

Apple also previewed new “communication safety” features for the Messages app. That update enables the Messages app to detect when sexually explicit photos are sent or received by children. Importantly, this feature is only available for children who are part of a family account, and it’s up to parents to opt in.

Apple

If parents do opt into the feature, they will be alerted if a child under the age of 13 views one of these photos. For children older than 13, the Messages app will show a warning upon receiving an explicit image, but won’t alert their parents. Though the feature is part of the Messages app, and separate from the CSAM detection, Apple has noted that the feature could still play a role in stopping child exploitation, as it could disrupt predatory messages.

Finally, Apple is updating Siri and its search capabilities so that it can “intervene” in queries about CSAM. If someone asks how to report abuse material, for example, Siri will provide links to resources to do so. If it detects that someone might be searching for CSAM, it will display a warning and surface resources to provide help.

When is this happening and can you opt out?

The changes will be part of iOS 15, which will roll out later this year. Users can effectively opt out by disabling iCloud Photos (instructions for doing so can be found here). However, anyone disabling iCloud Photos should keep in mind that it could affect your ability to access photos across multiple devices.

So how does this image scanning work?

Apple is far from the only company that scans photos to look for CSAM. Apple’s approach to doing so, however, is unique. The CSAM detection relies on a database of known material, maintained by NCMEC and other safety organizations. These images are “hashed” (Apple’s official name for this is NeuralHash) — a process that converts images to a numerical code that allows them to be identified, even if they are modified in some way, such as cropping or making other visual edits. As previously mentioned, CSAM detection only functions if iCloud Photos is enabled. What’s notable about Apple’s approach is that rather than matching the images once they’ve been sent to the cloud — as most cloud platforms do — Apple has moved that process to users’ devices.

Apple

Here’s how it works: Hashes of the known CSAM are stored on the device, and on-device photos are compared to those hashes. The iOS device then generates an encrypted “safety voucher” that’s sent to iCloud along with the image. If a device reaches a certain threshold of CSAM, Apple can decrypt the safety vouchers and conduct a manual review of those images. Apple isn’t saying what the threshold is, but has made clear a single image wouldn’t result in any action.

Apple also published a detailed technical explanation of the process here.

Why is this so controversial?

Privacy advocates and security researchers have raised a number of concerns. One of these is that this feels like a major reversal for Apple, which five years ago refused the FBI’s request to unlock a phone and has put up billboards stating “what happens on your iPhone stays on your iPhone.” To many, the fact that Apple created a system that can proactively check your images for illegal material and refer them to law enforcement, feels like a betrayal of that promise.

In a statement, the Electronic Frontier Foundation called it “a shocking about-face for users who have relied on the company’s leadership in privacy and security.” Likewise, Facebook — which has spent years taking heat from Apple over its privacy missteps — has taken issue with the iPhone maker’s approach to CSAM. WhatsApp chief, Will Cathcart, described it as “an Apple built and operated surveillance system.”

More specifically, there are real concerns that once such a system is created, Apple could be pressured — either by law enforcement or governments — to look for other types of material. While CSAM detection is only going to be in the US to start, Apple has suggested it could eventually expand to other countries and work with other organizations. It’s not difficult to imagine scenarios where Apple could be pressured to start looking for other types of content that’s illegal in some countries. The company’s concessions in China — where Apple reportedly “ceded control” of its data centers to the Chinese government — are cited as proof that the company isn’t immune to the demands of less-democratic governments.

There are other questions too. Like whether it’s possible for someone to abuse this process by maliciously getting CSAM onto someone’s device in order to trigger them losing access to their iCloud account. Or whether there could be a false positive, or some other scenario that results in someone being incorrectly flagged by the company’s algorithms.

What does Apple say about this?

Apple has strongly denied that it’s degrading privacy or walking back its previous commitments. The company published a second document in which it tries to address many of these claims.

On the issue of false positives, Apple has repeatedly emphasized that it is only comparing users’ photos against a collection of known child exploitation material, so images of, say, your own children won’t trigger a report. Additionally, Apple has said that the odds of a false positive is around one in a trillion when you factor in the fact that a certain number of images must be detected in order to even trigger a review. Crucially, though, Apple is basically saying we just have to take their word on that. As Facebook’s former security chief Alex Stamos and security researcher Matthew Green wrote in a joint New York Times op-ed, Apple hasn’t provided outside researchers with much visibility into how all this actually works.

Apple further says that its manual review, which relies on human reviewers, would be able to detect if CSAM was on a device as the result of some kind of malicious attack.

When it comes to pressure from governments or law enforcement agencies, the company has basically said that it would refuse to cooperate with such requests. “We have faced demands to build and deploy government-mandated changes that degrade the privacy of users before, and have steadfastly refused those demands,” it writes. “We will continue to refuse them in the future. Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government’s request to expand it.” Although, once again, we kind of just have to take Apple at its word here.

If it’s so controversial, why is Apple doing it?

The short answer is because the company thinks this is finding the right balance between increasing child safety and protecting privacy. CSAM is illegal and, in the US, companies are obligated to report it when they find it. As a result, CSAM detection features have been baked into popular services for years. But unlike other companies, Apple hasn’t checked for CSAM in users’ photos, largely due to its stance on privacy. Unsurprisingly, this has been a major source of frustration for child safety organizations and law enforcement.

To put this in perspective, in 2019 Facebook reported 65 million instances of CSAM on its platform, according to The New York Times. Google reported 3.5 million photos and videos, while Twitter and Snap reported “more than 100,000,” Apple, on the other hand, reported 3,000 photos.

That’s not because child predators don’t use Apple services, but because Apple hasn’t been nearly as aggressive as some other platforms in looking for this material, and its privacy features have made it difficult to do so. What’s changed now is that Apple says it’s come up with a technical means of detecting collections of known CSAM in iCloud Photos libraries that still respects users’ privacy. Obviously, there’s a lot of disagreement over the details and whether any kind of detection system can truly be “private.” But Apple has calculated that the tradeoff is worth it. “If you’re storing a collection of CSAM material, yes, this is bad for you,” Apple’s head of privacy told The New York Times. “But for the rest of you, this is no different.”