Can you trust Dr. Google? Data scientists say search engines are giving bad health advice

The digital world is constantly evolving—Facebook is Meta now!—yet internet search remains a magic box where we type questions and answers just appear on the screen. Critics say this disconnect has led to cluttered design, ads camouflaged as search results, even a search algorithm that some say shows signs of wear: ProPublica reporter Jesse Eisinger recently tweeted he thinks Google searches have gotten “significantly worse,” and he got 1,000 likes and nods of agreement from everyone from Chris Hayes to Time national correspondent Molly Ball.

A new study out this week looks at this from a different angle: The role of search engines as people’s go-to health advisers. Data scientists from Germany’s Martin Luther University of Halle-Wittenberg and Russia’s Ural Federal University wanted to see how search engines are managing medical misinformation. Given the myths that have circulated since the start of the pandemic, the authors argue it’s more important than ever to promote accurate and credible online health resources. After all, we live in a time when people panic-buy horse dewormer as a COVID-19 cure, and a White House press corps member raves on social media that the vaccines contain “a bioluminescent marker called LUCIFERASE so that you can be tracked.”

So the researchers studied the results to health-related queries they searched for in Google and its Russian counterpart, Yandex. They claim both sites’ top results often included bad health advice. To get these findings, they waded through 1.5 billion Yandex searches to identify the 30 most common queries that contained both a medical problem and some sort of alternative remedy (garlic to treat a toothache, ginger to cure a cough, leeches to stop hemorrhoids, milk thistle for hepatitis), some of which are medically dubious.

These queries were plugged into Yandex, and the authors examined the snippets for the top 10 results. Snippets are the short strings of text underneath the blue result links that search engines like Google and Yandex automatically extract to show how well webpages match users’ queries. This is all that most users will ever know about most of the pages that appear in their search results, which is why the authors chose to analyze only these snippets, and not to look at the pages themselves. Next, they entered the same queries into Google, and studied the snippets for Google’s top 10 results.

“We also annotated whether the snippets contain warnings about the potential health risks of using one or another alternative medicine, since some searchers would just use them at home,” lead author Alexander Bondarenko tells Fast Company.

Their findings were that nearly half the Yandex snippets confirmed some kind of medical misbelief, while only 13% noted that the treatments are linked to health risks. The Google snippets’ issues were “not as pronounced,” they write, but they were “still worrisome, since we found evidence of potential health risks for all questions in our study.”

Their takeaway is that search engine snippets for health-related queries don’t necessarily point people to medically sound resources. Rather, they may be reinforcing bad traits—confirmation bias in users, content bias in search engines—that the authors warn are “especially relevant” when users are researching health.

Fast Company showed Google the study, and a spokesperson responded that its methodology “is not an accurate way to measure the quality of search results, since it only looks at short snippets of text from the website, not who authored the website or what information it actually contained.”

Google also argued there are flaws in their research. For instance, it says the study oversimplifies complex medical treatment questions into yes-no binaries. (Leeches may be an unwise DIY home remedy, but studies show they can reduce hemorrhoid pain for some people. Also, the authors say it’s dangerous for pregnant women to drink coffee—even though the World Health Organization says occasional small amounts are OK.) Google adds the study doesn’t acknowledge, either, that a two-line snippet on the results page could contain seemingly solid medical advice, but people rarely base their internet research on snippets alone, and this doesn’t mean the website itself would be reputable and error-free.

High-stakes searches

Google has started to stress that health-related searches fall into a special category it calls YMYL (“Your Money or Your Life”). This category encapsulates things like medicine, finances, news, and voting information—high-stakes topics that, if presented wrong, could negatively impact a person’s health, happiness, financial stability, or safety. As an example, the company noted that googling “covid vaccines are dangerous” triggers an extra-high bar for pages to clear to ensure they’re trustworthy, authoritative, and relevant. Fast Company tried googling that phrase, and indeed the top hits were well-respected, geographically relevant health authorities (the CDC, FDA, Johns Hopkins, Mayo Clinic) or news outlets (the New Yorker, Reuters, NPR, Wall Street Journal).

Then again, Google’s search algorithm would probably rank those sites very high regardless. Niche queries—such as asking if an herb cures some ailment—seem exactly like the kinds of searches that could be problematic, and when Fast Company googled some of the paper’s search queries, it didn’t square with that laser-focus on health. The top 10 results for “Does garlic help with a toothache?” included three bigger sites—Colgate, Livestrong.com, and Healthline—but the rest were dentist offices located in Texas, California, Alabama, Michigan, and Australia that have recommended trying it, with one stating: “Garlic does wonders for tooth pain. Garlic has antibiotic properties that will help heal your gums.”

Meanwhile, for another of the study’s questions, about the poppy plant celandine being used to cure cancer, Google’s top featured snippet says, “Early research suggests that injecting [celandine] intravenously under medical supervision improves survival in some people with colorectal, bladder, pancreatic, or breast cancer.” But WebMD says, “Not enough is known about the safety of giving greater celandine products intravenously. Stay on the safe side and avoid use.”

Google may or may not want to have to police this, especially with so much other misinformation to deal with, but it looks like a problem somebody has to think about.