Why The Election Won’t Dampen Data Analysts’ Job Prospects

If you read blogs like FiveThirtyEight every day leading up to the election, like me, you may have felt a bit betrayed when the results came in. FiveThirtyEight‘s data and polling provided predictive analysis that overwhelmingly (71.4%) leaned toward a Clinton victory. Others like the New York Times (85%) and Huffington Post (98%) used data to infer that the Democrats would win by even wider margins.

As Clinton lost, there’s been an unsurprising backlash in the form of articles about the flawed science of data analysis and how it woefully misinformed the public.

While it’s true that much of the political analysis done pre-election relied on models that turned out to be flawed, data analysis is not dead, it’s going to be in demand for years to come. And more importantly, big data analysis is likely what won the election.

What Went Wrong?

Ken Strasma is an early champion of micro-targeting for Obama who helped out with the Votecastr project this year, which set out to more accurately report the incoming voting results. Votecastr had a slew of issues as the data came in. The problem—which Strasma can see now—was that people were understating their voting dispositions.

“Pretty much everyone was surprised by Trump over-performing,” he tells Fast Company. This was because people were understating their preference for the Republican then-nominee, Strasma explains. In addition, the polls that Votecastr and other similar projects used were unable to predict turnout. Most often, polls rely on past elections to weight what they expect will happen this time. In this case, it made the expectations incorrect. The ultimate problem that virtually all pollsters saw, says Strasma was that “we saw a very consistent over-prediction of Clinton support and under-prediction of Trump support.”

Another issue isn’t in the data pollsters reported, but in the way it’s consumed. Polling and predictions contain margins of error, and even when those are stated from the get-go, the average reader may not understand the implications. “You very often see that people’s eyes glaze over,” says Strasma.

What Went Right?

All the same, some have been using data to analyze the election, and see better results. Cambridge Analytica, for instance, was hired by the Trump campaign to use data to move the effort forward. Persuasion was their ultimate task for in the finals months of the Trump campaign, according to David Wilkinson, the firm’s lead data scientist. Using polling data, the firm was able to create specialized and highly targeted ad campaigns aimed at winning over voters. “Trump isn’t a typical candidate,” Wilkinson observes, so he had to “build something a lot more customized.”

Every day new polling data would come in and his team would analyze what it meant. Every week, with this data, Wilkinson would create new models and re-weight the numbers to understand where the voters stood. Looking at the information from the outset, Cambridge Analytica’s team knew that the only way for the Trump campaign to be successful was to focus on the Rust Belt. They did this by looking at the polling data and categorizing people between “who is a Trump supporter, who is a Clinton supporter, who is a persuadable supporter,” Wilkinson says. The team scored everyone based on what they could track.

Where the firm excelled was understanding how dynamic these voters were. The polling data would often be recalibrated to account for people’s changing opinions. And seeing this data come in and analyzing it accordingly helped Wilkinson and his team create paths to persuading unlikely voters. “We were able to effect a lot of different things,” he says.

Looking Toward The Future

So while many point to the failures of FiveThirtyEight and Votecastr, others in the data analysis field were actually getting it right. Either way, demand for data scientists (once called the “sexiest job of the 21st century“) and analysts hasn’t suffered. In fact, a recent study surveying found that 82% of the organizations queried “currently have or expect to have positions that require data analysis skills in 2016.” More, 78% said they had “difficulty recruiting for data analysis positions.”

Ronald Wasserstein, the executive director of the American Statistical Association—one of the organizations behind the study—tells Fast Company that over the last few years there has been an increased popularity in the field. “It’s exciting that more and more companies are seeing the value that’s brought by having statistical expertise as part of the companies’ core business strength,” he said.

Wasserstein’s been working as a professional statistician for decades and he’s found it surprising that it’s “become cool to be a statistician.” With this cultural recognition, there’s been an uptick in educational avenues. Wasserstein noted that many universities have been seeing heightened demand for data analysis training. Some schools are adding entire bachelors programs in the field.

At the same time, the data expert admitted that as of late he’s seen a disconnect with the general public and what he does. “The most troubling thing about the conversation is a sense that people have that data can’t be trusted,” he admits. A recent study from KPMG and Forrester Consulting revealed that only a third of over 2,000 corporate executives trust the analytics generated from their business operations.

“My answer to that is what everyone wants is certainty and what the world offers is uncertainty,” says Wasserstein. So while many data analysts got things wrong, they did account for uncertainty, it was just difficult to illustrate that to the general public. Strasma echoed this new tightrope, adding, “I do think a surprise like [the election results] will help people understand that big data is not a magic wand that can make any candidate win.”

The question for the future, then, isn’t whether people will rely on data analytics, but how best to propel the discipline forward, despite the strife about what went on methodologically. Examples like Cambridge Analytica prove why the industry will keep growing. Wilkinson explained that while the techniques he employed showed promising results, it’s only the beginning. “Polling is not dead,” he said, “it just needs to be a lot more sophisticated,” Wilkinson concluded, “We just need to rethink how that technology is used and improve it.”