Will Data Science Make Your Business Instincts Irrelevant?

For now anyway, intuition is alive and well in the era of data analytics. Only about a third of executives in a 2014 PwC survey reported relying first and foremost on the data for their last big business decision. For the rest, the advice of their colleagues and good, old-fashioned gut instinct both played major roles.

It’s no wonder, is it? Leaders tend to get where they are at least partly on the basis of their good judgment, which has guided them to success time and again. These days, with a deluge of new data insights closer to hand than ever before, wading through them all can seem overwhelming or even distracting. We’re still in the early days of big data—and of data science in business writ large—so it’s probably not a surprise that subjective measures still play a predominant role.

That doesn’t need to be a bad thing, but sometimes it is. Here’s what you need to know in order to keep your gut instinct and the cold, hard numbers in good balance.

Will Intuition Ever Be Obsolete?

It’s worth asking whether we’ll eventually be able to remove gut instinct and intuition from our decision-making process altogether, even if we wanted to.

Netflix founder Reed Hastings, who worked as an artificial intelligence engineer before starting the video company, cautions that the human element is still critical. “We start with the data,” he recently told VentureBeat, “But the final call is always gut. It’s informed intuition.”

While experts have found that intuition can be powerful, it can often lead us astray. In general, even the most experienced decision-makers among us are often quite irrational, making choices that aren’t in our best interests, acting on impulse, and blind to unseen biases.

Success comes from connecting the dots between our emotional selves and systematic thinking that can be checked quantitatively. It’s all about how you combine data, predictive analysis, and your time-honed intuition.

Intuition Helps Us Dig Deeper

Getting that balance right starts with understanding which types of real-world situations demand more or less subjective versus quantitative evaluation.

In 2012, researchers Erik Dane of Rice University and Kevin W. Rockmann of George Mason University found that people most rely on intuition when making broad evaluations, particularly in areas where they already have in-depth knowledge of the subject—often called “domain expertise.”

Today organizations may have huge amounts of data to analyze, but the challenge is to define what they can reasonably expect from any effort of data analysis. And intuition still comes into play when defining which questions to ask in order to churn out actionable insights. Here are a few:

• What market to purse
• What products and services to develop
• What business model to adopt
• What partnerships to pursue

It’s judgment and experience—often in the shape of gut instinct—that should and do cause business leaders to raise those questions at the right times. But it’s data analysis that can help answer them in their particulars, like these:

• Market segmentation
• Customer and product portability
• Customer retention strategies
• Operations and performance management
• Resource requirements

To be sure, this is an intentionally broad and far-from-comprehensive breakdown. But distinguishing between the strategic and tactical questions can sometimes still be a challenge; up until quite recently, both were subject to basically equal doses of intuition. And while market research has long been a staple, it’s only lately that data analysis has come to reshape how businesses approach the latter type of insight.

Validation Is Mandatory

Today a massive amount of data is being collected in real time, which makes the “human with machine” (instead of “human versus machine”) formulation more important to get right. Algorithms and systems can process information more efficiently and accurately than ever, which significantly increases that information’s business relevance.

Where past decisions were based on historical precedent, like customer satisfaction surveys and sales patterns, machines can now create predictive models, patterns, and trends that incorporate what-if scenarios. Here’s how author of Big Data: Using Smart Big Data, Analytics and Metrics To Make Better Decisions and Improve Performance Bernard Marr recently summed up that shift on LinkedIn:

U.S. retailer Target is now able to very accurately predict when one of their customers will expect a baby. Using big data, Telecom companies can now better predict customer churn; Walmart can predict what products will sell, and car insurance companies understand how well their customers actually drive.

But that hints at a reverse imperative, too—which is easier to miss: Just as we can now check our intuition with more reliable metrics than ever, we also need to subject our predictive models—what data tells us might happen—against our best-informed hunches. Information on what is happening, or already has, may be a surefire baseline to square our instincts against, but when it comes to the future, data may be able to guide us ahead even though intuition (so far, at least) still largely leads the way.

It won’t always be this way, of course. As data innovation speeds forward, the ways we combine our instincts with the numbers will continue to evolve.

But as they do, it’s unlikely that the quantitative and the subjective realms will ever be totally mutually exclusive. The challenge and opportunity for organizations is—and will continue to be—how best to harness their combined power. It’s a science, sure, but it’s a human one.

Serial entrepreneur Faisal Hoque is the founder of Shadoka, which enables entrepreneurship, growth, and social impact. He is the author of Everything Connects: How to Transform and Lead in the Age of Creativity, Innovation, and Sustainability (McGraw-Hill) and other books. Use the Everything Connects leadership app for free.

Copyright (c) 2016 by Faisal Hoque. All rights reserved.