Why Your Key Belief About Market Research Is Wrong

Many years ago, I was working at a software startup geared to helping salespeople set the best prices to charge customers. For a rep with a few dozen accounts, each of which purchased dozens of items at a time, the incremental profits added up to big money quickly. Since our tool was built to help sales reps optimize pricing, we felt we had a strong product and a proven user base—but there was a problem.

The overall metrics looked good, but our data showed that the best-performing sales reps weren’t using our product for accounts where the biggest opportunities were. This was worrisome, because we wanted these sales reps to be our main evangelists. So my company sent a few product managers into the field to ride shotgun with the sales reps and do some user research.

What we found out surprised us. Every entrepreneur has been told over and over again to spend time talking to real customers—and that’s good advice. But it doesn’t really get at the true value of market research, which isn’t just about getting heaps of feedback from users and then heeding the insights that crop up the most. In fact, it’s the opposite. The whole point of talking to loads of customers is to listen for the one or two who will tell you something really surprising—the feedback that doesn’t square with everybody else’s. Here’s why.

Inspiration, Several Bites Into A Camel Burger

I set out into the field to accompany a young saleswoman who worked for a big food distributor; we’ll call her Lucy. Lucy’s job was to call on restaurants and keep their kitchens stocked with every possible product you can imagine, and she was very good at her job—exactly the kind of person we wanted singing our product’s praises to her peers.

In what would become a recurring theme in my career, being in the field showed me things I never considered from the comfort of the office. For instance, I never thought about how frequently sales reps’ laptop batteries died on the road, making our product unavailable. I also never realized that the reps usually consulted our product after a customer meeting, rather than before. And I certainly hadn’t considered how our user experience held up when accessing the product on a 3G mobile Wi-Fi card while balancing a laptop on the steering wheel of a car.

Those were all really useful observations. But the biggest surprise came when we stopped for lunch at a pub whose big thing was serving exotic meats. Lucy was scoping the place out, figuring out how to turn the establishment into a customer. I think she had an antelope burger. I distinctly recall having a camel burger.

Now, I’ve heard camel is supposed to be like a cross between beef and veal, but I remember finding it tough and dry. I’ve also heard this means that I probably ate an older camel, which is just . . . super. Be that as it may, it was while I was laboriously chewing my old-camel sandwich that Lucy finally gave me the gold I was looking for:

Lucy: “So, places like this are why I don’t really use your product.”

Me: “Why not?”

Lucy: “We use specialty distributors for places like this to show we can get them the exotic stuff. Since the product doesn’t have those, I don’t use it.”

Me: “Not even for all the other customers?”

Lucy: “Not usually.”

Oof. That was tough to hear. But hooray for market research!

Innovation Doesn’t Follow The Rules Of Statistics

Listening to the market is arguably the most important thing you can do when building a product. As startup founders and entrepreneurs are well aware (and tend to enjoy repeating), products exist to solve problems. That means validating that there’s a sufficient market for a product is mission critical, and it’s the mission that market research is supposed to fulfill. And it’s true that in general, more observations are better than fewer—but not for the reasons you might think.

In statistics, sufficient sample size is critical for justifying your conclusions from the data. When you’re studying individuals’ behaviors as a market researcher, getting a big-enough sample improves your odds of encountering the stray spark that lights your imagination on fire. It may help you understand the baseline, too—what are the most common complaints, which features people like best—but it’s the outlier responses, like Lucy’s, that may be even more valuable.

Those are precisely the responses, though, that many product designers are too quick to discount (in my experience, this is especially true in enterprise software). When Aaron Patzer and his team were testing out the idea for what became Mint.com, he asked 80 people on the street if they would use his tool, Patzer later recalled, and only one said yes. Statistically, that lone “yes” was an outlier to be dismissed. But it was part of what inspired him to continue on to the massive success Mint eventually became, when it sold for $170 million just three years later.

How many other people would have looked at Patzer’s exact same research findings and concluded that the idea was dead on arrival? Most prudent entrepreneurs intuitively follow statistical best practices and toss the outliers in the garbage. But that’s the last thing an innovator should do. Disruptive ideas, by definition, come from out of the mainstream—treasure buried in the long tail of observations.

To be sure, that doesn’t mean you need to seize on the most contrarian feedback you hear and run with it at the expense of all the rest—but you do need to listen to it. I was lucky that Lucy and I sat down in a place that actually elicited the type of surprising feedback I needed. It’s much more common that the spark comes from untold hours of listening to recordings of customer interviews and focus groups.

I’m still amazed by how my entire interpretation of a discussion can change based on a single turn of phrase from the subject. Ultimately, moments like that were what led me to found UserMuse, a marketplace for connecting people like Lucy with software builders who will pay for their feedback.

Making decisions solely based on outliers is risky, but ignoring them is even riskier for businesses over the long term. Interpreting market feedback will never be an exact science, and treating it like one could lead you to miss your own moment of inspiration.

So go ahead—try the camel.

Christian Bonilla designs analytics software at Resonate, despite having no formal engineering training. You can follow his business ramblings at SmartLikeHow.com or on Twitter at @smartlikehow.