The Treachery of Surveys

1. You Cannot Measure What Has Not Happened.

When you ask a person about an experience that exists only in their imagination, they will give you imaginary answers.

You can measure only what has already happened.

In other words, you cannot measure what “would” or “would not” work. You can only measure what “did” and “did not” work.

2. The Question Influences the Answer.

“A question, even of the simplest kind, is not, and never can be unbiased. The structure of any question is as devoid of neutrality as its content. The form of a question may ease our way or pose obstacles. Or, when even slightly altered, it may generate antithetical answers, as in the case of the two priests who, being unsure if it was permissible to smoke and pray at the same time, wrote to the Pope for a definitive answer. One priest asked, “Is it permissible to smoke while praying?” and was told it is not, since prayer should be the focus of one’s whole attention. The other priest asked if it is permissible to pray while smoking and was told that it is, since it is always permissible to pray.”

– Dr. Neil Postman, New York University

3. Focus Groups are Plagued by a Basic Flaw of Human Psychology.

When a person is asked to sit in judgment, they go to a different place in their mind. They react as a critic rather than as a customer.

Asking a stranger to be a judge does not qualify them to be one.

On page 8 of today’s rabbit hole, Indy Beagle will entertain you with video highlights of two hidden-camera focus groups as they evaluate a potential TV ad for Apple. It is tragicomic to watch these honest, well-intentioned focus group participants reveal their prejudices and inexperience. In the end, both focus groups conclude the proposed TV ad is badly conceived and recommend to Apple that it not be produced, never realizing they were evaluating the script and storyboard visuals for the most successful TV ad in history.

4. What People Believe (and Say) They Will Do is Different From What They Will Actually Do.

In the words of Harvard Business School professor Gerald Zaltman, “The correlation between stated intent and actual behavior is usually low and negative.” Zaltman goes on to describe how Hollywood films and TV pilots—virtually all of which are screened by focus groups—routinely fail in the marketplace, and 80 percent of new products or services fail within 6 months when they’ve been vetted through focus groups.

Most of the thoughts and feelings that influence consumers’ behavior occur in the unconscious mind. “Unconscious thoughts are the most accurate predictors of what people will actually do,” Zaltman said in an interview. People think they will make an objective, transactional decision, when in reality they will make a subjective, relational one.

We believe we will decide with our mind. But in the moment of truth, we decide with our heart.

5. Data can Show You the Outcome of Your Past Decisions, But it Cannot Tell You How to Do What Has Never Been Done.

Do not look to survey recommendations when you seek innovation.

Innovation is a product of intuition.

Was stereo invented because customers said, “Instead of the music coming out of just one speaker, why not have part of it come from a speaker on the left and the rest of it come from a speaker on the right?”

Customers did not ask for stereo but after they were exposed to it, they couldn’t live without it.

Did Steve Jobs develop the iPhone because customers told him they wanted cameras in their cell phones?

Was inventory-on-demand perfected as a result of customers saying, “I think it would be better if you waited to create the product until after I order it?”

Did Tony Hsieh invest in Zappos.com because people told him they would like to buy shoes online without first trying them on? Yet 10 short years after Hsieh invested $2 million in Zappos, the company was making so incredibly much money that Steve Bezos bought it for $1.2 billion.

Keep in mind that Zappos charges full-price for shoes. So any argument of Zappos having an unfair “price advantage” goes out the window. Zappos elevated customer service to a new level and changed an entire industry.

Do You Really Need a Group of Strangers to Give You Permission to Do What You Want?

After many years of conducting focus groups for America’s largest companies, Joey Reiman, a founding partner of the BrightHouse Institute, told the New York Times, “Focus groups are ultimately less about gathering hard data and more about pretending to have concrete justifications for decisions that have already been made.”

If you have a weak idea that requires no courage and isn’t going to raise any eyebrows or make a difference, surveys and focus groups will tell you that you should definitely go ahead and do it.

But when you have an idea that can change the future of your company, those same people are going to tell you it’s a horrible idea and that you have lost your mind.

Don’t spend the money on a survey.

Save it to buy champagne when it’s time to celebrate.

Roy H. Williams