Download "Top 10 Mistakes in Market Research"

Easy Online Research Enables Bad Research: A Critique

This is from an email I received in February 2010 from a firm that specializes in providing focus group facilities.  I’ve shown just 2 of the questions.  This is not valid or reliable research.  These are not “learnings” as the author suggests.  The crux of the problem: presenting numbers that look as though they are from a valid survey when they are not. 

How could I use the results presented here? Just a few of the flaws: we don’t know how this group of respondents is defined, we don’t know how many of them are out there in the real world, and this is from surveying members of a  qualitative recruiting panel – surely not a representative sample.  Who knows how many of them were surveyed?  That impacts the reliability, the replicability, of the numbers presented here (think of statistical sensitivity).   The question wording is inappropriate as well, which skews the responses.

The danger here is that a reader who is not well versed in marketing research or survey design may assume this is a valid, representative survey because it is coming from a marketing research company, and use the numbers to help shape business decisions.  Even a savvy reader might be tempted to say “I’ll take it with a grain of salt”  and use the information, when really this is useless information.   That’s a travesty.

Here’s the relevant info in the email:

….(XYZ) Company’s  Highly Involved Grocery Shopping and Cooking Moms.  Our recent short poll of this qualitative panel group uncovered these learnings…   

  • How did the “great recession” of 2009 affect your household grocery spending?
    • Some (decrease of 1-5%) – 35%
    • Moderate (decrease of 6-10%) – 29%
    • None – 15%
    • Considerable (decrease of 11-15%) – 12%
    • Significant (decrease of 16+%) – 9
  • Did you find yourself trading down (switching to a less expensive alternative product) as a result of the 2009 recession? 
    • Yes – 82%
    • No – 18%

And here is another example from a major retailers’ trade journal.  Clearly, the nature of this sample is going to bias the results.

What Makes Consumers Try New Products?




A New Brand Trial survey conducted by [A Marketing Services Firm] among its network of 300,000 independent mystery shoppers and merchandisers when those surveyed were asked what made them buy a new product in the categories studied, four in 10 said “they saw it on the shelf or display.” Promotions such as coupons were mentioned as influencing factors by another three in 10. “Referrals by friends” was the next most popular reason behind new product purchases, while advertising was noted by just 8 percent.

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

One Response to Easy Online Research Enables Bad Research: A Critique

  1. Carol Bowen says:

    Very good point, Julia. This is a pervasive problem, especially in the era of research must be “faster and cheaper.” It’s critical to take the time design a survey so questions are asked in an unbiased manner. It’s also important to know who’s been surveyed, and how sampling may bias information too. The emphasis on constant “faster-cheaper” degrades the profession, and results in poor data.

Leave a reply