Quantitative research has its faults, no doubt. And even though mass research methodologies are far from perfect (self-reporting, random calls, influence recording...whatever) I still hold quantitative syndicated research as a better jumping off point for understanding/direction of a target/audience than 10 little old ladies in Des Moines IA, USA sitting round a qualitative table being manipulated to make sure they say the "right" brand idea statements on video.

Nor don't get wrong that qualitative research doesn't have its place. Understanding the personal face of a target/audience is something that quantitative can't do, so it does has a very valuable place.

Using each in its right place at its right time and learning for both sets of data (not manipulating either to prove the great brand ideas) can aid strategy.

Hello Richard!

This isn't relevant AT ALL to this post, but I just remembered that I did some planning work which could be relevant to the PG Tips argument you started:
http://mrsbelmot.blogspot.com/2006/09/whats-best-thing.html

It links the brand intrinsically with the home environment and removes the need for the 'get the kettle on' tagline.

I hope this is useful for your campaign.

Did you see that I won the vote?

Fantastic post and analogy. Everyone, regardless of profession should read Orwell's 1984 from time to time to remind us to stay open minded and avoid "not thinking". Life is too short and too exciting to let Ministry of Plenty figures to rule it. However I know many people are tempted to submit themselves to Ministry of Plenty figures as they believe it makes life easier and more predictable...sad but I fight against it as much as I can.
Excellent blog! cheers :-)

Excellent observation.

If orthodoxy IS unconsciousness, then there are far far too many brands in a coma right now...

Does it seem to anyone else like certain product types (like cars and beer) are suffering from a lot of Newspeak...

Many years ago a statistician bod at AMV told me that most quantatitive market research was fundamentally flawed. I'm doing this from memory but his point went something like this: the significance tests used by market research companies are designed for random samples. Most market research, tracking studies etc, are quota samples. If a quota sample is used then a factor of between 1 and 2 has to be built into the significance test. If this was done on an average tracker with a sample of 1000 people then there would need to be roughly 90 percentage points between two scores to make the data significant. As this would mean that nothing of any significance would ever come out of a study (and because true random sampling would be too expensive), research companies tend to leave the factor out. So, effectively, if he was right (and I'd love to hear that he wasn't) most of the data we rely upon is actually meaningless.

I have heard similar points from statisticians and the usual twist is "but it all depends on the question". Back to George Orwell for a minute. Some of his other work is very thought provoking too.
Road to Wigan Pier, Down and Out in Paris and London and Homage to Catalonia (all in the Penguin Complete Non-Fiction of George Orwell). Worth a read. Sorry to sound like the local book club.

Phil is absolutely right that standard significance tests are flawed for most of market research, since we use quota samples.
He is absolutely wrong when he says you therefore need 90% difference for significance.
Here is a link that discusses the effect for the FRS: http://www.dwp.gov.uk/asd/frs/2003_04/methodology/reliability.asp

You'll see it affects the standard error by a very small amount.

For those not interested in reading the statistics, just think of the last tracking study presentation you saw. If there really was that much wobble in the data, the lines would be all over the pace; instead the more usual criticism of tracking studies is about lines that don't move!

So - yes, almost all market research companies are guilty of not taking the Design Factor into account when calculating significant differences. But the reality is it makes little difference to interpretation.

But - statistics aside - if the quant figures don't make sense to you, it is often helpful to check what question was actually asked. Sadly that can sometimes get lost in over-interpretation!

Don't you just hate it when the truth gets in the way of a good story?