Giving research the needling it deserves
Using acupuncture as an anaesthetic during surgery. Image originally uploaded here
I spent some time yesterday talking to a friend that is studying Chinese Medicine.
She is training to be an acupuncturist and she is interested in the different research methodologies used in evidence based medicine.
This is a big issue for complementary therapies in general as they believe in using qualitative as well as quantitative research to understand a treatment’s efficacy since most complementary health treats the whole person and not just observable symptoms.
And it is a big issue for acupuncture in particular as the standard means to test any medical treatment, the randomised controlled trial (RCT) is notoriously problematic for this therapy. You can undertake a RCT but it involves using sham needles on the control and so while a single-blind test is therefore possible the gold standard of a double-blind test is out of the question.
She wanted my help in understanding different research methodologies.
But as a planner working in marketing communications I realised I could be of little help since I rarely come across research. Or rather I rarely come across real research and our little discussion finally proved this to me.
What we spend most of our time dabbling in is the application of once credible research techniques in wholly inappropriate ways to tasks for which it is ill suited. Indeed, that the sham research in which we engage goes by the name of research is one of the greater crimes against the English language.
Let me be a little more specific.
There appear to me to be three types of ‘research’ that we use in the marketing business, and only one should carry the title.
When we are using quantified data that reports actual behaviour by consumers I think we can call that research. Whether it is derived from the behaviour of the universe (e.g. a brand’s volume share) or from a sample of the universe (a brand’s penetration) it tells us about something that has actually happened in the real world. And when we test something in the market and report on it’s success I think we can call that research too – a regional launch, split copy testing, and classic direct marketing testing. This is all data you can depend on and make decisions with.
What we call qualitative research is also equally useful, when it is used appropriately. But when we use it appropriately I’m not sure that it deserves the name research either. Qualitative research works best in our business as ‘food for thought’ and as a powerful source for creating insight. Qualitative research, doesn’t tell us what is going on, at best it gives us some ideas about what might be happening. And it does this in a way that may help to trigger a new way to define or crack a problem in the planner’s mind but is laughable as a means for making critical business decisions.
I had a client recently that refused to call ‘creative development research’, research at all and maybe we should follow that lead. How about calling it qualitative exploration from now on, after all that is what we are doing it for – to have a little subjective, unaccountable rummage in pursuit of a little nugget that we can use to help understand what people think and why and what might change their minds.
Both real research and qualitative exploration are legitimate activities.
What passes for ‘quant’ nine times out of ten in this business is the bastard, as sham as those needles used on acupuncture control groups.
Whether it is a Usage and Attitudes survey, Pre-testing or Tracking these activities offer the illusion of respectability and certainty by borrowing the clothes of the kind of research that gets drugs approved for launch but represent little more that quantitative speculation.
The problem for me is part philosophical and part practical.
Philosophically the idea of quantifying claimed behaviour emotions and attitudes seems a most peculiar activity. It may be interesting in and of itself but it gives us very little idea of the behavioural implications. Medical researchers don’t ask people whether they think that the drug they have just tried is likely to work on a five point scale. They find out whether it did work using proper research.
And practically, as the medical profession knows all too well, undertaking proper quantitative research is a expensive and time consuming and the two things marketers are strapped for right now are cash and time. How many times have you given or sat in a debrief where the cell sample sizes you are seeing precentaged are lower than 100? How often is it that the quotas for the samples have not been met or are unbalanced because the researcher agreed to turn around times that are way too tight.
Do this kind off stuff by all means – after all we need some help to make decisions as we negotiate our way through the complex and risky process of building effective marketing communications. But lets stop calling it quantitative research.
So from now on I am going to reserve the name research for the real deal – actual data that reports on reality.
I am going to continue my affection for great qualitative exploration, it is not the only or often best source of insight but sometimes it is dynamite.
But when it comes to the other stuff, I’m going to call it quantitative speculation, I invite you to join me.
Discover more from
Subscribe to get the latest posts sent to your email.
Interesting… I think I’d agree with that; my post on my blog about research being like the weather doesn’t really have what you term ‘quantitative speculation’ in mind – more figures and ‘hard’ data.
So aye, I’m with you.
I’m with you on this.
On the pressed for “time and cash” issue, the Market Research (“quantitative speculators?”) people can often be quite cunning in some of their packaging.
I once had the misfortune to have my ads pre-tested in Canada by something called the PAR Test (which probably stood for Persuasion Action Return, or some such nonsense).
The ghastly thing about this was that you had to hit the “PAR” of “4” in order to be any good : perfect for the ten second catch up in the lifts between the CEO and the Marketing Director : “Did those limey ads hit Par, Danny? “No Boss, they just missed the cut at 3 and a half”. “Hiring a new agency?” “Good idea! Anyway how was Augusta?”.
Can’t quite see Doctors doing it like that.
Finally I like your re-branding suggestion. It reminds me of a response to the Daily Mail/Express re-branding of Inheritance Tax as “The Death Tax”. Calling it “The Lucky Kids Tax” is more revealing.
Certainly worth ditinguishing between qualities of research: after all, you get a better view of tower bridge on a clear day than a foggy one.
I’m a bit concerned about the absolute demarcation though – you seem to be suggesting that the view on a clear day is “right” and all the others are “wrong”. I don’t have time to set out why this would be (a) incorrect and (b) dangerous, but it is certainly something to be aware of (and if you’re not saying that then I apologise – the danger of quick blog comments).
Surely the point is just to never let people quote results without reflecting on the methodology – the two, after all, are mutually dependent.
Hey Jon, I actually used to work on PAR and PPS scores so I totally get how frustrating they can be for the creative agency. And I agree that they’re a ridiculous benchmark of how well the film works. And whether or not to bonus the agency!
But like James, I think there’s a danger in labelling all research as rubbish. When conducted and used in the right context, quant too can provide useful info which may spark off a thought or lead to an insight. I think it’s a question of approach and being aware of the caveat that most research is conducted in an artificial environment. It’s an attempt to understand how people may/behave or react. In isolation it rarely makes sense or is useful to a marketeer, but when put in as part of a composite picture, I do believe it has its role to play. That’s me – ex copywriter (hated the black box of research) ex researcher (learned how to work the research), now planner (finally, getting to understand the damn research!)
I started reading this post with trepidation but agree with your assessment of qual. I’d happily describing myself as a qualitative explorer rather than researcher.
I’d agree that many dubious conclusions are formed on the basis of extrapolations that are equally as dubious. And as with Qual research, ethnography can be brilliant, but only if used appropriately…Erwin Ephron also has written a good piece on this subject…
http://www.ephrononmedia.com/article_archive/articleViewer.asp?origin=AR&articleID=150&categoryID=24&categoryName=Research+
Absolutely no idea what you’re going on about.
But I just wanted to say that your choice of images for your posts is consistently excellent.
What I find to be a very funny consequence/observation to what you’re saying is the fact that there are agencies that are given extra money for their work depeding on the results of the client’s tracking studies. In other words, agencies are earning money due to some quantitative speculations. From this perspective, agencies seem to have very smart business models. :)
what I enjoy about adliterate is that it is a stretch and thought-provoking … and as Ben said, your pix are the best!
Qual is often used to explore things rather than to find things out (eg creative dev) and to that extent, I agree that ‘qual exploration’ IS a better description.
BUT.. (1) exploration is part of research! And (2) you can explore something well or not very well. And (3) qual exploration done well with the right people might better than a chat with your mate down the pub one lunchtime (but that might depend on the mate). And (4) if the exploration is done well, what is not ‘real’ about that? The holistic medicine qual research you quote at the start – is that ‘real research’? Even if we are all unreliable narrators (according to Mark Earls), surely honest introspection provides ‘actual data’ which is ‘reporting on a reality’?
You raise a fascinating (to me) issue about what makes research research. The thing is, if you include other applications of qual than just creative dev, what happens to your definition of real or proper research? Surely any ‘real’ research must be reliable, replicable and evidence-based. Just because someone says something, doesn’t mean it’s not true! People generally don’t systematically lie. So if you are doing qual which is NOT telling you something about ‘what is going on’, something is missing. So, qual that is about ideas/inspiration, call it exploration, absolutely. The source and quality and relevance of the ideas are not as relevant as the ideas it yields. But real qual research CAN provide this AND can tell you something about the world, or at least the people in it. And CAN be a valid input into (but not a substitute for) clients “making critical business decisions”.
Hope there’s more on this post! btw as I know nothing whatsover about medicine and double blind trials etc I have sent this to someone who does. Let’s see.
As a qual researcher this post made me choke on my coffee. Your problem with the word ‘research’ is that it should only be used to describe ‘double-blind’ tests for drugs, otherwise it’s a spurious use of the sacred ‘r’ word.
Well in a sense I have no problem with Qualitative Exploration, it sounds much grander than research. I’d far rather introduce myself at drinks parties as a brand Explorer rather than a Researcher, however it might sound even more pretentious, and therefore an even greater semantic crime.
I suppose my problem is that you are so dismissive of what we do. In marketing and advertising finding out what people think and why and what might change their minds is neither ‘little’ nor ‘unaccountable’ nor ‘a rummage’, nor ‘a nugget’. It’s useful and important work, which is worth a lot to clients.It’s a pretty succinct definition of what we all do for a living. And calling it ‘research’ is far less pretentious than various words commonly used in marketing and advertising. Like ‘strategy’ for instance, or dare I say ‘creative’. I once met a man from Y&R who was Creative Director of the Western Hemisphere. Now that’s pretentious. Perhaps we should reserve the word ‘creative’ for those who have had their work hung in the Royal Academy. And ‘Director of Strategy’ for people who work in the Pentagon or who are above Colonel level in the MOD. And as for so called ‘Planners’. . .well you get my drift.
for a reply to this post, visit http://www.theartofconversation.net
Another researcher, another opinion…
Rather than ‘exploration’ I think ‘transformation’ is a more suitable term – see here http://artofconversation.typepad.com/art_of_conversation/2007/03/transformation_.html
I really have to reply to this on my own blog as I’m spluttering at the screen with your casual association of qualitative and “bad” as inter-changeable concepts.