Merrill Dubrow lists four reasons why the market research business is broken:
He then asks a very simple question: what can we do about it?
I think that those issues are all intertwined. #4 is a consequence of #3, which is a consequence of #1 and #2. So, we need to fix #1 and #2 to fix #3, which will allow us to fix #4.
Here is my entire comment:
There are generally two ways to fix a problem: rely on a government/industry body, or let the market speak. When it comes to respondent cooperation, my point of view is that we need to come up with industry standards to allow the market to speak.
In a way, the market already speaks: respondents vote with their feet! While I am all for education and have nothing against raising funds for a good cause, the truth is that most surveys produced by our industry are not worth fighting for. If I had to educate someone, it would be survey designers, not respondents.
Thankfully, consumers today are too smart to just believe what they hear in commercials. They think surveys are a waste of time and none of us can blame them for that. Our surveys are generally such a poor experience that without incentives there would probably be no research industry at all. So, what we need is a way to incentivize ourselves to come up with better surveys, to not ask what we already know.
The technology is already there to make more exciting surveys, to mine data and store existing data. You don’t even need technology to write better shorter surveys. If we wanted to, we could come up with better surveys. But why would we? All of this has a cost, so what is the benefit of making all these additional investments when competition doesn’t and clients look at cost first?
In other words, what I think is missing is an accepted measure of survey quality and an incentive for researchers to improve their surveys.
An idea which comes back once in a while would be to let respondents rate surveys and show the rating to other prospective respondents. User-generated reviews work for books and restaurants, why not surveys? Not everyone is a fan a user-generated data, so another idea would be to let an organization rate all surveys going live. With either approach, long/boring surveys would be so poorly rated that few respondent would take them. Over time, the market dynamics would force researchers to improve their survey design, new design specialists would emerge to help researchers optimize their survey designs, poor technologies would be forced out of the market, some sample companies would differentiate themselves by only providing sample to high-quality surveys, etc.
The problem with that market-driven approach is that everyone needs to play the rules for it to succeed. If not, clients who (knowingly or not) abuse respondents with poorly-designed surveys will just move on to find another vendor. Here is a sad fact: few researchers have the guts to turn down business. So, we all accept to collect data, or recruit sample for poorly designed questionnaires. At the end of the day, since respondents can’t tell the difference between a good and a bad survey, they increasingly assume that all surveys are bad… And the only way to find respondents is to rely heavily on incentives, with all the problems it comes up with (professional survey takers, fraud, satisficing, etc).
That might be the role that the industry bodies could play: ensure that ratings are “fair and balanced”, help agencies share their ratings.
Thoughts anyone? As researchers and survey designers, would any of you be willing to let respondents rate your work?
Merrill asks us what we can do about it. While I cannot speak officially for OTX, I will say this: the current “noise” around data quality has the benefit of finally allowing us to ask our clients the tough questions about survey experience, allow us to push some better (but more expensive) survey designs, and re-launch tools which were sitting there but not used by a very cost-driven industry, prepare new ideas that now make more sense. We have already started to let users rate all our surveys, and while we don’t show the ratings back to the user, we are building a knowledge base to help our researchers understand where they stand and make progress. Again, all of this takes time so I cannot give more details, but I hope that I will be able to share some exciting new initiatives in the next few months.
2008 will be a year of change!
Hi Olivier,
I couldn’t agree more. We’ve all seen too many poorly designed surveys. It would be interesting if anyone has done any research on what respondents think contributes to a good survey. Is it length, content, layout, etc. This by it’s nature, it may be difficult to measure which leads me to my question.
If users were to ‘rate’ a survey, what kind of measure do we use? Is it a scale question, 4, 5, 7 or 9 point? Is the rating based on enjoyability or some other factor?
Unfortunately, I think many may fear adding a few questions on research would burden or deter respondents from continuing.
As you stated, the technology to make surveys better is already here. The internet has made it quicker to field and collect survey responses, but few are actually making surveys better. There are a few stars but these may be the exception.
Once criticism that I’ve always had is that surveys are still this linear series of questions. Mail, mall and internet surveys are not all that different. This notion needs to be challenged.
Cheers,
Paul