Kudos to John Zogby to respond with a lot of humor. According to John, they called 6,000-7,000 people for a sample size of 850 respondents. If it’s true, that’s rather good…
In the meantime, AAPOR investigates…
John Stewart on polls (and respondent cooperation)
Published January 18th, 2008 in market research. 1 CommentIn a recent post, I expressed my skepticism about the fad of branded social networks, where marketers expect people to “engage” into “authentic conversations” with a toothpaste or a new checking account. The Wall Street Journal’s Emily Steel covered the topic today. While Emily’s article is a nice introduction to the WSJ’s general business readers, I would like to add a few comments:
Disclaimer: OTX manages proprietary panels (branded or not), which some might consider as competing products to private networks. Proprietary panels are simple programs where users opt-in to participate in research projects (quant or qual) once in a while for a particular brand.
Merrill Dubrow lists four reasons why the market research business is broken:
He then asks a very simple question: what can we do about it?
I think that those issues are all intertwined. #4 is a consequence of #3, which is a consequence of #1 and #2. So, we need to fix #1 and #2 to fix #3, which will allow us to fix #4.
Here is my entire comment:
There are generally two ways to fix a problem: rely on a government/industry body, or let the market speak. When it comes to respondent cooperation, my point of view is that we need to come up with industry standards to allow the market to speak.
In a way, the market already speaks: respondents vote with their feet! While I am all for education and have nothing against raising funds for a good cause, the truth is that most surveys produced by our industry are not worth fighting for. If I had to educate someone, it would be survey designers, not respondents.
Thankfully, consumers today are too smart to just believe what they hear in commercials. They think surveys are a waste of time and none of us can blame them for that. Our surveys are generally such a poor experience that without incentives there would probably be no research industry at all. So, what we need is a way to incentivize ourselves to come up with better surveys, to not ask what we already know.
The technology is already there to make more exciting surveys, to mine data and store existing data. You don’t even need technology to write better shorter surveys. If we wanted to, we could come up with better surveys. But why would we? All of this has a cost, so what is the benefit of making all these additional investments when competition doesn’t and clients look at cost first?
In other words, what I think is missing is an accepted measure of survey quality and an incentive for researchers to improve their surveys.
An idea which comes back once in a while would be to let respondents rate surveys and show the rating to other prospective respondents. User-generated reviews work for books and restaurants, why not surveys? Not everyone is a fan a user-generated data, so another idea would be to let an organization rate all surveys going live. With either approach, long/boring surveys would be so poorly rated that few respondent would take them. Over time, the market dynamics would force researchers to improve their survey design, new design specialists would emerge to help researchers optimize their survey designs, poor technologies would be forced out of the market, some sample companies would differentiate themselves by only providing sample to high-quality surveys, etc.
The problem with that market-driven approach is that everyone needs to play the rules for it to succeed. If not, clients who (knowingly or not) abuse respondents with poorly-designed surveys will just move on to find another vendor. Here is a sad fact: few researchers have the guts to turn down business. So, we all accept to collect data, or recruit sample for poorly designed questionnaires. At the end of the day, since respondents can’t tell the difference between a good and a bad survey, they increasingly assume that all surveys are bad… And the only way to find respondents is to rely heavily on incentives, with all the problems it comes up with (professional survey takers, fraud, satisficing, etc).
That might be the role that the industry bodies could play: ensure that ratings are “fair and balanced”, help agencies share their ratings.
Thoughts anyone? As researchers and survey designers, would any of you be willing to let respondents rate your work?
Merrill asks us what we can do about it. While I cannot speak officially for OTX, I will say this: the current “noise” around data quality has the benefit of finally allowing us to ask our clients the tough questions about survey experience, allow us to push some better (but more expensive) survey designs, and re-launch tools which were sitting there but not used by a very cost-driven industry, prepare new ideas that now make more sense. We have already started to let users rate all our surveys, and while we don’t show the ratings back to the user, we are building a knowledge base to help our researchers understand where they stand and make progress. Again, all of this takes time so I cannot give more details, but I hope that I will be able to share some exciting new initiatives in the next few months.
2008 will be a year of change!
Maybe a Panel Peak, but no peak yet for survey takers
Published October 6th, 2007 in market research and sample. 3 CommentsRay points to an interesting post from Pete Comley about what he calls “Peak Panel” in reference to “Peak Oil”. Pete Comley is one of the researchers that inspires me most for his original thinking. I can’t recommend enough the reading of his paper “the game we play” from last year’s ESOMAR conference, and will listen carefully to his next presentation in Orlando at the end of the month, as I listen every time he talks about panels.
I tend to agree with Pete’s assumption that the population of panelists has reached its peak in the US, but with a caveat. I would say this: it depends what you call a panelist, or what you call an online panel. It’s not so much a matter of opinion actually (and who cares about mine anyway), but simple facts.
I have seen a clear erosion of the number of respondents each online panel is able to deliver in the past 3 three years that I have been sourcing online sample. The fresh respondents have mostly come from new entrants in the market, but most new generic gen. pop. online panels only resulted in creating more duplicates between databases, except for the rare few that used alternative recruitment methods. I also agree that there is finite universe of people for whom online panels are relevant. I don’t know if it’s 10% of the population. As we have all learned over time the number of members is irrelevant and what matters is the active population which I would define as the maximum number of people who would take a survey if the entire panel was invited. Based on my actual numbers and extrapolating to the US market, I would go for something like 1%.
But once we have talked about online panels, we have only covered half of the topic. Because online panels are only a small fraction of the universe that really matters, the universe of potential survey takers. I am not sure all we got all started with this panel business thing. Before online panels, some companies had tried to generate email addresses randomly and send email invitations to take surveys. Why not, random generation of email addresses is not so different from random digital dialing after all. But of course that was spam which, not only is illegal, but is now very inefficient. So, what followed is the birth and growth of online panels as a way to let users “opt in” to take surveys. That was 1997. But, while online panels were just one device to obtain permission, 10 years later they have become the norm.
But there are other ways than online panels to get permissions from Internet users to take surveys. Joining an online is agreeing not to take one survey, but to take many surveys, regularly. An online panel is only relevant to users who agree to take several surveys, after which they will have enough points to convert their points to a reward, or cash, or whatever. The irony is that market researchers don’t even need these very people they recruit in their panels. They attract them with promises to make money taking surveys, but then they treat them as professional survey takers and try to get rid of them.
What researchers really want is not a panelist but a survey taker. Take CATI/RDD as a comparison. Respondent cooperation is low because who wants to take a 30-minute survey at home after a long day at work, when the family is waiting for you to cook or watch TV? But once in a while someone falls for it. Sure, most people think that never again will they pick up the phone at night, and, as far as I know, professional survey takers are not a big concern for CATI/RDD (there are other issues, but it is not the topic here). So, and that can be applied to online research too, while there is only a small portion of the population that likes to take surveys often, there is a vast majority of people that don’t mind taking a survey once in a while.
In the past two years, my team and I have worked with one goal in mind: find online respondents, wherever they are. Today, some of our respondents come from online panels, but the vast majority comes from somewhere else. It is sometimes hard to explain to clients who can’t even envision online research without online panels, but fortunately a growing number of clients are starting to get it.
Potential survey takers are everywhere:
- permission to take surveys can be obtained anywhere, from the registration page of any web destination or even from web traffic. The question “do you want to take a survey now?” can be asked anywhere, by any recruiting agent
- incentives are all around you. So many destinations have loyalty point-based systems, not only panels and rewards sites. Any site or community with an economy (real or virtual) can entice their users to take a survey.
Of course it’s not easy, there are plenty of new challenges that arise with non-panel respondents, from potential biases to duplicates, drop-offs, sample consistency, delivery and many more. But who said it was easy?
In the end, the good news is that we have not reached the survey taker peak yet. The not so good news is that (online) researchers need to rethink the way that they conduct online research to use non-panel sample. The same way that car manufacturers need to re-engineer their cars to use alternative energies.
Kids like to party, so let’s build some walls
Published September 25th, 2007 in market research. 1 CommentAt first, this sentence doesn’t make so much sense… But it’s more or less the kind of things I have been hearing in the past few months.
I give you the logic: online communities and social networks have never been so popular, and people go increasingly online to socialize. So… brands should build their own community to talk to their customers.
Am I the only one to see the slip of logic here?
Engagement is about relevance. Asking someone to join yet another community when there are so many interesting new ones out there every day is not relevant. It’s a waste of your customers’ time.
Conversations are before all spontaneous and authentic. A community starts with a common interest that brings people together. That common interest provides context for the community to exist. My conversations on Wannasurf and AsmallWorld are not the same. Paying someone everyday to post a blog about your deodorant is not what I call an authentic conversation. Spending a fortune to have a cool design will not provide context. If incentives are your only way to make your community relevant, why don’t you just stick to quantitative research, there is something called “open ends” that helps you collect a large number of qualitative insights very easily, as long as you read them. Paying someone to spend time with your private community is not engagement, it’s corruption.
A message to brands: people are not waiting for your authorization to talk about you. They don’t need to be in your community to talk about you. In fact, they are talking about you right now.
So, if you want to listen and observe kids, you don’t need to organize a party yourself, and even less to build walls and hope that they will come and party. In fact, you don’t even need to organize the party yourself, because it will probably be lame anyway (Wal-Mart, you know what I am talking about - at least you failed early). So just join the party yourself and look around you! There are so many exciting things marketers can do to leverage existing communities and networks…
PS: a tip for brands. Don’t reinvent the wheel. If you just need a social network and let your customers share photos, videos, blogs etc, just create a Ning account and have your own MySpace in 20 minutes. Add $5 per month and you have it on your own domain. Now you have your own walls social network, and you saved a lot of money to do something useful with it.
Bacn vs Spam
Originally uploaded by jted
I like new words when they help understand a new concept.
The concept of “bacn” needs to be understood by anyone involved in e-mail marketing, which, with 50% of market research being conducted via online access panels, includes market research professionals.
From Wikipedia:
Bacn (pronounced bacon) is the term given to electronic messages which have been subscribed to and are therefore not unsolicited but are often unread by the recipient for a long period of time. Bacn is email you want but not right now.
They differ from spam messages in that they are not unsolicited and are not necessarily sent in bulk. Bacn derives its name from the idea that it is “better than spam, but not as good as a personal e-mail”
Common examples of Bacn messages include news alerts, messages from social networking sites and wiki watch lists.
You can add online survey invitations to the list.
I am often asked whether online sample should be opt-in, or double opt-in or whatever. Of course, you must make sure that people agree to be invited to take surveys. A panellist is nothing more than that. Someone who, at some point, agreed to take a survey. Just like someone who picks up the phone once and agrees to take a phone survey. But it doesn’t mean that he will want to participate in all your future surveys. And if he doesn’t take any of your surveys for a few weeks, it doesn’t mean that he wants out. So single or double or triple opt-in doesn’t matter. More opt-ins during registration will not help your future survey invitations become more relevant for your members and customers.
For the same reason, panel sizes are irrelevant metrics. What matters is how many unique users would take a survey at a given time if you invited all your panellists.
Like many things in life, permission is not black or white. “bacn” introduces the notion that there is a very large grey area between what you want to hear about and what you don’t want to hear about.
When you think about survey invitations by email, think about this: they are “bacn”. They are not spam. They are legal, but probably not the most important thing for your panellists when they receive it.
Don’t confuse “online research” and “panel research”
Published August 8th, 2007 in market research and sample. 0 CommentsI don’t know if I should be amused or worried about the new controversy surrounding the ISO and ARF “competing” initiatives to improve online research quality.
According to Research Magazine, “Erich Wiegand, chair of the ISO working group developing online access panel standards, has voiced his concern over the news that the Advertising Research Association (ARF) has embarked upon its own standards initiative”.
Now, I may be a little retarded or picky, but my understanding is that ISO is working on standards for “access panels” while the ARF is looking at standards for “online research”. I understand that, in the past few years, many online panel companies have emerged and gone public, that there is a vague consensus that using access panels is an efficient way to get respondents for online research.
But, if we all know that “access panels” are today an acceptable solution for many online research projects, it is just one solution among others, and maybe a temporary one. In some cases, it is not the best option.
Yes, the Internet is changing research, and guess what - we have only seen the beginning of the change. “Online access panels” started to appear at the end of the 90s, at a time when Google did not even exist. Innovation is all around us, and research tomorrow will not be done the way it is done today. It took a long time for the idea of an “online access panel” to spread, for researchers to accept the methodology, for research companies to launch their own copy-cats.
But the ISO’s worries about the ARF initiative mean implicitly that they consider that any online research project is necessarily using “online access panels” or, even worse, should use “access panels”, which is at best a candid mistake. If I was paranoid, I would say that panel companies are trying to protect the benefits of a position it took so much time and money for them to establish by nurturing this illusion that online research and online access panel research are the same thing.
The challenge with a professional blog in a competitive environment is that it is not possible to share everything. I have to keep most juicy discussions and experiments to myself because we work in a competitive market and because there is a first-mover advantage. But I will tell you this: there is life after the online access panel.
When manufacturers want to produce a higher-quality product, one way to do it is to look at all processes and try to improve each process separately. Another way is to make things differently. It is more difficult, sometimes more painful, but in the end that’s what really makes a difference (read Seth for more on this). And by the way, that’s what they call innovation.
So here are my two cents for the industry: while regulating to make things better, let’s be careful not to prevent anyone from making things different.
Update: it turns out that, while the ARF’s initiative is about “Online Research Quality”, it heavily focuses on “access panels”, which may explain the controversy. Since I have been invited by the ARF to participate in the Council, I will be able to give more details about all this in the near future.
-
Says Steve Rubel: page views are officially dead
Blogbang: a network for consumer-generated advertising
Published July 9th, 2007 in blogging. 3 CommentsMy friend Julien Braun just launched in France a new ad network called BlogBang . BlogBang is a marketplace for advertisers, “creatives” and publishers (bloggers).
An advertiser submits a creative brief, “creatives” (consumers, agencies, etc) generate their own commercial (mostly video), which show on the publisher’s blog. Example here or here. Publishers/bloggers can pick ads that might work well based on their editorial content. An optimization system (I am not sure how it works, but I think something like advertising.com) determines over time which ads work better for each blog.
The interesting part is that creatives and publishers split the revenues (with the ad agency of course). And anyone can be a creative and try to compete with the professional agencies to create the most efficient ads. As opposed to traditional creative contests, which are judged on their creative qualities, the best ads here are those that perform better. So, consumers who understand how to create ads that would perform well with a particular audience, on a particular blog, etc, can get a share of the action and help publishers make more money with ads that perform better.
Julien’s philosophy is that each media has its own advertising format (30 seconds for TV etc) but that social media has not found yet a format that works well. By letting users create their own ads, advertising becomes social and, while serving the advertiser’s brief, has a human voice.
I feel that there is something very interesting there. At least because a marketplace is a way to make “consumer-generated advertising” more scalable. But since I myself am not an expert, I am curious to hear what my favorites CGM bloggers like Pete , Max , David, Joseph and the other guys from Crayon or elsewhere think about it.
Julien previously worked at Publicis who took 60% of BlogBang. After the acquisition of Digitas, Publicis shows real ambitions to innovate in online advertising.
If you speak French, you can hear Julien’s interview by French blogging celebrity Loic Le Meur.
-
A couple of weeks ago, the movie theater down street, which has 12 screens, was only playing Shrek, Pirates and Spiderman. And as a result, I didn’t go see a movie, thinking about how inefficient that distribution system is.
