Back in June, I participated in CASRO’s respondent cooperation panel discussion in New York. In a subsequent post, I promised to post the exact transcript of the discussion once I would receive the tapes. I received the tapes last month, but I have been busy lately, so my apologies for publishing this so late.
The discussion was moderated by Bob Lederer, who is organizing with IIR the Summit on Improving Respondent Cooperation in Chicago next month. I thank CASRO for authorizing me to post this discussion on my blog.
Photo: Worldopinion
Below is the first part of the discussion.
MODERATOR: Good afternoon everyone. My name is Bob Lederer. Peter did something very stupid about two months ago. He called me up and said Bob; you tend to do some offbeat things. You’ve spoken at our high tech conference and we’d like you to do something carte blanche. Do whatever you want to do. We tried to get a horse and we got a camel. Umm, what we’re going to do today is discuss the impact of high tech market research tools and services on respondent cooperation. I’m going to do my very best to keep everybody focused on that issue. High tech and what it has done for respondent cooperation. Dennis touched on it a little this morning. What I was able to do thanks to Casro and Peter was to assemble the group that you see behind me. People who live and breathe respondent cooperation everyday. As a result of that they have very strong opinions. Good and bad about this subject and specific elements of respondent cooperation which is what we’re going to touch on today. I want to take a few minutes and introduce everyone on the panel.
First the Vice President for Data Collection at Arbitron, Dorothy Gould-Smith. The Director of Market Research at Church and Dwight, Barry Goldblatt, Senior Planner at Diageo, Jeff Fink, Senior V.P. at E-Rewards, Kurt Knapton, the Global Director at Online Center of Excellence at GfK and this is for the UK and the United States, Mike Cooke, the President of the Americas and the Executive Vice President of Global Client Services at GMI, Larry Shack.
In the back row, the Vice President of Corporate Marketing and Research at Golf Digest Company, Jon Last. The Chief Technical Officer and EVP at Greenfield Online Hugh Davis. The Director of Market Research at Guardian Life Mr. Mark Wolf, The Senior Research Scientist and Director of Research Methodology for Harris Interactive, Renee Smith, the Market Research Consultant at the Strategic Insight Department at Hartford Life in the Group Benefits Division, April Brackett, the Co-Founder and Chief Operating Officer at Imoderate Carl Rossow.
On the other side left to right, the Chief Technology Officer and Chief Strategist at Incentive Logic, Michael Edwards. We haven’t even met yet, Hi. It’s true for most of the people on this panel and I give them great credit for taking my phone call and accepting my invitation and actually coming here for this today. The President and Founder of Kinesis Survey Technology, Leslie Townsend. The Vice President of Panel Management at Knowledge Networks, Daniel Slotwiner. We haven’t met either yet, hi. Steve Coffey, one of the people I’ve known a long time. We actually gave him an award from one of our publications back in 1996. He is the Chief Research Officer at NPD, Steve Coffey. The Director of Global Research and Communication Insights at the OMD Advertising Agency, Mr. Mike Hess, and the Vice President of Online Community Department at OTX, Olivier de Gaudemar.
And finally in the back row, the New Product Manager at Survey Sampling, Joanne DiNapoli, but before her the Global Vice President and General Manager of Online Services for SPSS Software, David Suedkamp, the President and Founder of Survey USA, Mr. Jay Leve. The Director of Statistical Sampling at TNS, Robert Benford. The Vice President of Services at Vision Critical, Mr. Mike Rodenburgh and finally last and certainly not least, Chairman and CEO at Western Wats, David Haynes. I’d like some applause for this people who have been gracious enough to join us.
Okay, why are we here today. Here’s the scenario. I have recently acquired three major companies. One is CPG, one is B and B and the third is a media company. Don’t ask me who they are I can’t tell you, its top secret. Each of my businesses has a research department, and a darn good one from what I can tell. I’m not your typical CEO of these companies because I believe in market research and in its capacity to drive business and business decisions. I’m here today to really get as comprehensive an understanding of what research tools and techniques can do, good and bad for the research that needs to be done to help me make better decisions. I’ve been told that high tech research has revolutionized the research industry since the mid 1990s. For the purposes of this discussion today, I’m defining high tech research as anything that is not via land line phone, at a mall, through the mail or traditional face to face. I understand that high tech has some very positives and some negatives. I want to thank everyone on this panel today whose going to try to help me get an understanding an a workable knowledge of whether high tech research from the perspective of respondent cooperation is a net plus, a minus or a wash. Okay. That’s the basis from which we’re going to work today.
We’re going to go through a series of issues and questions. My first question is how willing is the U.S. population to engage in any kind of research. Joann Denapli you’re the first one. If you looked at the population, what percentage of the population do you think is willing to cooperate in a research offering online or off-line?
JOANNE: Could you repeat the beginning of the sentence, I couldn’t hear you?
MODERATOR: Okay, what percentage of the population do you believe is willing to cooperate if someone approaches them to do a survey, to take part in a survey I should say?
JOANNE: 30%.
MODERATOR:30%, no higher than that?
JOANNE: It depends on the topic and how relevant it is to the person.
MODERATOR: Bring the microphone a little bit closer and tell us what you mean by that.
JOANNE: Okay. If it’s a relevant study, say you’re doing a study of diabetics and you pull a sample of people who have diabetes, well that’s going to be relevant to them, so they’ll take the survey. You’ll get a high response rate. If you have the same group and you give them a study about Cheerios and they’re not interested in that and it’s probably not relevant that’s going to have a lower response rate.
MODERATOR: Do you think that number, that percentage was ever higher than it is today?
JOANNE: Yes.
MODERATOR: How long ago?
JOANNE: Twenty years ago.
MODERATOR: A long time ago.
JOANNE: It was when I first started at Survey Sampling so I know that number.
MODERATOR: Jeff Fink, percentage of the population that you think is willing to take a survey?
JEFF: Under force? Umm, I think it’s probably about 40% and I think you can correlate that back to the voter population. Yeah, probably pretty close to that.
MODERATOR: Okay, by the way, how long have you been in the industry?
JEFF: I’ve been in the industry for thirteen long years.
MODERATOR: Has it been long?
JEFF: It’s been wonderful, but long.
MODERATOR: Joanne, how long have you been in the industry?
JOANNE: Twenty.
MODERATOR: I think the person who has been in the industry the longest is Barry Goldblack. Grab the microphone, Barry how long have you been in the industry?
BARRY: More than anyone else up here.
MODERATOR: Barry, what percentage of the population do you think is willing to take a survey?
BARRY: I think it depends on how your contact them, when you contact them, what they’re doing. I think I’ll build on what we heard before. I think it’s also a function of the subject. In the end it’s probably, yeah, maybe a third.
MODERATOR: Okay. What do you think it was ten years ago, thirty years ago? You set the parameter.
BARRY: I think the number has decreased significantly. I started in the business when door to door interviewing was still popular. In fact I used to do it. I remember the days when you’d knock on someone’s door and if they were home, they’d open the door, they’d open it wide and they would almost always talk to you. Sometimes it meant inviting you into their home. Sometimes it meant standing on the porch. Sometimes it meant a half hour but they did it and it was almost like it was an honor to do it. They were flattered that anyone asked them their opinion and they liked doing it.
MODERATOR: And today?
BARRY: I don’t think that’s the case. I think some people still feel that way but I think the majority know what market research is, back in those days they kind of didn’t. I think they realize there’s money to be made. You know I look at focus group facilities, I see people just piling in. I’ve been solicited to do groups myself. And if you don’t want to do it they raise the ante. Everyone talks to friends; they know you can make some money doing this. There are sweepstakes on line, in a lot of ways it’s turned into kind of a commercial game.
MODERATOR: From your capacity on the client’s side as the head of a research department, and you’ve been at J&J before, Church and White and I think you were at P&G before that. Okay contrast for me your level of comfort with the research you’re getting strictly based on who do you think is answering the surveys that are going out, versus where you were previous in your career.
BARRY: You know we’re clearly making decisions and we’re making the same kind, the same level of decisions off the data, whether its from high tech research, telephone, mail or whatever it happens to be. So I feel okay about it. In my particular case I work for a company that has a lot of brands and a lot of categories, we have some small share brands. We have some very difficult consumers to contact. So, with all the pluses and minuses of high tech research for me, it’s a huge advantage because I can talk to consumers that I was never able to talk to before.
MODERATOR: Olivier what do you think about the number of unique respondents that exist in the industry today as opposed to five or ten years ago?
OLIVIER: First of all I wish I knew. I try to know what we buy. We buy approximately one million interviews per year. So, if we look at the numbers we saw this morning that’s probably around 3% of the market. I think we’re a pretty large buyer of sample in the market. We’re only 3% of the industry, not enough to have a perfect vision. But if you look at the numbers, if you have an industry of 1.6 billion dollars with a $40 CPI, on average that means we do 40 million interviews per year in the U.S. alone. I would assume from what I see that the population of people who took a survey, unique respondents, is probably around five to ten million users, which is around 2-5% of the U.S. Internet population). It should be more and I hope that we can all work together to find out how many there are.
MODERATOR: Okay. Do you think there are more total respondents taking a survey today then five years ago, or ten years ago out of the whole population?
OLIVIER: Well, I’ve only been here seven years. I don’t have the experience of many people around the table. I think it really depends on how you define a survey. I think we’re in the new world today where you take surveys all the time. Surveys are out there all the time. Customer satisfaction surveys, research surveys, the way we conceived them very traditionally, people are giving their opinion all the time. People vote for American Idol. How many people voted for American Idol? Probably the way we conceived research surveys maybe less than before but it’s not so exciting. I’m French, obviously. I come from a country where there’s a marketing research company called Ipsos. You may have heard about this company. When you’re French, you may think that Ipsos is the government. And Sofres, which is now TNS, you also think it is the government. People were watching the presidential elections on TV once every five years, and they saw the poll results from Ipsos - so when they were contacted by Ipsos, they thought it was an honor. They said “I’m going to be a citizen and give my opinion:. Well you know, the world has changed, people are more cynical. They know that marketing research is a business and they want something in return. They don’t think it’s an honor anymore. They need a reason to give their opinion. There are so many fun ways to give your opinion online, in social networks, everywhere…
MODERATOR: Jay Levy you had a thought about total number of respondents.
JAY: No I just have a housekeeping point. For those of us back here, it’s very difficult to hear both the moderator and the panelist. If you could adjust one of the speakers to give us feedback we would appreciate it.
MODERATOR: April Brackett, what do you think about the number of people willing to take a survey? Now you’ve actually got consumer focus and B to B focus is that correct?
APRIL: That’s correct.
MODERATOR: So what do you think about those two entities separately and the willingness to take surveys?
APRIL: Can you state the question again please?
MODERATOR: Okay, you have two different entities in your business that you cover. Consumer is one, B to B is another. Contrast those as to their willingness to take surveys.
APRIL: Within our category in the insurance industry, umm, we actually have a harder time on the business to business side reaching senior level executives or HR managers, the actual decision makers within an employer. As well as actually reaching the brokers who work with our employers. We often have listed samples and we have numerous ways of incenting employers and brokers to participate. With the right incentive they’re willing to participate. Consumers on the other hand we have, it’s a little bit easier for us to use. We traditionally use panels and focus groups and online focus groups to get the numbers that we need. So again, they’re incented.
MODERATOR: Larry Shack, are more people willing to take surveys today? Can we have all the microphones on here?
LARRY: There was a prior point that was made which I think is key to answer this question which is, how do you define a survey? Umm, I think that people are definitely more willing to give feedback and information today then any other time in the past. Particularly when something strikes them as interesting. American Idol is a great example, if you go onto any website that might have a spot poll on the issue of the day you can see tens of thousands of people on that day and the marks will move. So that actually goes back to something that Bob asked me about which scares me and I think drives down and that is participation is really the clients that we all work with in some respects and the idea that I’m going to take what used to be a twelve to fifteen minute standard on a survey and make it an eighteen minute survey, a forty minute survey, a forty five minute survey. In the past, people did not want to take that type of survey. When you try to replicate that on line you say it’s going to be ten or fifteen minutes and then it turns out to be forty minutes, that’s where we begin to run into problems.
MODERATOR: Carl Ross, more people or less people taking surveys today and why?
CARL: More people are taking surveys today because of the spot polls on all the different websites, in restaurants, hotel rooms.
MODERATOR: Does do it yourself also feed into that?
CARL: Yes, the do it yourself does feed into that. I think a lot of these do it yourself surveys are impacting respondent cooperation.
MODERATOR: So you think customer service and do it yourself and probably the samples themselves that everybody has . . .
CARL: Individuals are participating in surveys that are not written by researchers and therefore don’t understand how to write proper surveys and in fact are impacting responding cooperation because individuals become bored with surveys. They’re not intrigued with what the survey is about.
MODERATOR: Anyone else on the panel we haven’t expressed about cooperation and whether people are willing to cooperate?
MIKE: Unless I missed it, I think we didn’t hear the line about people being concerned that they’re going to be sold something that it’s not really a survey and I think that holds it down as well. I think because of that skepticism, especially if they’ve been burned in the past, then there are less people as a function of starting a survey, taking a survey and turning it into a sales routine instead.
MODERATOR: Show of the audience, how many people think people are more willing then ten years ago to take a survey? How many people think people are less willing than ten years ago to take a survey? How many people truly are uncertain one way or another?
Okay, can we get a microphone over there?
RESPONDENT: Depends on the number, which way you’re looking at the number, I guess maybe I’m too much into research.
MODERATOR: If we were looking at the total U.S. population.
RESPONDENT: As a larger percentage of the population?
MODERATOR: Is there a larger number out of three hundred million Americans?
RESPONDENT: I think more people are giving their opinions each day through all the various surveys that are conducted. There are just too many surveys out there, the do it yourself and all that so I’m actually not sure whether there are more people doing the survey or less percentage of people cooperate today. There are so many more surveys out there.
MODERATOR: Now who put their hands up and said they think more people are willing to take surveys than ever before?
RESPONDENT: I responded in a sense that, go back to the earlier comment that more people are giving their opinion across the board.
MODERATOR: Do you think they’re willing to give their opinion?
RESPONDENT: Willing or doing – blogs exist for people to give their opinion about various things. So I think the desire for potential respondents to give their opinion is very, very high. I think the way we go about asking for that opinion can drive them into misery. But I think they’re desire to give their opinion about any given topic can be very, very high, especially if it’s something they want to talk about. So I see the overall desire by the U.S. population and ability because of technology to do so to be ever increasing, but maybe not in the way that we think is informative or scientifically valid for our purposes.
RESPONDENT: Bob, I’d like to tie together Larry’s comment and this gentleman’s comment which is there is a greater desire than ever to have your voice be heard. I think the word survey actually impedes our ability to talk about that as opposed to facilitates it. My mother and father don’t think they’re giving survey responses when they speak out and tell a restaurant or hotel or car rental place what they thought about the service. They just think they’re being advocates for themselves. So I think that in general, people are more – there’s a time and a place when people would open the door wide and let you in and yes that phenomenon has changed. People are more circumspect about opening their home door but they are offsetting that. They are more willing than ever to speak up and tell you what’s on their mind. The whole society, American Idol in particular is driven on participatory democracy and that’s to our great advantage.
MODERATOR: As a CEO of these three companies I’m getting a little bit better idea of what this is all about. I’ve got a question for everyone on the panel, in twenty five words of less, define respondent cooperation. Dottie?
DOTTIE: Well using the toughest definition for us to respondent cooperation is someone who consented, and umm actually follows through so it’s a little more difficult in our case.
MODERATOR: Okay, David Haynes?
DAVID: In three words cooperation is willingness to participate. Usually defined as a rate of cooperation. A cooperation rate which would be the number of completed surveys as the numerator and the denominator being eligible respondents contacted.
MODERATOR: Okay, John Last?
JOHN: I would probably add to the definition willing participation in an objective and candid way.
MODERATOR: You’re reading, that’s not fair.
JOHN: I wrote it down when you asked it because I wanted to make sure that I added this part about candid and frank because I think from my perspective from a number of vantage points that we look at things I have a very serious issue with the whole topic of professional respondents and I’m sure that we’ll touch on that a little bit later so I would add candid and frank as part of that definition.
MODERATOR: We are going to touch on that later. Okay good, OLIVIER?
OLIVIER: I would say there is a short and a long one. The short one in online research I don’t think that for panel-based research there is a good definition. The longer one is I think it would be a product of the panelization rate, the response rate and the drop off rate. One key element in there is the panelization rate. I mean, how many people in there have seen banners to join the panels? Probably every Internet user. That’s probably two hundred million people. But if you ask the biggest sample providers, there is no one that has more than five million Internet users, although we can assume that they have contacted two hundred million people. That means the panelization rate is only 2.5%.
MODERATOR: Barry?
BARRY: From my perspective who is a buyer of research. What I expect is that the respondent is going to do what I want them to do. What I want them to do is pay attention to the questionnaire. I want them to answer the question. I want them to think about each question, read each question, listen to each question if it’s a listen question, honestly answer the question, take time to do it and complete it.
MODERATOR: Okay, Mike Roddenberg?
MIKE: I actually want to echo what Barry just said. The idea that there is a stimulus or impetus to participate in research is fundamental to the cooperation rate which is why I would reject the notion that quick survey polls, online polls that have no stimulus other than an opted in volunteer response isn’t necessarily research and shouldn’t fall into that definition of respondent cooperation.
MODERATOR: U. Davis?
U. DAVIS: I think respondent cooperation is the willingness of a potential survey taker to actively and honestly participate in the entire research process.
MODERATOR: Okay, Mike?
MIKE: In the media company we do a lot of targeting so we typically try to narrow the target before we do a survey so starting with that I would say the response rate is the percent of the qualified target that we reach and in fact complete the survey, that percentage.
MODERATOR: Okay, Jeff?
JEFF: I would define it as a consumer that we’re able to reach; we’re able to convince to take it and one that gives reliable data, echoing everyone else. I think the other point I would put in there is that they actually have a positive experience with that. We don’t actually think about that very much but it goes into cooperation rates in the future.
MODERATOR: Great, Robert?
ROBERT: I think it’s simply the compliance with a request for an interview.
MODERATOR: Okay, Mark?
MARK: Respondent cooperation I think I would start with what’s a respondent? A respondent has to meet a set of criteria important to the decision or the outcome of the research. Once you define that number of people, what proportion of those people, whether it’s a sample of a census, what number of those people who complete fully the activity, whether it’s a survey or a focus group or online chat. That becomes the numerator. The denominator becomes what’s the census or sample size of the people that meet the definition of respondent.
MODERATOR: Great thank you, Steve Coffey?
STEVE: There are really two words that we’re throwing around as if they’re interchangeable and they’re not. One is the response rate, the other is cooperation rate. We can all have our own definitions. Response rate is typically used by me to define out of a defined universe or population group that I want to make a defined measurement of, for example all adults in the United Sates, among those I contacted, what percentage agreed to participate in the research. That is a loose definition of response rate. Then cooperation is among those that agree, how many completed the survey and gave me qualified or competent answers. I think as we moved into online panels, we’re throwing around cooperation rates which is when I send 100 survey invitations out how many surveys do I get back as if it’s a comparable metric to the more traditional media response rates which there is no comparison whatsoever because the true response rates in online panels if we’re lucky is one or two percent.
MODERATOR: I’m going to get back to you about some of the points you just made in a couple of minutes. Curt?
CURT: I think of respondent cooperation as being a trust based mutually beneficial exchange between the researcher and the individual. Not just that one survey of that point in time but leaving the world better off from where you were there where that respondent wants to cooperate the next time, where they had a positive experience. I think as an industry we know what we want in terms of valid answers and cooperation but we failed to think from the respondent’s perspective and we failed to be as respectful as we should be in terms of telling the truth about how long the survey will be. We many times don’t tell them what we expect. We many times don’t provide a reward or the reward is not appropriate or meaningful to the respondent and many times not relevant to them. We should know their name at a minimum if we’re going to talk to them.
MODERATOR: You’re getting beyond where I wanted to go, but that’s good. Jay?
JAY: The active, willing and eager desire to be part of a process that will help the government run better and make the free market more efficient.
MODERATOR: It just flowed right off you’re tongue like you’d heard that. Renee?
RENEE: I agree with Curt in a lot of ways. I think one of the things that have happened in the industry is that we’ve had a too narrow definition of respondent cooperation as an action or set of actions that happened for a single survey. Cooperation to me implies a long run sort of relationship. It also implies a mutual relationship. We have plenty of things that we do in our surveys or that clients ask us to do in our surveys that give people a bad impression. So if we continue to just think of cooperation rates as the way in which we dispose of cases – the fact that we even use the term dispose of cases sort of suggests that we’re not into mutual respect as much.
MODERATOR: Okay. Thank you. Daniel?
DANIEL: I kind of think of it as the end result of choices that we make as researchers more than anything else. The choice we make about how we invite someone to participate, how we ask them a question, how we present it, and then response rate is how we measure our success of doing that.
MODERATOR: Mike Debitt, I’m going to get a real good technical cooperation of respondent cooperation right now I bet.
MIKE D: You’re actually going to get completely the opposite. What worries very much about the conversation that we’re having is that it’s very easy to give a very behavioralist and I’m a quant researcher by training straight out of a behavioralist school but this is not the world that we live in anymore. The world we live in is about co-creation, it’s about engagements, it’s about communications and really what we’ve got to do as an industry is to respond to that challenge. The tendency in our industry is always to revert back to the framework of the sixties and seventies and what you’ve got to understand is that the world has changed. Technology is embedded, it’s mobile, social communications are there, all of this stuff is going on. I wouldn’t talk about respondent cooperation; I would talk about cooperating respondents. That to me is the key in the new world. How we engage them, how we take them in and then how we derive valuable information we get from cooperating respondents and worry less about the concept – the behavioral concept of respondent cooperation.
MODERATOR: Thank you Mike, April?
APRIL: I think respondent cooperation is different again depending on the type of respondent or the audience you’re after. Again on the client side in the insurance industry we have a lot of listed samples and as long as we get a good mix of our customers by industry type, by region, by size of employer, we’re kind of putting all the checks and balances in there to make sure we have gone out to a robust sample and exhausted the list with maybe four or five call attempts or invitations and get a good variety of opinions and idle data, that’s my real loose definition of cooperation rate. That’s kind of what we’re after. I know there’s industry standards for response rate and there’s four or five of them and there’s cooperation rates and there’s a couple of those and I know it’s competes and completes over ineligibles combined but in our day to day we don’t really use that.
MODERATOR: Okay, Leslie?
LESLIE: I actually have to agree with Curt’s definition as well because we do a lot of research sending the survey to a mobile device or using camera phones to gather ethnographies. We’re doing some exploratory and unique things and we also want to look at the novelty factor and our people just responding because it’s a novel, new way of gathering information. Or are they really wanting to participate in the research process. One of the ways we look at that is will they do it again or will this be a one time occurrence?
MODERATOR: Okay, Joanne?
JOANNE: As a panel provider, we define response rate simply as our panel as taking a call to action. What I mean by that is the panelist opening the invitation to the survey and then clicking on the link to go to the client’s site to take the survey. From that point on, respondent cooperation is when that panelist completes the process and takes the survey to the end.
MODERATOR: Okay, Carl?
CARL: I think it’s something that has been addressed at every conference I’ve been to over the last four years and we’ve done very little to improve it.
MODERATOR: Have we defined it?
CARL: We’ve defined it but as researchers on the Internet most of us want to blame it on the panelists.
MODERATOR: We’re going to get to that. Michael?
MICHAEL: I tend to over simply some of these things just because of the background that I have. We look at respondent cooperation and we talk a lot about certainly the acronyms and the mixed rates but I think respondent cooperation really does come down to what Curt and Renee touched on which is a trust relationship. I really think it’s the existence of the trust relationship that you have with your community where they feel comfortable sharing relevant opinions over time. I think the over time part – just like within your relationship you don’t do the hit and run and establish a relationship where everyone is benefiting and over time they feel comfortable sharing those opinions.
MODERATOR: We have David Sunkamp and Larry Shack. Do you guys know each other? Do you want to arm wrestle as to who goes first? Okay, David go ahead.
DAVID: Okay so I think that a lot of my colleagues have adequately defined respondent cooperation. But the reason that we’re here today is because we feel that it’s dwindling in terms of respondent’s willingness to participate in research and so there are traditional measures here that everyone has quoted and talked about but I think one of our major issues here is that everyone is regarding that from an insular definition that is just to this industry and part of the problem is that there are forces outside of this industry that really drive respondent’s cooperation so I would broaden the definition to say that respondent cooperation is the willingness of a population to respond to a stimuli and give us information based upon that stimuli because when you look there are advertising messages – there is a purpose there. It’s a call to action to do something. So if we keep this definition just a strict MRI definition than I think we’re never going to solve this issue because we’ve got an issue that is beyond the scope of our industry.
MODERATOR: Larry?
LARRY: I think from my perspective it’s the number of folks that complete a survey based upon the number that are eligible. I think echoing everything we’ve heard here to day that it’s our goal as researchers to optimize that respondent experience and take full advantage of the different needs and methods that are available to do so that taps into their willingness to give feedback that exists today.
MODERATOR: So if I’m the CEO and I’m listening to these definitions my head is spinning. I want to know from the audience does the industry need a universal measurement system that everyone agrees on or no. If you think it does put your hand up. If you think it doesn’t put your hand up. Okay, Peter come on back here first. Why do you think we need one? Do you think we need a universal measurement that everybody here and here agrees with or is that not compatible with what the world we live in?
RESPONDENT: I sincerely believe that we should all have the same standard and yes we should be able to agree to it.
MODERATOR: Is it realistic?
RESPONDENT: It’s realistic, yes. It’s not easy but it’s realistic.
MODERATOR: Who doesn’t think it’s realistic?
RESPONDENT: I think it would be great to have a common definition of response rate and cooperation rate but only insofar as you can tie that to what can you do with information that was collected at a particular response rate or cooperation rate. Is a 1% opt in sample that you use for a survey, is the data you use from that valid for a particular decision? Or do I need a 90% in home survey. You have to tie it to what you’re going to do with the answer otherwise by itself it’s just a number.
MODERATOR: David Slotwinder do we need a different measure for the Internet for cell phone cooperation surveys, for email, for cooperation with blogs for research communities?
RESPONDENT: David’s my brother but I assume you meant me. I think we can apply some of the standard measures from survey research to some degree of success in the online world but to follow up on Michele’s comment it’s only useful to some extent that it tells you something about the data quality itself. What you can use that information for after you’ve collected it. So in the context of panels I think even a within panel response rate is useful because it tells you something about the cooperation of the people that our panel provider is telling you they had on their panel. If the panel provider is telling you they had one million people and there on average getting 20% to respond to a particular survey and another panel provider has one million people and is getting 40% I think it’s impossible – you can’t infer that the data is necessarily better but you can infer that there’s a different panel management operation going on and ask the next questions.
MODERATOR: Curt in the very competitive online panel business, do you measure respondent cooperation and response rates the same way as your competitors? I’m not asking for a definition. Either you feel that you do or you don’t.
CURT: I think our responsibility is to measure it to social science standards. I can’t speak to every competitor or how they define it or measure it. I think our fiduciary responsibility is to our clients and to the social science standards and to ensure that we get the highest level of cooperation to our clients and ultimately if someone is looking at the CPI and an X dollar CPI for that response and we feel like we’re giving a higher quality product to our clients when we can say our response rate is higher than the same CPI that they can get from another panel. So it’s something we’re very focused on and try to maximize that. And we actually wish that more of our clients and their clients focused on and care about because it’s something that we care about.
MODERATOR: Renee Smith?
RENEE: There’s a couple of interesting things that were said by the previous comments. One that response rate might tell you something about panel management practices. And then the other that a higher response rate might have positive benefits. If you have ever been involved in panels, one of the things you can realize is that no matter what mathematical formula you decide is a standard the response rate can be easily manipulated depending upon what you sampled, depending on what your panel management practices are, depending on how often you clean the panel, etc. Without the context of the panel management practices, response rates currently in the industry even if computed according to a social scientific standard are going to vary across panels. And I’ll just follow up and say it’s not necessarily a good thing – for example, suppose you clean out respondents you haven’t heard from in the last six months. Then suppose a client comes to you and says well actually I’m doing a survey about the use of a website or Internet sort of behavior and I need to talk to both frequent and infrequent Internet users. Well you’re not going to really be able to represent the infrequent Internet users if you’re cleaning out people who don’t respond in a very short period of time. So cleaning them out might increase your panel response rate but does it make your panel more representative or valid, not necessarily.
MODERATOR: U. Davis is it more important to come up with a measure of response rate or respondent cooperation or to just try to develop some best practices and just stick to them?
U. DAVIS: I think it’s a yes and yes. I think it’s important that you look at both. But I think the best practices aspect thing as an industry thing would be a lot easier to address because there are so many variables that go into the quality of how you manage your panel and what you define as an active panelist and how you define frequency of contact and so forth which all ultimately impact the cooperation of the respondent. So I think I would start with the best practices area.
MODERATOR: Mike Hess, for the simple sake of projectability do we need a universal standard or some standard that you can define?
MIKE H: I’m glad you asked the question of projectability. I do want to vent a little bit. It seems like everybody focuses on the percent response rate but I’ve never heard the word bias so far. So for example to get into your issue if I have a 10% response rate and its biased because they’re all heavy users or they’re all heavy adopters. I’m not going to feel as good about that assuming I can figure that out. If I have a 1% response rate and its actually random and projectable response rate. So I’m kind of more obsessed personally with the notion of projectability regardless of response rate. I think our working assumption as a panel here is that lower response rates mean bias but I’m not convinced that’s the case. I think that’s my jumping off point in this conversation.
MODERATOR: Jeff Fink, yours is a company where you do custom research and you’ve got tracking research going on. Talk to me about (inaudible) and remember what I consider traditional and what I consider high tech. Dottie if I own this media company and you have one of the more revolutionary tools that you’re actually testing and you’re going to be rolling out which is going to affect my company PPM and that stands for personal people meter, how important is respondent cooperation in putting something like that together?
DOTTIE: Well because we are an accredited or need to be an accredited service it’s of the utmost importance.
MODERATOR: So it’s one of the most important . . .
DOTTIE: It is the most important.
MODERATOR: Okay and you’ve been testing it. You tested it in the U.K and Manchester and you tested it in Philadelphia and you tested it in Houston. What do you find the respondent cooperation rate to be with a device that people are basically going to carry on them or wear on them during the course of the day and then put on some sort of downloading mechanism at night? How’s it compare with a diary which is what you guys have used as your basic business for years and years?
DOTTIE: The standard for a respondent cooperation is somewhat different. We don’t use the traditional one for that panel that we use for the diary service. Umm it’s almost like comparing apples and oranges. But it’s somewhat lower in the traditional diary service. People are being asked to participate for two years on this particular panel.
MODERATOR: It’s less than a diary?
DOTTIE: It’s somewhat less yes.
MODERATOR: Have you done in research as to your target audiences’ assumed willingness to participate and to compare it to diaries and such.
DOTTIE: I’m not quite sure I understand the question.
MODERATOR: Okay when you were comparing the PPM before you ever rolled it out in your R&D what was your presumption about people’s willingness to wear a high tech device that would collect information about what they would hear on the radio and television and to pass it on to you everyday.
DOTTIE: We knew that it was more of a burden than maintaining a one week log. Asking also younger members of the household to participate starting at age six versus our traditional twelve and up was certainly going to have an impact on those calculations and we were prepared with incentives to overcome much of that.
MODERATOR: Has it met your expectations?
DOTTIE: Yes.
MODERATOR: Jay Levy you do polling but you do polling a little bit different than everyone else does. Briefly explain how your polling works?
JAY: We ask TV anchormen to ask the questions over the telephone.
MODERATOR: Local anchor people.
JAY: Yes.
MODERATOR: So here in New York you have specific anchors that serve as voices when a poll is going to be taken?
JAY: You answer your ringing home telephone. You hear the voice of a TV news anchor from your community.
MODERATOR: What was your perception before you rolled it out as to what it was going to do to respondent cooperation?
JAY: I didn’t think about it in an abstract way I thought about it in a very specific way. We had to do human follow up calls and say did you just get a call from a TV news station and then ask them an open ended question and ask what they thought about it. If the response had been negative we stopped.
MODERATOR: But the response was?
JAY: The response was overwhelmingly positive. The reason for that was that people consistently said that I cannot believe that their anchor Bill Curtis, Walter Jacobsen thought enough to call me and ask what I thought about this issue.
MODERATOR: Leslie Townsend your business is basically collecting information for research via cell phones. Talk to me about the experience of respondent cooperation with respect to that.
LESLIE: We work many different kinds of projects. Some surveys for instance we do wireless advertisement testing and there you might just have a link that says click here to take a survey and it’s sitting out there on the wireless web and typically what we will see there – we’re just measuring the percent of people that are exposed to that link that will take the survey and its anywhere form 1% to 3%. So it’s very high compared to a link that’s sitting out there on the web. Probably ten times as high. But again, going back to the novelty factor we don’t know how long that will sustain. In projects like longitudinal diaries where people are asked to track items like impulse purchases through a form on their cell phone for instance each time that they order a cheeseburger or something like that. We’ve done longitudinal studies that have lasted anywhere from ten days to six months. We see the participation rate as being much higher than other paper based or other forms of diaries. And the dropout rate lower. But again, it’s new technology so we’re just hoping this holds up. For surveys where people have opted in to a wireless panel – they’re sent an invitation via SMS; they click on the link we typically get response rates of 80% to 100% but the people who opt in, it’s very difficult to get people over the age of 40 to opt in so we have a biased sample there. You can get a teenager to do anything on a cell phone. For those of you that have teen panels and have difficulty they want to take them again and again and again. We do ethnographies and these kids will do far more than is ever asked of them. They will send us videos, sound recordings, digital images, they will hunt us down and call us and ask us if they can be on more studies. We see the cooperation and response rates in wireless very, very high right now but we don’t think that over the long run this will be sustained. It will eventually have the same issues as the rest of the industry.
MODERATOR: My cell phone is ringing off the hook. My three research directors are calling me so we’re going to take a thirty minute break and I’d like everybody back here promptly at 3:30PM. Thank you very much.
0 Responses to “Transcript CASRO respondent cooperation part 1”