Don't Hang Up Yet:
A Comparison of Online and Telephone Survey Methodologies

by Julie Pokela, Ph.D., Elizabeth Denny, Ph.D., and Ingrid Steblea
of Market Street Research, Inc. and
Frank Melanson of Massachusetts General Hospital

Today, when decision-makers need data, they're increasingly turning to online surveys instead of more traditional telephone surveys. Why? At first glance, the answers seem obvious. Lower costs. An abundance of software on the market that makes it easy for anyone to conduct their own online survey. Quick turnaround times. So why should anyone consider doing things the old-fashioned way?

In March of 2007, Market Street Research, Inc. conducted a study with Massachusetts General Hospital (MGH) to assess the comparability between online and telephone survey methodologies in researching awareness of and attitudes toward hospitals among residents in MGH's market area. The results showed that there were clear discrepancies between the two survey methodologies and that the telephone survey was the most representative of the area residents.

In this article, we will discuss the reasons behind the current popularity of online surveys. We will describe the results of our comparison of methodologies and the arguments in favor of using the telephone methodology for surveying the general public. We will conclude with a brief discussion about the challenges ahead for survey researchers.

The Lure of Online Surveys. Recent industry trends studies are clear: more and more companies are turning to online surveys when they need data. The number of online research studies increased 80% in 2005, with 90% of major corporations, 86% of research companies, and 61% of smaller corporations conducting research online in 2007. It isn't surprising that online surveys have become so popular. First, they are remarkably affordable. Online surveys don't carry the costs of postage and data entry associated with mail surveys, nor the costs of phone service and interviewing staff associated with telephone surveys. Subscriptions for survey software tools can run under $20 a month. One survey software site even offers a free account option that allows users to post an unlimited number of online surveys.

Not only are such tools priced low enough to be affordable for the average consumer, they are designed to be easy enough for the average consumer to conduct. It requires no special training to get a survey online. As one survey software company states, its goal is to "enable anyone to create professional online surveys quickly and easily." Companies advertising these online tools claim-and rightly so-that their software packages effortlessly handle skip patterns, feed-ins, conditional questions, and other complexities of questionnaire design. Online surveys also solve other headaches associated with traditional research methodologies: there are no interviewers to train and supervise, no issues arising from calling across time zones, no worry that the call will interrupt consumers during the dinner hour. Some companies enable their customers to track responses and automatically create reports with charts and graphs.

Finally, once the survey is online, it is possible to get a large volume of responses very quickly. A survey that would take weeks to complete with telephone interviews can be completed in a few days online.

But What About The Data? Given these advantages, it's easy wonder why anyone would bother with a telephone survey. To answer that question, Market Street Research, Inc. teamed with Massachusetts General Hospital (MGH) in March 2007 to conduct a survey of area residents' awareness and image of hospitals, employing both telephone and online methodologies concurrently. One of the goals of this research was to determine the comparability between the two survey methodologies. Would the data from the two methodologies be similar enough to justify using the cheaper, faster online survey in the future?

Comparing The Two Methodologies. We began the study by designing the questionnaire. Identical questionnaires were used for the telephone and online surveys to ensure maximum comparability. The main topic areas of the questionnaire included: (1) residents' awareness of area hospitals; (2) the hospitals residents believe has the best reputation overall; (3) the hospitals residents believe have the best reputations for specific types of care, such as cardiology, oncology, and GI disorders; (4) the hospitals residents would prefer to use if they required care for a serious medical problem; and (5) demographic questions.

For the telephone survey, we obtained a list of telephone numbers of area residents from Survey Sampling International, a vendor of telephone samples headquartered in Fairfield, Connecticut. The phone numbers were generated using a Random-Digit Dialing (RDD) process.

For the online survey, respondents were selected from online panels compiled by e-Rewards Market Research. An online panel is a list of volunteer respondents that is highly normalized in order to correlate closely with the demographics of the U.S. population as a whole, and extensively profiled to include geographic, psychographic, life cycle/circumstance, and future purchase intent data. Online panels are currently a major source for sample for online surveys of the general population, with some researchers claiming that nearly 90% of online studies conducted in the U.S. in 2006 utilized such panels.

e-Rewards online panels are recruited by email invitation. e-Rewards partners with companies like Borders, Sony, and Continental Airlines, using their email lists of customers who elected to receive offers from these companies and their partners. By definition, consumers must have an email address in order to be recruited for an e-Rewards panel. Consumers are issued an email invitation to participate in e-Rewards, for which consumers consent to participate in survey research in exchange for compensation. For every survey a consumer participates in, he or she receives reward currency that accumulate and can be redeemed for gifts offered by e-Rewards' partners. Panel members may be able to redeem their reward currency, for example, for a discount on a Sony Camcorder.

For both the online and telephone surveys, we screened each household to ensure that residents lived in the specified county, that they were between the ages of 30 and 70. In all households, we conducted the survey with an adult involved in making decisions about health care for themselves and/or for others in the household.

In terms of conducting the surveys, Research Data Design, Inc. conducted the telephone interviews from their call center in Fort Collins, Colorado. Interviews were conducted between March 1 and March 18, 2008. The response rate for the survey was 35.4%.

The online surveys were conducted using Vovici's (formerly Websurveyor) online survey tools. Surveys were completed between March 7 and March 13, 2008. The response rate for the online survey was 12.2%. We surveyed a total of 370 residents by telephone and a total of 370 residents online.

The two studies were designed to be as similar as possible. In addition to using identical questionnaires, we surveyed the same geographic area (eastern Massachusetts and southern New Hampshire), at the same time of year (early March), and completed interviews with the same number of respondents (370 using each methodology). The telephone survey took longer than the online survey to complete, with data collection lasting eighteen days for the telephone survey, compared with six days for the online survey. As expected, the data collection costs for the telephone survey were about three times that of the online survey.

Table 1: Methodological Comparison

Demographic Differences. Our comparison of the two methodologies yielded several key findings, the first of which was the significant difference in the demographic characteristics of the two survey populations. Table 2 provides comparisons between the two survey populations and the actual characteristics of residents in the target geographic area. As can be seen here, the demographic characteristics of telephone and online survey respondents are similar in a few respects. The proportions of female and white respondents are similar in the two methodologies. Compared with telephone survey respondents, however, residents who completed the survey online are younger, better educated, more affluent, more likely to be employed, and are more likely to have private health insurance.

While the demographic characteristics of our online survey respondents might not have been closely aligned with the actual characteristics of residents in the geographic area, they appear to be fairly closely aligned with that of another population: not surprisingly, that of Internet users. Studies have found that compared with the general population, Internet users have higher levels of education and are more affluent.

As can also be seen on this table, the demographic characteristics of respondents to both survey methodologies differ in some respects from the actual characteristics of residents in the targeted geographic area. To some extent, these differences are related to the fact that we interviewed the primary health care decision-maker in each household. In most households, women make most of these decisions; as would be expected, therefore, a majority of survey respondents are female.

Table 2: Demographic Comparison of Unweighted Data

We weighted the data from the telephone and online surveys by age and education. Weighting is a common statistical technique used to adjust survey results so that groups are represented to the same extent in the survey as they are in the general population. Typical parameters for weighting are age and education, because people of different ages and educational levels differ in their likelihood of answering the telephone when interviewers call them. As can be seen above, as with most telephone surveys, telephone survey respondents skew older than the general population, as older residents are generally more likely to answer the phone and agree to participate in a telephone interview.

We were limited, however, in our ability to weight the data based upon education. Only 19 respondents to our online survey had a high school or less education, representing about 5% of our online survey sample. The actual proportion of residents in the targeted geographic area without at least some college education is closer to 40%. Clearly, weighting 5% of the online survey results to represent 40% of a population is not a good practice, as this would gravely distort the findings. Because of this, we needed to combine respondents with a high school or less education with those with some college education to weight the data based on education.

Weighting the results from both methodologies so that the data better reflected the age and educational levels of residents in the surveyed areas would ideally have provided us with online survey demographic characteristics that closely approximates those from the telephone survey. Unfortunately, as we discovered, this was not the case.

Weighting eliminated the significant differences between the two survey populations in terms of employment and age. Even after weighting, however, other significant differences remained between the two survey populations, with online survey respondents better educated, more affluent, and more likely to have private health insurance-a critical difference in a health care study.

The table below shows the shift in demographic characteristics after weighting the data by age and education. As can be seen here, the weighted data for telephone survey respondents now closely matches the actual characteristics of residents in the targeted geographic area, while online survey respondents still appear to be much better educated and more affluent. While it is theoretically possible to further weight the data based upon the full range of demographic characteristics, in practice, that can result in additional distortions because of the problem of applying large weights to small numbers of respondents, as we described in the case of education.

Table 3: Demographic Comparison of Weighted Data

Attitudinal differences. The most critical discovery we made in our comparison of the two methodologies, however, was that even after the data were weighted, people who responded to the online survey expressed significantly different attitudes about hospitals, compared with people who responded to the telephone survey.

Online survey respondents tended to have attitudes and preferences about area hospitals that are more common among younger, better educated, more affluent residents-exactly how the online survey population was skewed. Furthermore, online survey respondents are likely to be more technologically savvy. e-Rewards, one of the industry's most popular sources for online panels and the source used for the panel in this research, partners with airlines and uses frequent flier lists as a typical source for recruiting participants. This creates an added dimension to the unique type of respondent companies are likely to hear from using online survey panels: not only are these respondents likely to be younger, better educated, and more affluent than the general population, but they are also more likely to be "road warriors."

The most striking differences between the telephone and online survey respondents lies in their attitudes toward community hospitals vs. academic medical centers, with online survey respondents far more strongly oriented toward academic medical centers. Specifically, compared with telephone respondents:

• When asked to name the first hospitals that came to mind when thinking about New England hospitals, online survey respondents were significantly more likely to name academic medical centers, such as Massachusetts General Hospital, Brigham and Women's Hospital, and Beth Israel Deaconess Medical Center.


• Online survey respondents were more likely to have heard of specific academic medical centers outside of New England, including Johns Hopkins and Memorial Sloan Kettering.


• When asked what New England hospital has the best reputation, online survey respondents were more likely to name academic medical centers, while telephone survey respondents were less likely to be able to name any hospital.


• When asked what New England hospital had the best reputation for seven different specialty areas, online survey respondents were more likely to identify academic medical centers for every single type of care. By contrast, telephone survey respondents more frequently identified community hospitals or were unable to name a specific hospital they thought was best.


• When asked what New England hospital they would prefer to use if they needed to be hospitalized for a serious or complicated medical problem, online survey respondents were significantly more likely to identify academic medical centers. Telephone survey respondents, on the other hand, were significantly more likely to say they would use community hospitals for care.

Clearly, if we had relied on the weighted online survey results alone, we would have dramatically overstated the market position of academic medical centers in eastern Massachusetts and southern New Hampshire.

Four Problems With Online Surveys. The most important issue for organizations to understand when choosing a research methodology is that online surveys skew results in favor of Internet users, which at this point in time isn't everyone. Use of the Internet is evolving rapidly, but as of mid-2007, about 29% of U.S. adults either did not have computers or they have access to computers but don't use them for email or the Internet. A significant proportion of the population can't be reached via online surveys.

Second, even among people who do use the Internet, no method exists for selecting these individuals randomly for survey purposes. There is no universal list of email addresses that researchers can use to obtain sample for online surveys. This problem has been solved for telephone surveys through random-digit dialing (RDD), but as one researcher put it succinctly, "there is no analogue of RDD for e-mail addresses." As a result, researchers are forced to use consumer panels, comprised of individuals who are not just Internet users, but a particular breed of Internet user that clicks on banner headings, has elected to receive direct marketing email, or has agreed to be a regular survey subject.

At the time of this writing, there is only one organization, Knowledge Networks, that offers an online survey panel that claims to be representative of the general population, having recruited participants by telephone using a random digit dialing technique, and equipping households without Internet access with the necessary hardware and Internet access to join the panel. The vast majority of companies offering online survey panels have not used an RDD recruitment methodology, but have recruited consumers via email off of lists used for direct marketing.

Third, the legitimacy of online panelist responses must be carefully considered. In a telephone survey, respondents are rarely offered compensation for their time, beyond the knowledge that their opinions will be used to improve, for example, health care delivery in their area. Trained telephone interviewers may be able to identify respondents who do not meet the survey criteria. For example, in a survey of women under the age of 30, a telephone interviewer is likely to have some clues if a potential respondent is actually a male over the age 50. Interviewers are trained to identify respondents who are not providing serious answers and to elicit opinions from taciturn respondents. Finally, the telephone interview format itself prevents respondents from rushing through the survey.

None of these legitimacy measures exist for online survey, however. There is enormous industry pressure to attract online survey panelists, especially those who persevere through a questionnaire even if the topic doesn't interest them. The going rate for joining a paid online panel ranges from around $5 to $50 per survey, to daily or monthly sweepstakes. There is often an element of "snowball sampling" as well, with some companies paying people not only for their own participation but for referring friends or family members as well. Some websites actually specialize in directing people to panels that pay the highest rates. As one such site explains, "Here we review the best online paid survey sites that pay you for your opinions. Our database is frequently updated and is absolutely FREE."

This environment increases the risk of obtaining completed surveys from "professional respondents," individuals who participate frequently in studies and respond in a very different manner than conventional respondents-at best providing responses that are not seriously considered, and at worst providing responses that do not reflect their own views or have been provided in an effort to manipulate the outcome of the research.

Finally, the single most important problem with online surveys is the extent to which meaningful, actionable conclusions can be drawn from the data obtained using this methodology. As we have described above, online surveys use convenience samples. With a scientific probability sample we could calculate a margin of error, which we would then be able to use to know the extent we can trust a survey's findings. We could also weight the data appropriately so that it reflects the general population. None of this is possible with online surveys, which are fundamentally not scientific samples.

Online surveys today are akin to the infamous straw polls conducted by the Literary Digest in 1936 that confidently predicted a presidential win for Republican Alf Landon over Democrat Franklin Roosevelt. The Literary Digest's erroneous predictions were based on a sample size of more than two million people, but because the sample was drawn from lists of people who owned cars or telephones, the findings significantly distorted public opinion:

"The failure of the Literary Digest straw poll is usually attributed to bad sampling techniques. The sample itself was self-selected. Some people who received questionnaires responded, but most did not. In addition, the sampling 'frame' was biased; the mailings went to people who had 1936 auto registrations and who were listed in telephone books. In 1936, millions of Roosevelt supporters were too poor to have phones or own automobiles. So, sampling only people with cars or phones did not produce a cross section of likely voters."

The Gallup Poll, on the other hand, now predicts election results with consistent accuracy using samples of 1,500 or fewer voters. Gallup achieves this level of confidence because it polls the nation by telephone, using scientific sampling methods.

The Future of Survey Research. Survey research is clearly facing some fundamental shifts. Right now, U.S. households' access to telephones is nearly universal (about 97% as of 2000). Telephone surveys are expensive to conduct, however, and there are social and technological changes that are making it harder to conduct good quality telephone surveys. First, people are increasingly unwilling to cooperate over the telephone, in part because they are called regularly by telemarketers posing as researchers, or they heed media and law enforcement warnings not to give out personal information to avoid identity theft.

Secondly, households' use of telephones is changing rapidly-the percentage of U.S. adults who live in households with only wireless telephone service (no land lines) has been steadily increasing since 2003, and has reached nearly 14%, or one out of every seven homes. More and more households have two or more telephone lines, which further complicates sampling.

Technology is changing how people and organizations communicate with each other. Internet use is changing as rapidly as telephone use, as more and more U.S. households obtain Internet access. It is quite reasonable to expect that at some point in the future, U.S. households' access to the Internet will be as universal as telephone access is now. As the industry matures, researchers will undoubtedly discover methods for surveying people online that can produce broadly accurate results.

There are definitely situations in which online surveys are an excellent, cost-effective methodology. Online surveys would be a very good choice for an organization that wants to survey a captive audience, for which the entire population has a known email address, such as a company wanting to conduct employee satisfaction surveys. An online survey would also be a good choice for surveying certain rare populations, for which a dual-frame, web-telephone methodology may be the only effective means of finding eligible respondents.

Online surveys are also a good methodology when the questionnaire includes a multi-media component or a large number of open-ended questions. Here, online surveys have an advantage over telephone surveys (where responses may not being captured in full depending on the limitations of the technology employed or the interviewer's typing speed) as well as mail surveys (which have been shown to elicit much briefer open-ended responses).

In order for online surveys to become a reliable method for general population surveys, however, it will require a significant change in the ways email addresses are generated, comparable to the changes the U.S. telephone system underwent in the last two centuries. In 1910, reaching a person beyond the limits of one's city was almost the same as reaching someone whose email address isn't immediately known. It meant telling the operator which city you wanted, then the person's name, and then, if you were lucky, a short alphanumeric sequence associated with the person's building or residence. Our current system of telephone numbers-seven digits conveniently tied to geography-wasn't fully established nationwide until the late 1950s. If the Internet emerges as people's predominant means of communication (and if no new technology is developed that people think is better), a similar change in the ways people's email addresses are assigned could make the web a formidable resource for surveys.

Until that time comes, however, organizations needing research that will be representative of the general population are well advised to consider the problems endemic to online surveys. The people who respond to online surveys simply do not reflect the characteristics of the larger population: they tend to reflect the population of Internet users, which at this point means a better-educated, more affluent consumer. Surveys conducted exclusively through online surveys are very likely to provide skewed data and provide companies with a false sense of consumers' needs and how they are perceived by consumers. Traditional telephone surveys are still the best methodology for achieving a reliable understanding of the beliefs, opinions, and preferences of the general population.