ERIC Identifier: ED458291
Publication Date: 2001-12-00
Author: Solomon, David J.
Source: ERIC Clearinghouse on
Assessment and Evaluation College Park MD.
Conducting Web-Based Surveys. ERIC Digest.
The growth of the Internet has impacted on virtually every aspect of society.
Survey research is no exception. Two years ago in an informal search of Yahoo,
Kay and Johnson (1999) identified over 2,000 Web-based surveys(1) in 59 areas.
The interest in Web-based surveying is not surprising as it offers a number of
distinct advantages over more traditional mail and phone techniques. Examples
include reducing the time and cost of conducting a survey and avoiding the often
error prone and tedious task of data entry (Medin, Roy & Ann, 1999).
Email offers one option for distributing Internet surveys. Up until a few
years ago email surveys were the predominate means of Internet surveying. As the
World Wide Web (WWW) has grown in popularity, the use of Hypertext Markup
Language (HTML) forms or Web-based surveys are becoming the dominant method of
gathering survey data. These forms streamline the data collection process
formatting and entering responses directly into a database for analysis. Since
HTML forms can be made programmable, it is also possible to have real time error
checking and correction increasing the accuracy of the data collection process.
The formatting capabilities of HTML allow the creation of easy-to-read and a
attractive forms that may improve response rates. In addition, the
programmability of HTML forms makes it possible to randomly order responses and
tailor options based on information the respondent supplies earlier in the
Combining an email "cover letter" as a means of contacting sampled people
with the use of an HTML form for data collection provides an especially
effective and efficient approach to Internet surveying. Modern email packages
automatically convert universal resource locators (URLs) or web-addresses in the
text of an email into a hyperlinks. Placing the URL of the survey form in a
cover letter email allows the respondent to "click" their mouse on the URL to
display the survey form and subsequently fill it out.
CONCERNS WITH WEB-BASED SURVEYING
surveying is very attractive, at this point it should be used with caution.
Currently the biggest concern in Internet surveying is coverage bias or bias due
to sampled people not having or choosing not to access the Internet (Kay &
Johnson, 1999; Crawford, Couper & Lamias, 2001). Despite expediential growth
of the Internet there are still large numbers of people who do not have access
and/or choose not to use the Internet. It is also clear that there are wide
disparities in Internet access among ethnic and socioeconomic groups (Selwyn
& Robson, 1998).
There are specific populations where Internet access is extremely high and
coverage bias is likely to be less of a concern. College students and university
faculty within the USA, Canada and Western Europe are examples of such
populations. Even though coverage bias may be less of an issue in these groups,
experience and comfort with Internet-based tools such as Web browsers is another
serious potential source of bias both in response rates and the way people
respond to the survey (Dillman, Tortora & Bowker, 2001).
Web-based surveying is still in the early stages of development. The WWW is a
unique media and it is not clear to what extent the knowledge we have gained
over years of experience with more traditional surveying techniques fully
applies to Internet surveying (Dilman, Tortora & Bowker, 2001). Studies are
just beginning to done to learn the optimal ways to structure and format
Internet surveys to limit biases and increase response rates. It is also likely
that the best way to design an Internet survey depends in part on the
familiarity and comfort of the respondents in using Web browsers and email
clients. It is also quite likely that the type of Internet connection as well as
the hardware and software used in accessing the Internet will impact on response
rates and possibly how a person responds to an Internet-based survey.
The use of HTML forms for surveying poses a unique set of issues and
challenges that need to be addressed to ensure valid data. The Web is a very
public place and unless steps are taken to limit access to a survey, it may be
found and responded to by people who are not among those sampled by the
researcher. This can either happen by accident or maliciously. Since one only
has to "click" their mouse pointer on the "submit" button to respond to a
Web-based survey instrument once it is filled out, it is also quite possible for
respondents to either mistakenly or purposefully submit multiple copies of their
While Internet-based surveying techniques need to be used with caution, their
benefits warrant continued exploration and the cautious use. It is also pretty
clear that coverage bias and familiarity with Internet tools will be less and
less of an issue over time. Additionally our knowledge about how best to conduct
Internet surveys will continue to improve with research and experience.
RESEARCH ON INTERNET-BASED SURVEYING
Although the research
on Internet-based surveying is limited, findings are beginning to appear in the
literature. Several studies have found that response rates for Internet surveys
are lower that equivalent mail surveys (Medin, Roy & Ann, 1999; Cooper,
Blair & Triplett, 1999). As noted by Crawford and colleagues (2001), this
may be due to our lack of knowledge on how to achieve high response rates using
the Internet surveys. The lower response rates for internet surveys may also
reflect coverage bias, the lack of familiarity with the media and/or lack of
convenient access to the Internet. In the author's experience, Web congestion
can also be a factor in lowering response rates for Web surveys particularly
with people who have relatively little experience with the Internet.Cook and
colleagues (2000) conducted a meta-analysis of factors influencing response
rates in Internet-based surveys2. They found that follow-up contacts with
non-respondents, personalized contacts, and contacting sampled people prior to
sending out the survey were the three dominant factors in higher response rates.
Kittleson (1997) in a study of email-based surveying found it was possible to
double the response rate with follow-up memos though in general this may be
somewhat optimistic. As with mailed surveys, repeated follow-ups have
diminishing returns and at some point risk irritating potential respondents
without noticeably increasing response rates. Additionally, Dillman, Tortora,
Conrad & Bowker (2001) found that relatively plain Web surveys that load
quickly resulted in higher response rates than "fancier" surveys that take
longer to load. Jeavons (1998) analyzed detailed server logs from three separate
large-scale surveys. He found a relatively high percentage of potential
respondents stopped completing the surveys 1) when encountering the first
question, 2) when encountering a complex question grid, and 3) when asked to
supply their email address. This suggests that some potential respondents have
difficulty with the media and give up early in the process of completing the
survey or when encountering complex questions. Others may be reluctant to give
out personal information such as an email address. The logs were also merged
with demographic data collected via the surveys. Somewhat surprisingly no
patterns in failure to complete rates were found by gender, age or education
level. In two of the surveys, people with lower income were found to have a
higher rate of repeating screens of questions mainly due to improperly filling
DEVELOPING WEB SURVEYS
As noted, due to their inherent
advantages, most Internet surveying is now being done using HTML forms with
potential respondents often contacted via email cover letters. While some
developers still directly code these forms in HTML, there are dozens of HTML
editors available, and they are becoming increasingly sophisticated and easy to
use. There are two general methods of capturing the data entered into an HTML
form. The form can be programmed to email the data back to a specified email
address or captured by a program on the server called a common gateway interface
(CGI) script. Using CGI scripts is more robust, offers more flexibility and is
the far more commonly used method of capturing data. There are several HTML
development packages that both provide HTML editing capabilities and automate
the process of developing the CGI scripts necessary to capture data from HTML
forms developed with the package. Two widely used examples of these packages are
Microsoft's FrontPage and Macromedia's ColdFusion. While these packages are
general purpose Web development tools, there are also a growing number of
software development systems designed specifically for Web-based surveying.
Examples include Perseus's Survey Solutions for the Web, Creative Research
System's The Survey System, and Survey Said Survey Software. These packages tend
to offer additional features specific to survey research. Examples include as
managing the distribution of email cover letters, built-in statistical analysis
and reporting capabilities, and automatic tracking of people who have responded
coupled with the ability of sending out follow-up email reminders to those who
have yet to respond. Their HTML editors are also geared for survey form
development, allowing them to simplify and streamline the process of developing
and formatting the question response fields. Web Survey Mailer System
The author has developed a set of software tools that provides many of the
complex Web survey administration functions included in Web surveying
packages(3). The software, Web Survey Mailer System (WSMS), is an integrated
survey administration system that will send out personalized email cover
letters, track which of the sampled people have completed the survey while
keeping their responses anonymous and send out subsequent reminder emails to
only those sampled people who have not responded to the survey. WSMS will block
people who have not been sampled from accessing and responding to the survey and
will keep respondents from submitting more than one set of survey responses. The
system includes a customizable CGI script to capture the survey responses and
place them in a tab-delimited ASCII database format that can easily be
downloaded from the server and imported into a standard PC data base or
statistical package. WSMS is written in PHP and uses the MySQL relational
database to store information on the sampled people. Both PHP and MySQL are
stable and powerful "open source" packages widely available on university and
commercial Web servers and can be obtained free of charge in a variety of
versions that will run on most common server operating systems. The WSMS scripts
and documentation are available free-of-charge and can be downloaded from
Internet surveys are clearly going to continue to
grow in popularity as the problems of coverage bias and unfamiliarity with the
Internet subside. For the foreseeable future there will be people who will lack
Internet access either by choice or circumstance though this will be less and
less of an issue. Additionally the tools for conducting Web-based surveys will
continue to grow in sophistication and ease of use as will our knowledge on how
best to employ this survey methodology. At present researchers should use this
technique with caution in carefully chosen populations and with an eye to
learning as much as possible about how to do it better.
1.In this paper we use the term "Internet survey" for
both email and HTML form-based surveying while the term "Web-based survey" is
reserved for HTML form-based surveys.
2.Cook, Heath, & Thompson (2000) included studies of both Web- and
3.Detailed documentation for the Web Survey Mailer System is provided.
However, installing and using these tools requires a good working knowledge of
HTML and some background and understanding of server-based programming.
4.This Digest is based on an article first appearing in Practical Assessment
Research and Evaluation
Best, S.,J., Krueger, B., Hubbard, C. & Smith, A. (2001) An assessment of the generalizability of internet surveys.
Social Science Computer Review, 19, 131-145.
Cook, C, Heath, F, & Thompson, R. (2000) A meta-analysis of response
rates in web or Internet-based surveys. Educational and Psychological
Measurement, 60, 821-836.
Couper, M.P., Blair, J. & Triplett T (1999) A comparison of mail and
e-mail for a survey of employees in federal statistical agencies. Journal of
Official Statistics, 15, 39-56.
Crawford, S.D., Couper, M.P.& Lamias, M.J.. (2001) Web Surveys:
Perception of burden. Social Science Computer Review, 19,146-162.
Dillman, D.A., Tortora, R.D. & Bowker, D. Principles for constructing web
surveys. Working paper available from
http://survey.sesrc.wsu.edu/dillman/papers.htm [Accessed 6/01]
Dillman, D.A, Tortora, R.D, Conrad, J. & Bowker D. Influence of plan vs.
fancy design on response rates of Web surveys. Working paper available from
http://survey.sesrc.wsu.edu/dillman/papers.htm [Accessed 6/01]
Jeavons A. Ethology and the Web: Observing respondent behavior in Web
surveys. Proceedings of the Worldwide Internet Conference, Amsterdam: ESOMAR,
1998, available from http://w3.one.net/~andrewje/ethology.html [Accessed 7/01]
Kaye B.K. & Johnson T.J. (1999) .Research Methodology: Taming the Cyber
Frontier. Social Science Computer Review, 17, 323-337.
Kittleson, M. (1997) Determining effective follow-up of e-mail surveys.
American Journal of Health Behavior. 21, 193-196.
Medin, C., Roy, S. & Ann, T. (1999) World Wide Web versus mail surveys: A
comparison and report.. Paper presentation at ANZMAC99 Conference, Marketing in
the Third Millennium, Sydney, Australia, available from
http://www.anzmac99.unsw.edu.au/anzmacfiles/papers.htm [Accessed 6/01]
Selwyn, N., Robson, K. (1998) Using e-mail as a research tool, Social
Research Update, available from http://www.soc.surrey.ac.uk/sru/SRU21.html