ERIC Identifier: ED410233
Publication Date: 1996-11-00
Author: Frary, Robert B.
Source: ERIC Clearinghouse on
Assessment and Evaluation Washington DC.
Hints for Designing Effective Questionnaires. ERIC/AE Digest.
The purpose of this digest is to offer tips in designing quality
questionnaires and on avoiding common errors. Some of the more prevalent
problems in questionnaire development are identified and suggestions of ways to
avoid them are offered.
Do keep the questionnaire brief and concise. Some questionnaires give the
impression that their authors tried to think of every conceivable question that
might be asked with respect to the general topic of concern. The result is a
very long questionnaire causing annoyance and frustration on the part of the
respondents resulting in non-return of mailed questionnaires and incomplete or
inaccurate responses on questionnaires administered directly. To avoid this
first potential problem the investigator must define precisely the information
desired and endeavor to write as few questions as possible to obtain it.
Peripheral questions and ones to find out "something that might just be nice to
know" must be avoided. A clear-cut need for every question should be
Do get feedback on your initial list of questions. Feedback may be obtained
from a small but representative sample of potential responders. A field trial of
a tentative form of the questionnaire is also desirable.
Do locate personal or confidential questions at the end of the questionnaire.
The early appearance of unsettling questions may result in respondents
discontinuing the questionnaire.
Do order categories. When response categories represent a progression between
a lower level of response and a higher one, it is usually better to list them
from the lower level to the higher in left-to-right order, for example,
1) Never 2) Seldom 3) Occasionally 4) Frequently
Do consider combining categories. In contrast to the options listed just
above, consider the following:
1) Seldom or never 2) Occasionally 3) Frequently
Combining "seldom" with "never" might be desirable if responders would be
very unlikely to mark "never" and if "seldom" would connote an almost equivalent
level of activity, for example, in response to the question, "How often do you
tell you wife that you love her?" In contrast, suppose the question were, "How
often do you drink alcoholic beverages?" Then the investigator might indeed wish
to distinguish those who never drink. When a variety of questions use the same
response scale, it is usually undesirable to combine categories.
Do ask responders to rate both positive and negative stimuli. There is
sometimes a difficulty when responders are asked to rate items for which the
general level of approval is high (the "apple pie" problem). There is a tendency
for responders to mark every item at the same end of the scale. By offering
positive and negative responses the respondent is required to evaluate each
response rather than uniformly agreeing or disagreeing to all of the responses.
Do choose appropriate response category language and logic. The extent to
which responders agree with a statement can be assessed adequately in many cases
by the options:
1) Agree 2) Disagree
However, when many responders have opinions that are not very strong or
well-formed, the following options may serve better:
1) Agree 2) Tend to agree 3) Tend to disagree
These options have the advantage of allowing the expression of some
uncertainty. In contrast, the following options would be undesirable in most
1) Strongly agree 2) Agree 3) Disagree
4) Strongly Disagree
Some would say that "Strongly agree" is redundant or at best a colloquialism.
In addition, there is no comfortable resting place for those with some
Avoid open-ended questions. In most cases open-ended questions should be
avoided due to variation in willingness and ability to respond in writing.
Avoid the response option "other." Careless responders will overlook the
option they should have designated and conveniently mark the option "other" or
will be hairsplitters and will reject an option for some trivial reason. An
exception to the foregoing advice is any case in which the categories are
clear-cut, few in number, and such that some responders might feel uncomfortable
in the absence of an applicable response.
Avoid category proliferation. A typical question is the following:
Marital status: 1) Single (never married) 2) Married
3) Widowed 4) Divorced 5) Separated
Unless the research in question were deeply concerned with conjugal
relationships, the distinctions among all of these categories are not useful.
Usually, such a question reflects the need to distinguish between a conventional
familial setting and anything else. If so, the question could be:
Marital status: 1) Married and living with spouse
Avoid scale point proliferation. In contrast to category proliferation, which
seems usually to arise somewhat naturally, scale point proliferation takes some
thought and effort. An example is:
1) Never 2) Rarely 3) Occasionally 4) Fairly often
5) Often 6) Very often 7) Almost always 8) Always
Such stimuli run the risk of annoying or confusing the responder with
hairsplitting differences between the response levels. Psychometric research has
shown that most subjects cannot reliably distinguish more than six or seven
levels of response. Offering four to five scale points is usually quite
sufficient to stimulate a reasonably reliable indication of response direction.
Avoid responses at the scale mid-point and neutral responses. The use of
neutral response positions had a basis in the past when crude computational
methods were unable to cope with missing data. In such cases, non-responses were
actually replaced with neutral response values to avoid this problem. The need
for such a makeshift solution has long been supplanted by improved computational
methods. Consider the following questionnaire item:
The instructor grades fairly.
1) Agree 4) Tend to disagree
2) Tend to agree 5) Disagree
There is no assurance whatsoever that a subject choosing the middle scale
position harbors a neutral opinion. A subject's choice of the scale midpoint may
result from: Ignorance, Uncooperativeness, Reading difficulty, Reluctance to
answer, or Inapplicability.
In all the above cases, the investigator's best hope is that the subject will
not respond at all. Unfortunately, the seemingly innocuous middle position
counts, and, when a number of subjects choose it for invalid reasons, the
average response level is raised or lowered erroneously (unless, of course, the
mean of the valid responses is exactly at the scale midpoint).
In the absence of a neutral position, responders sometimes tend to resist
making a choice in one direction or the other. Under this circumstance, the
following strategies may alleviate the problem:
1)Encourage omission of a response when a decision cannot be reached.
2)Word responses so that a firm stand may be avoided, e.g., "tend to
3) If possible, help responders with reading or interpretation problems, but
take care to do so impartially and carefully document the procedure so that it
may be inspected for possible introduction of bias.
4) Include options explaining inability to respond, such as "not applicable,"
"no basis for judgment," "prefer not to answer."
The preceding discussion notwithstanding, there are some items that virtually
require a neutral position. Examples are:
How much time do you spend on this job now?
1) Less than before 2) About the same 3) More time
The amount of homework for this course was
1) too little 2) reasonable 3) too great
It would be unrealistic to expect a responder to judge a generally comparable
or satisfactory situation as being on one side or another of the scale midpoint.
Avoid asking responders to rank responses. Responders cannot be reasonably
expected to rank more than about six things at a time, and many of them
misinterpret directions or make mistakes in responding. To help alleviate this
latter problem, ranking questions may be framed as follows:
Following are three colors for office walls:
1) Beige 2) Ivory 3) Light green
Which color do you like best? _____
Which color do you like second best? _____
Which color do you like least? _____
By carefully evaluating the need of every question used in an instrument and
carefully wording the responses, you will collect information which will yield
more satisfactory and meaningful results.
Dillman, D. A. (1978). Mail and
telephone surveys: The total design method. New York: John Wiley.
Frary, R.B. (1996) Brief Guide to Questionnaires Development. Washington, DC:
ERIC Clearinghouse on Assessment and Evaluation (30 pages).
Grunlund, N.E. (1993) How to make achievement tests and assessments. Needham
Heights, MA: Allyn and Bacon.
Hinkle, D. E., Oliver, J. D., & Hinkle, C. A. (1985). How large should
the sample be? Part II--the one-sample case. Educational and Psychological
Measurement, 45, 271-280.