Scientifically Based Research: What Does It Mean
for Counselors? ERIC Digest.
by Walz, Garry R. - Jeanne C. Bleuer
A core component of many new educational programs funded under Bush
administration's No Child Be Left Behind Act of 2001 is that they be based
on scientifically based research. Further emphasizing the important role
that this type of research will play in future U.S. Department of Education
priorities has been the establishment of the What Works Clearinghouse.
Under the leadership of Dr. Grover Whitehurst, this new clearinghouse will
"...establish standards for research and then determine...which studies
meet those standards" (Taub, 2002).
Like others in the educational community, counselors and counseling
researchers welcome the opportunity to empirically test the validity of
their practices and then, just as important, demonstrate that what they
do makes a positive difference in students' lives. But they also have many
concerns about the practical feasibility of implementing scientifically
based studies. The purpose of this Digest is to provide an overview of
the characteristics of scientifically based research, compare it to other
types of research, and address some of the challenges and issues that are
particularly relevant to conducting scientifically based studies of counseling
interventions. To accomplish this, the authors have included selected portions
of the ERIC/CASS publication, Research in Counseling & Therapy (Loesch
& Vacc, 1997).
WHAT IS SCIENTIFICALLY BASED RESEARCH?
In a New York Times article (November 10, 2002), James Taub pointed
out that "Journalists using the most exacting method available to social
science - that is, counting - have determined that the phrase 'scientifically
based research' occurs more than 100 times in the No Child Left Behind
Act of 2001." Other terms that are being used more frequently now by educators
and policy makers include empirical evidence, empirical research, evidence
based education and randomized control trials (or RCTs). Whatever term
is used, the intent is to refer to experimental research studies that employ
two fundamental procedures - the use of control groups and the random assignment
of subjects to the different treatment groups. Typically, experimental
research is undertaken when the research question involves a question of
causality, i.e. whether changes in one variable cause changes in another
variable (Hadley & Mitchell, 1995). It is precisely this type of research,
which will provide evidence of what works (or what works better).
CHARACTERISTICS OF EXPERIMENTAL RESEARCH METHODS
Isaac and Michael (1981) identified the purpose of experimental research
as investigating "possible cause-and-effect relationships by exposing one
or more experimental groups to one or more treatment conditions and comparing
the results to one or more control groups not receiving the treatment"
(p.52). They also identified seven characteristics of experimental research
implied in their definition: (a) management of the predictor and criterion
variables along with the conditions in which the investigation is conducted;
(b) use of a control group; (c) attempting to control variance among the
predictor and criterion variables: (d) internal validity; (e) external
validity; (f) ability to manage multiple predictor, criterion, and extraneous
variables; and (g) exercise of control which makes experimental research
powerful (but also somewhat artificial) when applied to human subjects.
PREDICTOR AND CRITERION VARIABLES
There are two primary types of variables in experimental research (Hadley
& Mitchell, 1995). The independent (aka treatment) variable is manipulated,
managed, or administered by the researcher. The result of the manipulation
is the measured or observed change in the dependent variable. While the
terms independent and dependent variables have been used traditionally,
the terms predictor and criterion variables are better descriptors, particularly
in the context of experimental research. Predictor variables must be carefully
chosen or designed to maximize differences due to their effects. Reliable
and valid criterion variables must be selected or designed to accurately
measure change caused by the predictor variables (Pickering, 1997).
Kerlinger (1973) offered an often-cited mnemonic to define what is meant
by variance control in experimental research. MAXMINCON refers to MAXimizing
the variance associated with the relationship between the predictor and
criterion variables, MINimizing the error variance associated with measurement
of the criterion variables, and CONtrolling extraneous variance attributable
to other variables not included in the investigation. According to Kerlinger,
maximizing the variance related to the interaction of the predictor and
criterion variables requires designing the levels of the predictor variables
to be as different from each other as possible. Minimizing the error variance
is accomplished by controlling the conditions in which the investigation
is conducted and choosing reliable measures of the criterion variables.
Controlling extraneous variance may involve any of a variety of procedures,
such as selecting a group of subjects who are homogeneous on the variable,
randomly selecting and assigning subjects to groups, or perhaps adding
the variable to the investigation as another predictor variable.
Internal validity refers to the level of confidence that the predictor
variable(s), rather than an extraneous variable, produced the change found
in the criterion variable(s). Hadley and Mitchell (1995), Heppner, Kivlighan,
and Wampold (1992), and Isaac and Michael (1981) listed a variety of threats
to internal validity including group composition, experimental mortality,
history, maturation, practice effects, placebo effects, the Hawthorne effect,
the John Henry effect, experimenter bias, demand characteristics, rater
and observer effects, instrumentation, and statistical regression. For
example, when subjects are selected because of group membership rather
than being randomly selected, when subjects know they are part of an experiment
and merely respond to receiving "special" treatment, or when the measuring
instruments are not reliable and valid, it is doubtful whether the treatment
variable caused the change in the criterion variable or whether it was
caused by some other extraneous factor(s) (Pickering, 1997).
External validity is the degree to which the results can be generalized
to other populations. Hadley and Mitchell (1995), Heppner, et al. (1992),
and Isaac and Michael (1981) described a variety of threats to external
validity including initial population-sample differences, mortality, artificial
research arrangements, pretest influence, and multiple-treatment influences.
For example, generalization to other populations probably will be limited
if the sample chosen is not actually representative of the intended population,
if subjects who leave the investigation differ in some way from those who
remain, if subjects studied in laboratory settings perform differently
than they do in naturally occurring situations, if pretesting sensitizes
subjects to the treatment, or if multiple treatments are administered to
each subject. Managing threats to external validity involves attempting
to insure that both the subjects and the context in which the investigation
is conducted are appropriately representative.
RIGOR VERSUS RELEVANCE
Experimental research almost always results in procedural compromise
because control of one type of variance may cause problems in attempting
to control another type of variance. Gelso (1979) labeled this "the bubble
hypothesis," referring to the difficulty which arises when someone attempts
to place a decal on a window and a bubble appears. When the bubble is depressed
in one area, it arises somewhere else. Gelso also discussed how experimental
rigor is related to internal validity. Threats to internal validity are
most easily managed in controlled laboratory conditions, but human behavior
rarely occurs in tightly controlled laboratory situations and thus generalization
ISSUES IN DESIGNING AND IMPLEMENTING RESEARCH ON COUNSELING
Experimental methods are often touted as the sine qua non of research
in counseling. However, they are not always the most appropriate to answer
questions. Heppner, et al. (1992) stated that just as there is no uniform
method of counseling, there is no uniform method for conducting research.
They offered five guidelines to determine whether experimental methods
should be used: (a) does the professional literature support the use of
experimental methods or is a less rigorous approach more appropriate? (b)
does the literature include a variety of research methods or have one or
two methods dominated? (c) are sufficient resources available to support
the type of research method desired? (d) can rigor and relevance be balanced
to answer the research question? and (e) can responses to the previous
four guidelines be balanced with each other? Whenever experimental methods
are preferred or recommended to answer the research question, the issues
presented should be considered carefully in designing the investigation.
In reviewing the characteristics of experimental research presented
above, it is apparent that some aspects of counseling are not readily amenable
to the application of scientific research methods. This is not to say that
randomized control trials cannot or should not be undertaken when feasible.
However, due to the policies of some organizations that preclude use of
no treatment control groups and/or parental groups opposed to the involvement
of their children in research, these studies will be limited. Given the
many potential threats to validity, consumers of the results will need
to carefully evaluate the quality of the evidence as well as its relevance
to other counseling situations. Of particular interest to counselors is
the comparison of the effectiveness of different counseling approaches
in obtaining desired outcomes. Thus, an alternative to the treatment-no
treatment design is a Treatment A vs. Treatment B design. However, even
this design can produce flawed results due to differences in the implementation
of treatments by individual counselors. Thus, in assessing the usefulness
of results obtained from experimental studies, one must always consider
which is better - to base actions on poor experimental research (i.e.,
research which has not fully met the rigors of controlling all of the relevant
variables) or to base actions on good qualitative research (i.e., research
that applies intensive and extensive analysis from a number of perspectives
At the present time, it appears that those who teach and practice counseling
will have to rely heavily upon a careful evaluation of the results of all
types of research, well conducted meta-analyses of counseling research
(e.g., Sexton, et.al., 1987), and best human judgments. In fact, in a speech
delivered at a seminar where leading experts in the fields of education
and science discussed the meaning of scientifically based research hosted
by the Assistant Secretary for Elementary and Secondary Education, Feuer
(2002) stated that, "...what scientists themselves often acknowledge is
that there is a dimension of human judgment that can be missed with an
overzealous focus on the rigor of scientific method."
Campbell, D. C., & Stanley, J. C. (1966). Experimental and quasi-experimental
designs for research. Chicago, IL: Rand-McNally.
Feuer, M. (2002). The logic and the basic principles of scientific based
research. Paper presented at the Scientifically Based Research - U.S. Department
of Education seminar, February 6, 2002. (www.ed.gov/offices/OESE/esea/research/feuer-towne.html)
Gelso, C. J. (1979). Research in counseling: Methodological and professional
issues. The Counseling Psychologist, 8 (3), 7-35.
Hadley, R. G., & Mitchell, L. K. (1995). Counseling research and
program evaluation. Pacific Grove, CA: Brooks/Cole.
Heppner, P. P., Kivlighan, D. M., Jr., & Wampold, B. E. (1992).
Research design in counseling. Pacific Grove, CA: Brooks/Cole.
Huitema, B. E. (1980). The analysis of covariance and alternatives.
New York, NY: Wiley.
Isaac, S., & Michael, W. B. (1981). Handbook in research and evaluation
(2nd ed.). San Diego, CA: EdITS.
Kerlinger, F. N. (1973). Foundations of behavioral research (2nd ed.).
New York: Holt, Rinehart and Winston. Loesch, L. C., & Vacc, N. A.
(Eds.) (1997). Research in counseling & therapy. Greensboro, NC: ERIC
Counseling and Student Services Clearinghouse.
Sexton, T. L., Whiston, S. C., Bleuer, J. C., & Walz, G. R. (1997).
Integrating outcome research into counseling practice and training. Alexandria,
VA: American Counseling Association.
Taub, J. (2002). Does it work? The New York Times, November 10, 2002.