ERIC Identifier: ED414525
Publication Date: 1995-00-00
Author: Flynn, Robert J.
Source: ERIC Clearinghouse on
Counseling and Student Services Greensboro NC., Canadian Guidance and
Counselling Foundation Ottawa (Ontario).
Follow-Up Evaluation of Career-Counseling Programs: ERIC
In this digest, "career counseling" refers to activities intended to improve
individuals' ability to make career decisions (Spokane, 1991). This includes
individual and group career and employment counseling, job-search training,
career education, career-planning courses, etc. "Follow-up evaluation" refers to
the assessment of program outcomes (effects) on one or more occasions after
completion of a program.
Meta-analyses (Lipsey & Wilson, 1993; Oliver & Spokane, 1988) have
shown that career counseling produces gains as large as those generated by
well-developed psychological, educational, or behavioral interventions in
general. Additional follow-up evaluations are needed, however, to improve our
understanding of why career counseling is effective, with whom, on which
outcomes, for how long, and under what conditions. This seems especially true of
Canada, where a mere 15-30% of career-counseling programs include follow-up
assessments; 35-45% are evaluated only through counselor-client contact during
counseling, and 25-40% are not evaluated at all (Conger, Hiebert, &
FOLLOW-UP EVALUATION OF CAREER COUNSELING: PURPOSES, OUTCOMES, AND PROCEDURES
The fundamental reason for conducting follow-up evaluations of career
counseling is the impossibility of judging the true value of such programs
without the extended time perspective afforded by follow-up (Morell, 1979). The
more time that elapses after clients finish a program, the more likely that the
program effects will have either decreased to pre-intervention levels,
maintained themselves, increased, and/or emerged as unanticipated consequences.
Other purposes for follow-up include establishing realistic expectations of what
a program can and cannot accomplish, learning how to improve a program, helping
decision-makers change the structure or funding of a program, or gathering
political information for defending (or attacking) a program (Morell, 1979).
Outcomes to Assess.
Researchers should use various instruments to assess career information:
measures of the same outcome (e.g., questionnaires, interviews, and
counseling-center records). Different types of measures (e.g., behavioral,
archival, and attitudinal) (Oliver, 1979).
measures (e.g., the attainment of clients' individual career-counseling goals)
and global measures (e.g., job satisfaction) should be used (Oliver, 1979).
longer the interval between program completion and follow-up, the harder it
becomes to relate clearly participants' actions to program activities (Morell,
1979). Thus, short-term outcome measures should often receive priority (Oliver,
of known reliability and validity from previous research are recommended, and
objective, non-reactive measures (e.g., archival data, cost data) should be
employed along with subjective measures (e.g., ratings, self-reports) (Oliver,
1979). Furthermore, both intermediate and ultimate outcomes of career counseling
should be assessed. Intermediate outcomes (e.g., job-seeking skills) lead to
final outcomes (e.g., employment status), and show why a program succeeds or
fails, and allow improvements to be made (Morell, 1979)."Learning outcomes"
(e.g., self-awareness, opportunity awareness, decision-making skills, and
transition skills) are important intermediate outcomes of career counseling and
precursors of subsequent socio-economic outcomes, such as earnings (Conger et
Major outcome domains to consider (Oliver & Spokane, 1988) include career
decision-making (e.g., accuracy of self-knowledge, appropriateness and realism
of choice, career information-seeking, decidedness, satisfaction); effective
role functioning (e.g., academic performance, job-interview skills, career
maturity, self-esteem, anxiety, need for achievement); and evaluation of
counseling (e.g., ratings of satisfaction or effectiveness).
A multidimensional set of rating scales has recently been proposed for
supplementary evaluation outcome measures (Spokane, 1991, pp. 219-224). These
scales cover the domains of persistent search and exploratory behavior,
information, realism, barriers, hope and morale, activity level, congruence,
cognitive framework, commitment and predicament appreciation, goals and options,
decisional process, anxiety, and performance.
key purpose(s) for conducting a follow-up evaluation (see above) needs to be
specified in advance, to guide choices about the most appropriate follow-up
time-frame, outcome measures, and procedures.
evaluations are typically "post hoc." implemented only after a program has
begun, and are thus unable to benefit from random assignment of participants to
programs or adequate control groups. Nevertheless, post hoc evaluations are well
worth doing as long as the evaluator considers plausible rival hypotheses and
recognizes that the evaluation will inevitably be less informative than if it
had been pre-planned and well controlled (Morell, 1979).
out more than one follow-up assessment after program completion allows a profile
of program effects over time to be determined. Confidence in the results of post
hoc evaluations increases when there is convergence among several "naturally
occurring" comparison groups: groups similar to those being studied but not
receiving career counseling, the past performance of the study group itself
prior to receiving career counseling, or successive program cohorts (Morell,
1979). Although post hoc evaluations do not allow the establishment of causal
relations, they do permit reasonable judgments about possible or even probable
relationships between program activities and client changes during the follow-up
period (Morell, 1979).Qualitative data based on program participants' opinions
should be used as a check on quantitative data, and vice versa. Also testing a
priori hypotheses about expected relationships will enhance the interpretability
of findings in post hoc, correlational evaluations (Morell, 1979).
Numerous techniques can increase response rates in follow-up surveys,
including personalized letters, repeated telephone or mail reminders, registered
mail, and payment for participation. A surprisingly high proportion of former
program participants can often be located through the mail, telephone
directories, public records, personal visits, specialized newspapers, alumni
associations, etc., and programs can maximize successful follow-up rates by
obtaining information during counseling that is relevant to maintaining contact
(Morell, 1979). Some understanding of the direction and magnitude of attrition
bias can be gained by comparing early and late responders, and responders and
Two Examples of Follow-Up Evaluation.
In a follow-up study conducted 3-6 months after career counseling had ended,
Nevo (1990) found that clients rated discussions with their counselor as the
single most useful component of career counseling, followed by ability tests,
career-related reading, and interest inventories; felt that career counseling
helped them more in promoting self-understanding than in fostering a specific
career decision; were more satisfied with counseling if they had been helped in
both the personal and career spheres, rather than in one sphere only; rated
their counselors' assistance in helping them organize their thinking and become
more aware of their interests and abilities as the most important factor in
their satisfaction with counseling.
In a case study, Kirschner, Hoffman, and Hill (1994) found, at an 18-month
follow-up, that a former career-counseling client had maintained her original
counseling gains and crystallized her career goals. At a 5-year follow-up, the
client identified her career-counseling experience as very influential in
helping her achieve several important outcomes: a positive job change; a high
degree of job satisfaction; greater awareness of the need to be more active in
her career decisions and interpersonal relationships; and increased
self-understanding, self-acceptance, and self-esteem.
We have identified some of the key purposes,
outcomes, and procedures that career-counseling practitioners and administrators
need to consider in planning useful follow-up evaluations of their services.
Follow-up assessment provides advantages that no other evaluation strategy
offers--information on program staying power, a profile of program effects over
time, and the identification of unintended consequences (Morell, 1979). Thus,
follow-up evaluation merits systematic application.
Conger, D. S., Hiebert, B., & Hong-Farrell,
E. (1994). Career and employment counselling in Canada. Ottawa: Canadian Labour
Force Development Board.
Kirschner, T., Hoffman, M. A., & Hill, C. E. (1994). Case study of the
process and outcome of career counseling. Journal of Counseling Psychology, 41,
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological,
educational, and behavioral treatment: Confirmation from meta-analysis. American
Psychologist, 48, 1181-1209.
Morell, J. (1979). Follow-up research as an evaluation strategy: Theory and
methodologies. In T. Abramson, C. K. Tittle, & L. Cohen (Eds.), Handbook of
vocational education evaluation (pp. 217-248). Beverly Hills, CA: Sage.
Nevo, O. (1990). Career counseling from the counselee perspective: Analysis
of feedback questionnaires. Career Development Quarterly, 38, 314-324.
Oliver, L. W. (1979). Outcome measurement in career counseling research.
Journal of Counseling Psychology, 26, 217-226.
Oliver, L. W., & Spokane, A. R. (1988). Career-intervention outcome: What
contributes to client gain? Journal of Counseling Psychology, 35, 447-462.
Spokane, A. R. (1991). Career intervention. Englewood Cliffs, NJ: