ERIC Identifier: ED414517
Publication Date: 1995-00-00
Author: Schulz, William E.
Source: ERIC Clearinghouse on
Counseling and Student Services Greensboro NC., Canadian Guidance and
Counselling Foundation Ottawa (Ontario).
Evaluating Career Counseling Centers: A Collaborative Approach:
Professional evaluation of career counseling programs is becoming
increasingly important as funding becomes more limited and accountability more
pervasive. The challenge is to make evaluation a positive experience for the
counselors, administrators, and clients, while, at the same time, to satisfy the
more quantitative needs of the policy planners and program funders. Greene
(1994) has suggested that the more narrative, participatory approach of
qualitative evaluation, can foster greater understanding, solidarity, and
contextualization of a career counseling program. This paper describes the
approach used by an outside team to evaluate seven small career counseling
programs. The evaluations were funded mainly by the major sponsor of the
counseling centers. Since the evaluation team wanted to involve and satisfy both
the funder and those involved in service delivery, a quantitative/qualitative
evaluation process and a strength challenge approach to counselor feedback was
Although the evaluation mandate varied somewhat among the seven centers,
evaluators addressed two basic goals:
Appraise the effectiveness and efficiency of the administration and delivery of
career counseling services, and
Suggest areas of improvement and identify areas where initiatives had been taken
or where innovative procedures had been developed to improve counseling
services. The following eight areas were identified for review: client
inventories and record keeping; service-needs determination; leadership,
planning and organization; client perception of employment coordination and
counseling; liaison with referral agencies; individual and group counseling;
employment success rate; and additional services and innovations.
DISCUSSION OF THE EVALUATION PROCESS
The evaluation team
realized that all the stakeholders in the career counseling centers were
apprehensive about "these outsiders" evaluating them and possibly affecting
their future funding. Therefore, a collaborative approach was adopted, in which
evaluators request input from the counseling centers and invite evaluatees
reactions to the observations of the evaluation team. The evaluation process
included the following steps:
Examination of materials such as written mission statements, objectives, monthly
and annual reports. The evaluation team wanted to learn as much as possible
about the center before the on-site review.
Preparation of suitable review forms. Counselors and administrators had an
opportunity to see various forms and select the ones they felt would be most
helpful. The "Project Worker Issues" form, was used most frequently, since it
helped to stimulate discussion. This open-ended form included five major areas:
a) Client Issues: Describe your typical clients. What do your clients expect of
you? What changes would you like to see so that clients could be served better?
b) Operational Issues: What is done at your center to foster team building? What
is done with client follow-up? What structural changes would help you offer
better services to clients?
c) Accountability Issues: Comment on client feedback services provided by your
center. What measures are taken to be accountable to funding agencies? Does
management provide written materials on management style, standards of counselor
d) Self-Assessment Issues: Do you have a written set of standards for the
services you offer (e.g., number of sessions, wait time, referrals, etc.)? Can
you "deliver" the services that you offer?
e) Professional Development Issues: How are the competency levels of counselors
assessed? Are professional development days provided for all counselors? What
are your future plans for professional development?
Meeting of the review team members to plan the specifics of the 2-3 day on-site
A 2-3 day on-site evaluation by the evaluators.
Evaluators meet to discuss findings and plan the recommendations.
Preliminary report of the evaluation team sent to the career counseling center
for comments and to correct any inaccuracies. This report contained information
on the eight areas for review identified earlier.
Final report sent to the career counseling center.
On-site visitation by the review team to discuss the recommendations and
possible ways of implementing the recommendations.
The key part of the whole evaluation was the 2-3 day on-site visitation by
the review team. The attitude the team attempted to foster was "what can we do
in the next 3 days that might make this process useful to counselors and
clients?" At an introductory meeting with all staff (sometimes lasting several
hours) this question, other open-ended questions, and fun introductions helped
to set a relaxed tone for the evaluation process.
The evaluation team found that helping clients, counselors and administrators
become more comfortable talking about themselves and their career counseling was
of great value in getting a better picture of what the counselors were doing and
what clients were receiving or expecting. Savickas (1989) used the term
"enhancing narrative ability" to describe this process. Several activities that
worked well included the following:
evaluation team shared their own experiences, talking about heroes or heroines
who had influenced them. One counselor said that when she was seven years old
she adored Florence Nightingale, and later, as an adolescent, she decided she
wanted to be like Mother Theresa. One of her colleagues volunteered, "You are
Mother Theresa to us and all the people you see." The evaluators learned a lot
about this woman's attitudes toward people and counseling from her story.
us about significant people in your life who would be good to have as colleagues
at your center."
five of the most important things that are occurring at your center.
The remainder of the on-site evaluation time was spent talking with staff
members, examining and discussing files, monthly and annual reports, and
referral systems, and being observers in individual and group counseling
In response to the question, "What could make the evaluation worthwhile?",
the majority of counselors asked for feedback on their counseling with clients.
Most had arranged with a client to have an evaluator present during a counseling
session. Formative evaluation for individual counselors was enhanced by using a
strength-challenge style of feedback (Borgen, Amundson, & Westwood, 1988).
In this approach the counselor receiving feedback collaborates beforehand about
the focus or direction of the feedback. When the feedback session begins, the
receiver is allowed some self-analysis, followed by specific comments from the
observer. The feedback is very specific and focuses on what the observer saw.
The emphasis is on the strengths or competencies that a counselor demonstrates
and the possible need to make greater use of the talents or strengths. For
example, an evaluator might say: "I thought your use of reflection of feeling
was very effective. You said: '...and now you're really frustrated and angry
that you can't find suitable employment...' Your client responded to this and
vented some of her anger and frustration. This type of reflection of feeling
works well for you and I encourage you to use it more" (Schulz, 1990.)
Several hours were used at the end of the on-site visit to again meet with
all counselors and administrators. This final meeting gave the staff an
opportunity to ask questions, to "dream a little" and list some things needed at
their counseling center, and to talk about any special counseling services that
their center had introduced for clients.
RECOMMENDATIONS FOR EVALUATORS
The collaborative approach
used to evaluate career counseling centers was effective because of reduced
threat to stakeholders and the resulting cooperation and involvement of
counselors and administrators. The following recommendations should help build a
collaborative approach to evaluation:
time to build rapport with the career counselors before beginning the actual
all the stakeholders beforehand that they will be involved in reviewing the
evaluation report before release.
staff and administration input into the review process and review instruments.
reviewers should spend much time with the evaluation process and should be aware
of the process interacting significantly with the outcome goals. Both formative
and summative evaluation should be stressed.
studies and anecdotal information should be part of the career counseling
follow-up of the review was highly valued and is strongly recommended as the
final step of the evaluation process.
Borgen, W.A., Amundson, N.E., & Westwood,
M.J. (July, 1988). Presentation on feedback given to counsellor educators in
Greene, J.C. (1994). Qualitative program evaluation: Promise and practice. In
N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp.
530-544). Newbury Park, CA: Sage.
Savickas, M.L. (1989). Career-style assessment and counseling. In T. Sweeney
(Ed.), Adlerian counseling: A practical approach for a new decade (3rd ed.) (pp.
289-320). Muncie, IN: Accelerated Development Press.
Schulz, W.E. (1990). Making feedback a positive experience. Unpublished
manuscript. Winnipeg, MN: University of Manitoba.