ERIC Identifier: ED414519
Publication Date: 1995-00-00
Author: Hiebert, Bryan
Source: ERIC Clearinghouse on Counseling and Student Services Greensboro NC., Canadian Guidance and Counselling Foundation Ottawa (Ontario).
A Changing Focus in Evaluation: Linking Process and Outline: ERIC Digest.
Career counselors continually evaluate their work; they draw conclusions and develop action plans based on numerous client activities: homework completion, client engagement, acquisition of interview skills or relaxation skills, number of employers contacted, and so forth. Both counselors and clients typically know when counseling is successful. Unfortunately, the evidence used to gauge success often is not considered evaluation, is not documented, and therefore cannot be used to back up claims that counseling has been successful.
Conger, Hiebert, and Hong-Farrell (1993) found that counseling is rarely evaluated in Canada. In some sectors, 40% of counselors reported never formally evaluating their work. This is unfortunate, since research has shown that career guidance and counseling consistently produce positive results. (See Killeen & Kidd, 1991; Oliver & Spokane, 1988.) In fact, Killeen and Kidd found that 90% of the studies recorded some positive counseling effects. Given these findings, it is surprising that counselors are not eager to evaluate such a predictably beneficial experience. Perhaps a new approach to evaluation is needed; one that counselors see as relevant, practical, and capable of embracing the informal observations that counselors and clients make about counseling progress.
AN ALTERNATIVE FRAMEWORK
Counseling is interactive. Client-need determines counseling intervention. The intervention is tailored to client characteristics and desired outcome. As counseling progresses, counselors adjust the approach to fit the client's changing situation. Evaluation models must accommodate this interplay between counselor and client and between process and outcome.
Figure 1 depicts a framework connecting long-term global impacts of counseling (e.g., job satisfaction, employability, career maturity); immediate outcomes associated with counseling (changes in client attitudes, knowledge, skills); client engagement in the counseling process; and, counselor approaches.
The loops in the framework illustrate the interactive nature of counseling.
Process and outcome interact in a circular fashion. Certain processes foster particular types of learning and in turn, create certain types of global impact. Reciprocally, the types of client skill, knowledge, and attitude necessary to achieve a certain kind of global impact can be identified, and the process needed to facilitate that learning can also be identified.
============================ FIGURE 1
1. Evaluation Components
2. Counseling Process
3. Counseling Outcome
4. System Requirements
5. What is the counselor doing:
Skills, Strategies, Tools.
6. What is the client doing?
Skill Practice, Homework, Self-Examination.
7. What did the client learn?
Skills, Knowledge, Attitudes.
8. What difference did the learning
make? Precursors, Presenting problem,
Evaluating the counseling process requires detailed information which links counselor activities and client reactions. This helps identify the processes that promote client change and that aid the development of alternative plans when sufficient progress is lacking. Client documentation might include engagement in the counseling process, homework completion, client openness and honesty, and client follow-through. Such data demonstrate that clients are doing their part in counseling. Evidence depicting the counselor as an indispensable part of the process might include the pattern of micro-skills used, the focus in a counseling session (both content and process), and data showing that an acceptable procedure for an intervention followed. These factors can be assessed from counselor case notes, client check-lists, semi-structured interviews, or formal questionnaires.
Counselors need to be clear about the legitimate outcomes (effects or products) of counseling (Hiebert 1989, 1994; Killeen & Kidd, 1991; Killeen White, & Watts, 1993). In Figure 1, counseling outcomes have two major components:
(1) Learning outcomes: the knowledge, skills, and attitudes that are directly linked to counseling. These are the legitimate outcomes of counseling:
knowledge about self, the labor market, job descriptions, entrance qualifications, and how to overcome barriers
skills for decision making, job interviews, self-management, making transition, and overcoming barriers like anxiety, substance abuse, poor financial planning
attitudes towards being planful, belief in self, motivation to look for work, self-esteem, increased optimism Learning outcomes arise both from counseling and from learning to apply existing skills to new contexts (e.g., using communication skills in conflict resolution). These are the legitimate outcomes of counseling, outcomes that counseling can reasonably hope to influence.
(2) Global outcomes reflect counseling's larger impact on the client's life. They include:
changes in client-presenting problem
getting along better with co-workers
increased job satisfaction
less stress during job interviews
fewer tension headaches
length of time to job offer
Administrators and sponsors often focus on global impact outcomes, but outcomes are influenced greatly by factors over which counseling has little control (e.g., the number of jobs available, corporate climate, etc.). Ultimately, it is important to demonstrate that counseling has affected these variables, but it is also important to refrain from promising that counseling can facilitate change in areas over which it has little control.
In addition to the above factors, program administrators and sponsors often are interested in factors such as adherence to mandates, types of clients, types of client problems, and client satisfaction with service. These variables are important to the functioning of the agency, but they are not indicators of the effectiveness of counseling per se. To emphasize this difference, they are called "system requirements." System requirements are part of the policies and procedures governing an agency and departures from these requirements should be negotiated between the counselor and the agency manager or sponsor.
Although standardized assessment is emphasized in counseling evaluation, there is frequent incongruence between the aims of standardized tests and the needs of clients (Killeen & Kidd, 1991). Implicit in Figure 1 is a far greater emphasis on informal procedures that document the judgments counselors and clients make about counseling progress. These include checklists for homework completion, skill mastery, skill implementation, steps completed in a program; subjective ratings of affective state (e.g., depression, motivation, stress level, job satisfaction); use of job interview skills during job interviews (see Hiebert (1991, 1994a, 1994b) for specific examples).
Currently, these informal measures are in their infancy. Better ways must be developed to track variables that are part of the counseling process and which have an influence on client change. For example, "planful" attitude is an important prerequisite (or co-requisite) to developing a career-action plan. Therefore, it is important to have a trustworthy and easy-to-use procedure to track changes in such attitudes.
SUMMARY: A CALL FOR ACTION
Two points underline the main arguments in this paper.
1. Counselors, program administrators, and sponsors need to reformulate their view of evaluation so that it provides an essential link between process and outcome; and, is an integral part of counseling, coequal with relationship building and intervention planning.
Evaluation needs to be planned and implemented alongside client-change intervention-not conducted at the end of a program by an external expert.
2. The scope of evaluation (what constitutes acceptable evidence) needs to be expanded to include the sorts of data that counselors and clients already collect on a regular basis. This includes client self-monitoring data, homework data, quantification of counselor observations in case notes, documentation of client in-session skill practice, goal attainment scaling, and performance assessments.
An evaluation model should assess the informal observations counselors and clients use to indicate whether they are on the right track, the amount of progress they are making, and the achieving of desired outcomes. It should encourage counselors to develop creative ways for documenting and quantifying those observations, and it should create non-quantified ways of portraying the evidence that clients, managers and supervisors find acceptable. This will ensure that evaluation needs are seen as relevant by all concerned: clients, counselors, agency managers, district supervisors, and funders.
Conger, D. S., Hiebert, B., & Hong-Farrell, E. (1993). Career and employment counselling in Canada. Ottawa, ON: Canadian Labour Force Development Board.
Hiebert, B. (1989). Meeting the accountability challenge. In B. O'Byrne, (Ed.), Natcon 15 (pp. 239-255). Toronto, ON: Ontario College Counsellor's Association.
Hiebert, B., (1994a). A Framework For Quality Control, Accountability, And Evaluation: Being Clear About The Legitimate Outcomes of Career Counseling. Canadian Journal of Counselling, 24.
Hiebert, B. (1994b). Moving to the future: Outcome-based comprehensive guidance and counselling in Alberta schools. Edmonton, AB: Alberta Education.
Killeen, J., & Kidd, J. (1991). Learning outcomes of guidance: A review of recent research. London, England: National Institute for Careers Education and Counselling.
Killeen, J., White, M., & Watts, A. G. (1993). The Economic Value of Careers Guidance. London, England: Policy Studies Institute.
Oliver, L. W., & Spokane, A. R. (1988). Career intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447-462.
Library Reference Search
Please note that this site is privately owned and is in no way related to any Federal agency or ERIC unit. Further, this site is using a privately owned and located server. This is NOT a government sponsored or government sanctioned site. ERIC is a Service Mark of the U.S. Government. This site exists to provide the text of the public domain ERIC Documents previously produced by ERIC. No new content will ever appear here that would in any way challenge the ERIC Service Mark of the U.S. Government.