ERIC Identifier: ED305497
Publication Date: 1989-00-00
Author: Halasz, Ida M.
Source: ERIC Clearinghouse on Adult
Career and Vocational Education Columbus OH.
Evaluation Strategies for Vocational Program Redesign. ERIC
Digest No. 84.
The educational reform movement of the 1980s has spurred interest in
evaluation as a step in redesigning vocational programs to meet the needs of the
work force in the year 2000. According to Hull (1987), "the delivery of
vocational education to secondary students may be the single most important
concern confronting American education today...because the competitive strength
of the nation depends on a skilled work force" (p. iv).
This ERIC Digest examines the role of evaluation in vocational education
program redesign. Following an assessment of previous evaluation efforts,
evaluation strategies for program redesign are described. The Digest concludes
with comments about the use of information in program evaluation.
ASSESSMENT OF PREVIOUS EVALUATION EFFORTS
Much of the
evaluation of vocational education conducted at the state and local levels has
been conducted in response to federal legislation (Merkel-Keller 1988). Although
some of the evaluative information has been useful for program improvement
(Wentling and Barnard 1984), much of it has been collected only for compliance
with federal and state mandates (Starr 1986).
Despite the lack of use and usefulness of some of the conventional evaluation
methods, the educational reform movement has prompted school officials to turn
to evaluation to answer the difficult questions about redesigning their
vocational programs at the state and local levels (Hull 1987). There has been a
great deal of criticism about the usefulness of evaluation results, the criteria
used in evaluation efforts, and the methodology used to evaluate vocational
education programs. McKinney and his colleagues (1985) state that "if program
evaluation is to point the direction for program improvement in vocational
education, then vocational education must be willing to reexamine program
evaluation and to identify and resolve the crucial programs limiting the
effectiveness of evaluation" (p. v).
The theoretical and practical criticisms of strategies commonly used for
evaluating vocational education programs include the following described by
Farley et al. (1985) and Sirotnik (1984): o Vocational education is treated as a
simple economic "input-output" model, focusing on outcomes such as job
placement. This is too simplistic for a complex, multifaceted phenomenon such as
vocational education and does not provide useful information about problems and
deficiencies in delivering vocational education. o This model focuses the
outcomes on the economic consequences of students' participation in vocational
education at the expense of noneconomic social and educational goals and
outcomes (for example, interpersonal relations skills, problem-solving skills,
career planning) that are valued by local practitioners. o Current strategies
are based on the empirical-analytic model of scientific inquiry, which deals
only with unbiased or objective facts. This excludes values, a very important
aspect of understanding human behaviors and the dynamics underlying practical
problems in vocational education programs and schools. o Evaluations have not
adequately considered other school processes (for example, counseling,
administrative support) and the community (social mores, labor market structure)
that interact and influence the quality of vocational education programs. o
Experts dominate current evaluation practices, often to the exclusion of the
individuals closest to the program (stakeholders) who not only have the most
information to offer, but who also have the most to gain or lose from the
changes recommended in the evaluations and will ultimately be responsible for
implementing the recommended changes.
STRATEGIES FOR PROGRAM REDESIGN
Research on school
effectiveness and program improvement indicates that if desired change or
redesign is to occur, school culture and stakeholders' needs must be considered
in the evaluation (Goodlad 1984; Sirotnik 1988). Evaluators such as Guba (1987)
and Lincoln (1986) have been developing evaluation models and strategies
(naturalistic inquiry, critical evaluation, qualitative methods) that
incorporate stakeholders and broader, more realistic contextual information.
McKinney et al. (1985) offer a model for evaluating vocational education
based on Habermas' critical theory, which assumes that individuals bring
different perceptions of and vested interests in a given issue. Through formal
discussions, the main activities of the critical evaluation process are the
following (McKinney et al. 1985): o Framing evaluation problems --understanding
program history and culture --identifying stakeholders' assumptions o
Interpreting dynamics of problems --clarifying distorted understandings
--constructing explanations of problems o Overcoming problems --conducting
activities to remove obstacles and solve problems --revising explanations and
perhaps uncovering additional problems that must be solved
USES OF INFORMATION IN PROGRAM REDESIGN
While espousing the
critical evaluation or other nonconventional models, most evaluators do not
disregard the need for and the uses of information and data nor the advantages
of microcomputers in data analysis. Starr (1986) offers a method for evaluating
programs that increases the relevance of vocational education offerings in
meeting employment needs of students, employers, and labor market areas. He
incorporates the use of both quantitative data and explicit value judgments for
comparing and ranking ongoing programs or those that might be offered. His
method differs from conventional approaches in its reliance on selecting and
applying quantitative data by explicitly specifying the relative importance they
attribute to the employment context, educational process, and the benefits of
Sirotnik (1987) believes that there is a great need for valid information
about the ongoing schooling process itself, including the daily circumstances,
activities, and human orientations constituting the programs in the local
schools. A variety of information should be collected (personal, instructional,
institutional, societal, and so on) from multiple sources (teachers, students,
administrators, parents) using multiple methods (survey, interview, participant
observation, historical, archival).
Since the advent of microcomputers in the schools, the processes of
collecting, storing, retrieving, analyzing, and reporting the multiple forms of
information should not be a major problem, at least technologically. The
database of information collected by each school should be used in the formal
evaluation process (for example, critical evaluation) to provide stakeholders
with relevant information about what occurs in their school.
Farley, J.; McKinney, F. L.; Kohan, A.; Smith,
M.; and Pratzner, F. RECONCEPTUALIZATION OF VOCATIONAL EDUCATION PROGRAM
EVALUATION. Columbus: The National Center for Research in Vocational Education,
The Ohio State University, 1985. (ERIC Document Reproduction Service No. ED 255
Goodlad, J. I. A PLACE CALLED SCHOOL: PROSPECTS FOR THE FUTURE. New York:
Guba, E. G. "Naturalistic Evaluation." NEW DIRECTIONS FOR PROGRAM EVALUATION (EVALUATION PRACTICE IN REVIEW), no. 34. San Francisco: Jossey-Bass, Summer 1987. (ERIC No. EJ 355 077).
Hull, W. L. COMPREHENSIVE MODEL FOR PLANNING AND EVALUATING SECONDARY
VOCATIONAL EDUCATION PROGRAMS IN GEORGIA. Columbus: The National Center for
Research in Vocational Education, The Ohio State University, 1987. (ERIC
Document Reproduction Service No. ED 284 983).
Lincoln, U. S. "The Development of Intrinsic Criteria for Authenticity: A
Model for Trust in Naturalistic Researches." Paper presented at the American
Educational Research Association Annual Meeting, San Francisco, April 1986.
(ERIC Document Reproduction Service No. ED 270 459).
McKinney, F. L.; Farley, J.; Smith, M.; Kohan, A.; and Pratzner, F. CRITICAL
EVALUATION FOR VOCATIONAL EDUCATION. Columbus: The National Center for Research
in Vocational Education, The Ohio State University, 1985. (ERIC Document
Reproduction Service No. ED 255 761).
Merkel-Keller, Claudia. "The Battle for Vocational Education: An Evaluation
of the Study Plan for the National Assessment (NAVE) and Its First-Year
Implementation." Paper presented at the American Education Research Association
Annual Meeting, New Orleans, LA, April 5-9, 1988. (ERIC Document Reproduction
Service No. ED 292 990).
Sirotnik, K. A. "An Outcome-Free Conception of Schooling: Implications for
School-Based Inquiry and Information Systems." EDUCATIONAL EVALUATION AND POLICY
ANALYSIS 6, no. 3 (Fall 1984): 227-239. (ERIC No. EJ 307 806).
Sirotnik, K. A. "The Information Side of Evaluation for Local School
Improvement." INTERNATIONAL JOURNAL OF EDUCATIONAL RESEARCH 11, no. 1 (1987):
77-90. (ERIC No. EJ 346 460).
Sirotnik, K. A. "School-Centered Decision Making and Renewal." PHI DELTA
KAPPAN 69, no. 9 (May 1988): 660-664. (ERIC No. EJ 370 312).
Starr, H. INCREASING THE RELEVANCE OF VOCATIONAL PROGRAMS: A DATA-BASED
APPROACH. Columbus: The National Center for Research in Vocational Education,
The Ohio State University, 1986. (ERIC Document Reproduction Service No. ED 284
Wentling, T. L., and Barnard, W. S. A REVIEW OF LITERATURE RELATED TO
OUTCOMES OF VOCATIONAL EDUCATION. Urbana: University of Illinois, 1984. (ERIC
Document Reproduction Service No. ED 250 485).