ERIC Identifier: ED305497
Publication Date: 1989-00-00
Author: Halasz, Ida M.
Source: ERIC Clearinghouse on Adult Career and Vocational Education Columbus OH.
Evaluation Strategies for Vocational Program Redesign. ERIC Digest No. 84.
The educational reform movement of the 1980s has spurred interest in evaluation as a step in redesigning vocational programs to meet the needs of the work force in the year 2000. According to Hull (1987), "the delivery of vocational education to secondary students may be the single most important concern confronting American education today...because the competitive strength of the nation depends on a skilled work force" (p. iv).
This ERIC Digest examines the role of evaluation in vocational education program redesign. Following an assessment of previous evaluation efforts, evaluation strategies for program redesign are described. The Digest concludes with comments about the use of information in program evaluation.
ASSESSMENT OF PREVIOUS EVALUATION EFFORTS
Much of the evaluation of vocational education conducted at the state and local levels has been conducted in response to federal legislation (Merkel-Keller 1988). Although some of the evaluative information has been useful for program improvement (Wentling and Barnard 1984), much of it has been collected only for compliance with federal and state mandates (Starr 1986).
Despite the lack of use and usefulness of some of the conventional evaluation methods, the educational reform movement has prompted school officials to turn to evaluation to answer the difficult questions about redesigning their vocational programs at the state and local levels (Hull 1987). There has been a great deal of criticism about the usefulness of evaluation results, the criteria used in evaluation efforts, and the methodology used to evaluate vocational education programs. McKinney and his colleagues (1985) state that "if program evaluation is to point the direction for program improvement in vocational education, then vocational education must be willing to reexamine program evaluation and to identify and resolve the crucial programs limiting the effectiveness of evaluation" (p. v).
The theoretical and practical criticisms of strategies commonly used for evaluating vocational education programs include the following described by Farley et al. (1985) and Sirotnik (1984): o Vocational education is treated as a simple economic "input-output" model, focusing on outcomes such as job placement. This is too simplistic for a complex, multifaceted phenomenon such as vocational education and does not provide useful information about problems and deficiencies in delivering vocational education. o This model focuses the outcomes on the economic consequences of students' participation in vocational education at the expense of noneconomic social and educational goals and outcomes (for example, interpersonal relations skills, problem-solving skills, career planning) that are valued by local practitioners. o Current strategies are based on the empirical-analytic model of scientific inquiry, which deals only with unbiased or objective facts. This excludes values, a very important aspect of understanding human behaviors and the dynamics underlying practical problems in vocational education programs and schools. o Evaluations have not adequately considered other school processes (for example, counseling, administrative support) and the community (social mores, labor market structure) that interact and influence the quality of vocational education programs. o Experts dominate current evaluation practices, often to the exclusion of the individuals closest to the program (stakeholders) who not only have the most information to offer, but who also have the most to gain or lose from the changes recommended in the evaluations and will ultimately be responsible for implementing the recommended changes.
STRATEGIES FOR PROGRAM REDESIGN
Research on school effectiveness and program improvement indicates that if desired change or redesign is to occur, school culture and stakeholders' needs must be considered in the evaluation (Goodlad 1984; Sirotnik 1988). Evaluators such as Guba (1987) and Lincoln (1986) have been developing evaluation models and strategies (naturalistic inquiry, critical evaluation, qualitative methods) that incorporate stakeholders and broader, more realistic contextual information.
McKinney et al. (1985) offer a model for evaluating vocational education based on Habermas' critical theory, which assumes that individuals bring different perceptions of and vested interests in a given issue. Through formal discussions, the main activities of the critical evaluation process are the following (McKinney et al. 1985): o Framing evaluation problems --understanding program history and culture --identifying stakeholders' assumptions o Interpreting dynamics of problems --clarifying distorted understandings --constructing explanations of problems o Overcoming problems --conducting activities to remove obstacles and solve problems --revising explanations and perhaps uncovering additional problems that must be solved
USES OF INFORMATION IN PROGRAM REDESIGN
While espousing the critical evaluation or other nonconventional models, most evaluators do not disregard the need for and the uses of information and data nor the advantages of microcomputers in data analysis. Starr (1986) offers a method for evaluating programs that increases the relevance of vocational education offerings in meeting employment needs of students, employers, and labor market areas. He incorporates the use of both quantitative data and explicit value judgments for comparing and ranking ongoing programs or those that might be offered. His method differs from conventional approaches in its reliance on selecting and applying quantitative data by explicitly specifying the relative importance they attribute to the employment context, educational process, and the benefits of vocational education.
Sirotnik (1987) believes that there is a great need for valid information about the ongoing schooling process itself, including the daily circumstances, activities, and human orientations constituting the programs in the local schools. A variety of information should be collected (personal, instructional, institutional, societal, and so on) from multiple sources (teachers, students, administrators, parents) using multiple methods (survey, interview, participant observation, historical, archival).
Since the advent of microcomputers in the schools, the processes of collecting, storing, retrieving, analyzing, and reporting the multiple forms of information should not be a major problem, at least technologically. The database of information collected by each school should be used in the formal evaluation process (for example, critical evaluation) to provide stakeholders with relevant information about what occurs in their school.
Farley, J.; McKinney, F. L.; Kohan, A.; Smith, M.; and Pratzner, F. RECONCEPTUALIZATION OF VOCATIONAL EDUCATION PROGRAM EVALUATION. Columbus: The National Center for Research in Vocational Education, The Ohio State University, 1985. (ERIC Document Reproduction Service No. ED 255 762).
Goodlad, J. I. A PLACE CALLED SCHOOL: PROSPECTS FOR THE FUTURE. New York: McGraw-Hill, 1984.
Guba, E. G. "Naturalistic Evaluation." NEW DIRECTIONS FOR PROGRAM EVALUATION (EVALUATION PRACTICE IN REVIEW), no. 34. San Francisco: Jossey-Bass, Summer 1987. (ERIC No. EJ 355 077).
Hull, W. L. COMPREHENSIVE MODEL FOR PLANNING AND EVALUATING SECONDARY VOCATIONAL EDUCATION PROGRAMS IN GEORGIA. Columbus: The National Center for Research in Vocational Education, The Ohio State University, 1987. (ERIC Document Reproduction Service No. ED 284 983).
Lincoln, U. S. "The Development of Intrinsic Criteria for Authenticity: A Model for Trust in Naturalistic Researches." Paper presented at the American Educational Research Association Annual Meeting, San Francisco, April 1986. (ERIC Document Reproduction Service No. ED 270 459).
McKinney, F. L.; Farley, J.; Smith, M.; Kohan, A.; and Pratzner, F. CRITICAL EVALUATION FOR VOCATIONAL EDUCATION. Columbus: The National Center for Research in Vocational Education, The Ohio State University, 1985. (ERIC Document Reproduction Service No. ED 255 761).
Merkel-Keller, Claudia. "The Battle for Vocational Education: An Evaluation of the Study Plan for the National Assessment (NAVE) and Its First-Year Implementation." Paper presented at the American Education Research Association Annual Meeting, New Orleans, LA, April 5-9, 1988. (ERIC Document Reproduction Service No. ED 292 990).
Sirotnik, K. A. "An Outcome-Free Conception of Schooling: Implications for School-Based Inquiry and Information Systems." EDUCATIONAL EVALUATION AND POLICY ANALYSIS 6, no. 3 (Fall 1984): 227-239. (ERIC No. EJ 307 806).
Sirotnik, K. A. "The Information Side of Evaluation for Local School Improvement." INTERNATIONAL JOURNAL OF EDUCATIONAL RESEARCH 11, no. 1 (1987): 77-90. (ERIC No. EJ 346 460).
Sirotnik, K. A. "School-Centered Decision Making and Renewal." PHI DELTA KAPPAN 69, no. 9 (May 1988): 660-664. (ERIC No. EJ 370 312).
Starr, H. INCREASING THE RELEVANCE OF VOCATIONAL PROGRAMS: A DATA-BASED APPROACH. Columbus: The National Center for Research in Vocational Education, The Ohio State University, 1986. (ERIC Document Reproduction Service No. ED 284 978).
Wentling, T. L., and Barnard, W. S. A REVIEW OF LITERATURE RELATED TO
OUTCOMES OF VOCATIONAL EDUCATION. Urbana: University of Illinois, 1984. (ERIC
Document Reproduction Service No. ED 250 485).
Library Reference Search
Please note that this site is privately owned and is in no way related to any Federal agency or ERIC unit. Further, this site is using a privately owned and located server. This is NOT a government sponsored or government sanctioned site. ERIC is a Service Mark of the U.S. Government. This site exists to provide the text of the public domain ERIC Documents previously produced by ERIC. No new content will ever appear here that would in any way challenge the ERIC Service Mark of the U.S. Government.