ERIC Identifier: ED284522
Publication Date: 1986-00-00
Author: Conrad, Clifton F. - Wilson, Richard W.
Association for the Study of Higher Education.| ERIC
Clearinghouse on Higher Education Washington DC.
Academic Program Reviews. ERIC Digest.
Within the last few years, the role of academic program review has
emerged as one of the most salient issues in American higher education. Nestled
within a context of accountability, program reviews have become a dominant and
controversial activity. Although the institutional authority varies greatly,
higher education agencies in all 50 states now conduct state-level reviews; 28
of those agencies have authority to discontinue programs. Moreover, a majority
of the multicampus systems have introduced program reviews, and over
three-fourths of the nation's colleges and universities employ some type of
program review. The heightened interest in program review can be traced to a
widespread interest in improving program quality and the need to respond
creatively to severe financial constraints and to external constituencies'
expectations for accountability.
The literature contains a generous amount of controversy regarding the
purposes, processes, and outcomes of program review. The intent of this digest
is to illuminate this terrain: to capture the diverse institutional approaches
to review, to examine the central issues, and to reflect on ways in which
program review might be improved. Toward that end, the report is based on a
review of the literature and an analysis of program review practices at 30
WHAT DISTINGUISHES CURRENT PROGRAM REVIEWS?
Colleges and universities have a longstanding tradition of program
evaluation, a tradition that can be traced from colonial and antebellum colleges
to modern American universities. Until well into this century, program reviews
were viewed largely as internal matters, initiated most often to reform and
revitalize the curriculum. The idea that program reviews should be conducted to
demonstrate accountability to external constituencies is a phenomenon of the
twentieth century. The gradual development of regional and professional
accrediting associations and the creation of statewide governing and
coordinating boards are at least partly the result of a belief that programs
must be responsive to the needs and expectations of external as well as internal
Especially in the last several years, program reviews have been designed to
achieve another major objective: aiding those making decisions about the
reallocation of resources and program discontinuance. Thus, a broad range of
expectations now exists for program review in higher education. Program
improvement, accountability to external constituencies, and resource
reallocation are the purposes cited most often. Despite this growth in
expectations, there is little evidence to suggest that an evaluation system can
be designed to address multiple purposes simultaneously. It is especially
difficult to pursue both program improvement and resource reallocation at the
same time, and an institution's interests are served best if reviews focused on
program improvement are conducted separately from those concerned with
WHAT DO FORMAL EVALUATION MODELS CONTRIBUTE?
Program reviews at most institutions draw heavily on one or more of several
models: goal-based, responsive, decision-making, or connoisseurship. Although
these models are seldom explicitly identified in descriptions of institutional
review processes, they can be inferred from the procedures used.
The goal-based model has had the most influence, offering the advantages of
systematic attention to how a program has performed in relation to what was
intended and concern for the factors contributing to success or failure. The
characteristic of the responsive model that has influenced program reviews in
higher education is the attention given to program activities and effects,
regardless of what program goals might be. The central concern of an evaluation,
according to a proponent of responsive evaluation, ought to be the issues and
concerns of those who have an interest in the program, not how a program has
performed relative to its formal goal statements.
The major contribution of the decision-making model to program review in
higher education is the explicit attempt to link evaluations with decision
making, thus focusing the evaluation and increasing the likelihood that results
will be used. The connoisseurship model of evaluation has a long tradition in
higher education. It relies heavily on the perspectives and judgments of
experts, which are valued because of the individual's assumed superior knowledge
and expertise and a commonly shared value system.
HOW SHOULD QUALITY BE ASSESSED?
The assessment of quality has generated more confusion and debate than any
other issue for those engaged in program review. Pressure to define what quality
means and what types of information should be collected has always existed, but
interest has been heightened by the relatively recent emphasis on program review
for resource reallocation and retrenchment.
Four different perspectives have been offered on how quality should be
defined: the reputational view, the resources view, the outcomes view, and the
value-added view. The reputational view assumes that quality cannot be measured
directly and is best inferred through the judgments of experts in the field. The
resources view emphasizes the human, financial, and physical assets available to
a program. It assumes that high quality exists when resources like excellent
students, productive and highly qualified faculty, and modern facilities and
equipment are prevalent.
The outcomes view of quality draws attention from resources to the quality of
the product. Faculty publications, students' accomplishments following
graduation, and employers' satisfaction with program graduates, for example, are
indicators used. The problem with the outcomes view is that the program's
contribution to the success of graduates, for example, is not isolated. It is
assumed that if the graduate is a success, the program is a success.
The value-added view directs attention to what the institution has
contributed to a student's education (Astin, 1980). The focus of the value-added
view is on what a student has learned while enrolled. In turn, programs are
judged on how much they add to a student's knowledge and personal development.
The difficulty with this view of quality is how to isolate that contribution.
Most institutions assess quality by adopting aspects of all four views. The
assumption is that quality has multiple dimensions and, in turn, that multiple
indicators should be used for its assessment. A large number of quantitative and
qualitative indicators have been suggested for making such assessments.
DO PROGRAM REVIEWS MAKE A DIFFERENCE?
Perhaps the most significant issue relating to program review is the effect
of the considerable activity at all levels of higher education. The assessment
of impact requires that attention be given to the longer-term effects of
decisions that are made, that is, whether a program is stronger, more efficient,
or of higher quality. The major criterion to use in assessing impact is whether
an evaluation makes a system function better.
Only a few studies have analyzed impact systematically. The University of
California and the University of Iowa benefited from program reviews, including
providing a stimulus for change and improving knowledge among decision makers
about programs. Not all analyses of impact are as positive, however. A small
number of studies have focused on cost savings and have found that little money
is saved--that, in fact, reviews frequently require an increased commitment.
Program reviews can have negative effects--unwarranted anxiety, diversion of
time from teaching and research, and unfulfilled promises and expectations.
The continued existence and growth of program review processes suggest that
such efforts are supported and that the results can be beneficial. Given the
plethora of program reviews at all levels of higher education, the need to study
the effects of such reviews more systematically is urgent.
FOR MORE INFORMATION
Astin, Alexander W. "When Does a College Deserve to Be Called 'High
Quality'?" In IMPROVING TEACHING AND INSTITUTIONAL QUALITY, Current Issues in
Higher Education 1. Washington, DC: American Association for Higher Education,
1980. ED 194 004.
Barak, Robert J. PROGRAM REVIEW IN HIGHER EDUCATION: WITHIN AND WITHOUT.
Boulder, CO: National Center for Higher Education Management Systems, 1982. ED
Clark, Mary Jo, Rodney T. Hartnett, and Leonard L. Baird. ASSESSING
DIMENSIONS OF QUALITY IN DOCTORAL EDUCATION: A TECHNICAL REPORT OF A NATIONAL
STUDY OF THREE FIELDS. Princeton, NJ: Educational Testing Service, 1976. ED 173
Conrad, Clifton F. and Robert T. Blackburn. "Research on Program Quality: A
Review and Critique of the Literature." In HIGHER EDUCATION: HANDBOOK OF THEORY
AND RESEARCH, vol. 1, edited by John C. Smart. New York: Agathon, 1985.
Cronbach, Lee J. "Remarks to the New Society." EVALUATION RESEARCH SOCIETY
NEWSLETTER 1 (1977): 1-3.
Gardner, Don E. "Five Evaluation Frameworks: Implications for Decision Making
in Higher Education." JOURNAL OF HIGHER EDUCATION 8 (1977): 571-593.
George, Melvin D. "Assessing Program Quality." In DESIGNING ACADEMIC PROGRAM
REVIEWS, edited by Richard F. Wilson. New Directions for Higher Education No.
37. San Francisco: Jossey-Bass, 1982.
Seeley, John. "Program Review and Evaluation." In EVALUATION OF MANAGEMENT
AND PLANNING SYSTEMS, edited by Nick L. Poulton. New Directions for
Institutional Research No. 31. San Francisco: Jossey-Bass, 1981.
Skubal, Jacqueline M. "State-Level Review of Existing Academic Programs: Have
Resources Been Saved?" RESEARCH IN HIGHER EDUCATION 2 (1979): 223-232.
Smith, S. "Program Review: How Much Can We Expect?" Unpublished report.
Berkeley, CA: University of California-Berkeley, 1979.
Wilson, Richard F., ed. DESIGNING ACADEMIC PROGRAM REVIEWS. New Directions
for Higher Education No. 37. San Francisco: Jossey-Bass, 1982.