ERIC Identifier: ED435710
Publication Date: 1999-09-00
Author: Scriven, Michael
Source: ERIC Clearinghouse on Assessment and Evaluation Washington DC.
The Nature of Evaluation. Part I: Relation to Psychology. ERIC/AE Digest.
The discipline of evaluation is devoted to the systematic determination of merit, worth, or significance. It is divided into fields according to the type of entity evaluated for example, program evaluation, or personnel evaluation and there are more than twenty of these recognized fields of evaluation. Some specific aspects of evaluation methodology have been developed to solve problems of evaluation in only one or a few of these fields (e.g., bias control in panel selection, systematic side-effect identification in program evaluation, road-testing techniques in product evaluation).However, the underlying logic of the process of evaluation for example, the difference between merit and worth, or between grading and ranking and a substantial portion of its general methodology (e.g., techniques of measurement, causality determination, applying the requirement of informed consent) are shared across all or many of these fields. Many of these general techniques (the 'general methodology') come from the applied social sciences, and are learnt by students in the normal course of education in those fields. But the logic of evaluation has been developed for and applies only to evaluation; and the field-specific methodologies of evaluation must also be mastered in order to deal with evaluation in the fields to which they apply. Teaching evaluation therefore focuses on these evaluation-specific topics, the general logic and the special methodologies.
This article addresses the role of evaluation, the basic logic, and a description of how the field is structured. A separate article describes some of the basic logic of evaluation skills and methodological skills that need to be mastered.
EVALUATION IN APPLIED PSYCHOLOGY
Just as it was previously found that a good grasp of probability and statistics had become an essential tool for a great deal of work in applied psychology, so we now find that a knowledge of the logic of evaluation and of some of its specialized methodologies are increasingly crucial for much investigation in applied social science. Funding agencies letting evaluation contracts or assigning staff to evaluation increasingly want to know not only what is happening when an intervention is supported, and exactly what causes the results - familiar questions for applied social science - but also (I) whether the intervention is worth what it cost, (ii) whether there are unintended bad results as well as planned good ones, (iii) whether the methods used in the intervention were proper by current professional and ethical standards, and (iv) whether there are better ways to do the same thing. This is the domain of evaluation, and none of those questions can be answered reliably without some use of its special logic and its special methodologies.
THE BASIC LOGIC OF EVALUATION
In order to reach evaluative conclusions it is usually necessary to establish or identify two kinds of premises: factual premises (e.g., about nature, performance or impact) and value premises (e.g.,about the relevant legal or scientific principles).There are usually many of these premises in the evaluation of complex entities (or entities with complex functions),and there may be several hundred of them. To obtain the required kind of overall evaluative conclusion, it is typically necessary to combine all of them by means of what is called 'the internal synthesis process.'This synthesis step is one of the key logical processes in evaluation and is a long way from the simple deduction and statistical inference that are more common elements in scientific inference.
Factual premises in a field like program evaluation are commonly established using the standard procedures of social science, sometimes with the assistance of other disciplines such as history and jurisprudence. Value premises usually come from one or more of eight sources: legal principles; scientific and mathematical standards of truth (especially relevant when the program disseminates information or is based on scientific theories or common assumptions); professional, cultural, or organizational standards of proper conduct (e.g., the APA testing standards); needs assessments; definitions (which provide linguistic standards of propriety); market research; logic; and ethics. Again, the social sciences are a common source for several of these types of values e.g., the scientific standards of truth used in judging the quality of the assertions or assumptions built into or propagated by the program. From psychology, we frequently encounter premises about maturational rates, cognitive processes, or leadership research. The logic of evaluation comes in with the frequent need to balance these value considerations when they conflict: that logic originates in jurisprudence and moral reasoning, but has been expanded to cover other fields of evaluation besides these, e.g., evaluation of alternatives in high-stakes decision-making.
The better-known fields of applied evaluation vary considerably in quality as well as in their relevance to and dependence on the social sciences. While most of program and personnel evaluation is heavily dependent on the social sciences and capable of a high degree of objectivity and utility, others vary independently on these dimensions. Some come close to being pseudo-evaluative (e.g., wine tasting, art criticism), some are partially valid (architectural criticism, portfolio management, literary criticism), and some support highly valid evaluations but are not dependent on the social sciences (e.g., the reviews done by appellate courts, the evaluations of claimed proofs of Fermat's Theorem in mathematical journals).Eight of these applied fields are of particular importance, for practical or logical reasons: they fall into two groups. The 'Big Six' are the fields of program, personnel, performance, policy, proposal, and product evaluation (the latter including technology assessment).The 'Super Two' are: (I) intradisciplinary evaluation the evaluation of the entities that are the currency of a discipline's economy such as theories, hypotheses, classifications, data, research designs and results, practitioners, contributions, and journals and of the discipline itself: and (ii) meta-evaluation, the evaluation of evaluations themselves, a practice that demonstrates the reflexive nature of evaluation and the reflective integrity of its practitioners. The first is the backbone of all disciplines it is what makes them disciplines. The second is the backbone of evaluation it is what makes it consistent by making it practice what it preaches.
Of the Big Six fields the conventional fields of evaluation program evaluation is the one with the largest associated job market at the moment, with personnel evaluation (an Human Resources staple) and performance evaluation (especially in the educational area, where its academic fountain is often referred to as 'tests and measurement') coming next. Additional Reading
Chelimsky, E and Shadish W.R (eds) Evaluation for the 21st Century : A Handbook. Sage Publications
Joint Committee on Standards for Educational Evaluation (1998). Program Evaluation Standards : How to Assess Evaluations of Educational Programs. Corwin Press
Scriven, M. (1991) Evaluation Thesaurus 4th edition. Sage Publications.
Shadish W. R. (Chair) (1998) Guiding Principles for Evaluators. A Report from the American Evaluation Association Task Force on Guiding Principles for Evaluators.[available online http://www.eval.org/EvaluationDocuments/aeaprin6.html].
Shadish, W.R. (1998) Some Evaluation Questions. ERIC/AE
Library Reference Search
Please note that this site is privately owned and is in no way related to any Federal agency or ERIC unit. Further, this site is using a privately owned and located server. This is NOT a government sponsored or government sanctioned site. ERIC is a Service Mark of the U.S. Government. This site exists to provide the text of the public domain ERIC Documents previously produced by ERIC. No new content will ever appear here that would in any way challenge the ERIC Service Mark of the U.S. Government.