Guidelines for the Development and Management of Performance Assessments. ERIC Digest.

This digest is based on time proven guidelines developed for use in training workshops for state and local educators to outline the processes by which performance assessments could be created, validated, and used in large-scale assessment. The term performance assessment as used in this digest is reserved primarily for those assessments that go beyond paper-and-pencil, group-administered assessments. This type of assessment is an important and unique tool available for measuring student performance at the state or local level.

These guidelines are provided to offer guidance to district and state policy makers and assessment directors concerning some of the issues of managing the development, administration, and use of performance assessments in large-scale assessment programs. The following sections present information about how performance assessments can be developed, administered, scored, and reported. The paper suggests informal, less costly means to develop and use these types of measures. However, some of the uses outlined above might require the use of external contractors or technical advisors.

PRE-ASSESSMENT DEVELOPMENT ACTIVITIES

Before assessment development can or should occur, several important planning activities must set the stage for assessment. These steps take place at the outset to insure that the assessment is developed in a manner that fits the content area to be assessed and is within the resources available. These steps include the development of the assessment framework, creation of the assessment plan, determination of assessment resources, and production of the assessment blueprint.

The framework for the assessment serves as the guide to the entire assessment. This consists of the assessment objectives. It is from the framework that the assessment is developed. There are many ways to determine these objectives. The traditional manner in which to do this is to gather a group of content area experts and classroom teachers and ask them to indicate what students should know and be able to do by the end of certain points of instruction. However, an alternate procedure is to allow the framework of expectations to evolve out of research and practical experience of educators on what outcomes students are capable of at particular times in their school career.

The assessment plan provides an overview and description of the types of assessment to be developed and used, as well as the manner in which assessments will be implemented. The plan serves the useful purpose of describing the types of assessments that are envisioned and how assessments will be administered, scored, and reported. The assessment plan should force the agency sponsoring the assessment development to carefully consider the resources needed and available in order to make decisions about the assessment at the outset. Once the assessment plan has been written and the assessment resources have been determined, it is possible to make any adjustments needed in the assessment plan.

The blueprint will describe the characteristics of an adequate assessment for each content area of the assessment framework as well as the characteristics of an adequate assessment for each student outcome. Once completed, this should guide the development of the assessments that are needed given the resources available. The final step in the pre-assessment development activities is for the assessment plan and the assessment blueprint to be approved by the sponsoring agency and any additional advisory groups or individuals.

ASSESSMENT DEVELOPMENT STEPS

After the blueprint has been created, it is time to formulate the assessment prompts. Throughout this process the developer must consider several things. These include: 1) in what manner should the assessment questions or instructions be presented to the student, 2) what additional stimulus materials will be needed, 3) how the students will respond and how such responses will be recorded and scored, 4) what criteria will be used to judge student responses, 5) the number of scale points for scoring student responses, and 6) samples of each level of response.

Once the exercises have been written, the next step is to edit them. This is an important step in assuring that the exercise and assessment administration processes are understandable, that the exercise warrants the special time and attention which performance measures require, and that there is consistency among the performance exercises. The editor also can check to ensure that the stimulus materials fit with the exercises.

One of the more difficult aspects of performance exercises is writing the assessment administration directions. Many performance exercises are administered to students individually or in small groups. These exercises require the assessment administrator to set up a standard situation to which each student can respond, as well as, to read a standard set of directions to each student. Developing standard assessment administration directions that are complete and accurate is usually the result of trying the exercise out one or more times and noting areas of students confusion, responses that students provided which are vague or incomplete, and ways in which some or all of students responded that were not anticipated. If a majority of student responses indicate that the assessment instrument is ineffective, considerable restructuring of the testing tool may be in order before any further testing of the instrument is completed.

Once the sample student responses are gathered, someone needs to review the responses and attempt to score them according to the criteria and preliminary scoring guide developed by the exercise writer. After the initial work on the scoring guide is complete, an expert panel of judges should be convened. This panel should be asked to review each exercise, confirm the preliminary judgments for each score scale point, and discuss those student responses that did not appear to be scorable according the preliminary scoring guide. This panel may note changes that needed to be made in the assessment administration process. Following the institution of these changes, the new test should be retested and reviewed by the panel for final approval.

PREPARATION FOR ASSESSMENT ADMINISTRATION

The next important set of activities is getting ready for the actual assessment. This involves selecting the schools that will participate, preparing the schools for participation in the assessment, and training the individuals who will gather the data from students.

Since the assessment materials which have been developed and refined are intended for large-scale assessment use, it is presumed that the exercises will be given to some or all of the students at one or more grade or age level. The original assessment plan will help to determine how and to whom the exercises will be administered. This information will aid in the decision of which schools are selected to participate in the standardized performance-based assessment. Letters of notification should be sent to the school coordinator and to the district coordinator if applicable. These letters should include such details as when the assessment will occur, what will be involved in the assessment, what the schools responsibilities will be, who will administer the test, and so forth.

A major key to the success of the entire performance assessment project is the quality of the individuals who are chosen to conduct the assessment. After the selection process for new assessment administrators has taken place, training should occur. When the assessment administrators are comfortable with the administration of the tests and the manipulatives and equipment that are used in them, the focus of the training is to shift to recording and scoring of responses. Trainees with acceptable scoring prowess will be certified as assessment administrators; others must be either cycled back through more training or dismissed from the assessment project.

ASSESSMENT ADMINISTRATION

Once the assessment administrators are trained, they also should contact the appropriate assessment coordinators. The purpose of this contact is to schedule the date(s) and times for the assessment administration to occur, to remind the school coordinator to have a complete listing of students at the appropriate grade level(s) available on assessment day, and specify what facilities and equipment will be needed.

After all contacts have been made, the field assessment administrators can begin the process of assessment administration. This will start with the drawing of the sample of students to be assessed and a list of alternates. It may be helpful to have the school assign an aide or a student who can work with the assessment administrator to locate the students when needed for the assessment and bring them to the assessment administration site.

It will also be most helpful if the designated contact person and others associated with the project but not involved directly in the assessment administration would select some schools in which to observe one or more students taking part in the performance assessment. In addition to observing the assessment administration, these people can also discuss the assessment with some of the students following the assessment. This can provide valuable insight on why and how students responded and their motivation and interest in the assessment.

POST-ASSESSMENT ADMINISTRATION ACTIVITIES

Various student responses are selected by an expert in the area to represent the different types of responses that students may have given. Such expert judgments are next confirmed by an expert panel of judges; some of the sample will appear in the scoring guide prescored and will be used to train the scorers. The others will be used to judge the accuracy of the scorers following the initial training.

The scoring process requires a number of things to be prearranged for the scoring to flow smoothly. First, a determination would have to be made about whether there will be one or two scorers for each response. Second, arrangements will need to be made to distribute booklets and other materials to be scored, plus rating sheets, to each of the scorers. Third, routine reliability checks should be built into the scoring process.

The various scores need to be summarized and prepared for reporting. The summaries will be most efficient if the individual(s) who will be doing the reporting and those people directing the scoring discuss the needed and desired data summarization process before the scoring is conducted. One thing to keep in mind when reporting the data is that typical audiences are most interested in knowing how well students performed, why students performed as they did, if the experts were surprised in any way by the level or types of student performances, and what the experts believe needs to be done to help students improve.

CONCLUSION

While performance assessment may be new to some people, it is not new nor is it untried. National and statewide performance assessments were successfully conducted in a reliable and cost-efficient manner decades ago. As this guide has illustrated, performance assessment is feasible and manageable. Such assessments are vitally needed in the assessment landscape so that those interested in assessing what students are capable of doing have access to more complete information on student performance. Although the steps are more complex and more involved, such assessments are important in the determination of what skills our students need to have and whether or not they do in fact have them. Performance assessment is an important adjunct to overall large-scale assessment strategies.

REFERENCES AND ADDITIONAL READING

Roeber, E. (1995). Guidelines for the management of performance assessments in large-scale assessment programs. Oak Brook, IL: North Central Regional Educational Laboratories.

Fairtest, b. (1993). Standardized tests and our children: A guide to testing reform. Cambridge, MA: Author.

Neill, M., Bursh, P., Schaeffer, B., Thall, C., Yohe, M., & Zappardino, P.(1994). Implementing Performance Assessments: A guide to classroom, school and system reform. Cambridge,MA: Fairtest.

Rudner, L.M. & Boston, C. (1994). Performance Based Assessment. ERIC Review, 3(1), 2-12.