ERIC Identifier: ED435708
Publication Date: 1999-09-00
Author: Quellmalz, Edys - Schank, Patricia - Hinojosa, Thomas -
Source: ERIC Clearinghouse on Assessment and
Evaluation Washington DC.
Performance Assessment Links in Science (PALS). ERIC/AE Digest.
Understanding the central role that performance assessment plays in
standards-based reform, educators are seeking ways to use these assessments to
test student learning. Education agencies need pools of performance tasks to use
in their student assessment programs and in evaluations of state and federally
funded programs. Reform projects need standards-based assessment, too, as do
teachers who are trying to implement reforms. Experience indicates, however,
that the level of effort and costs of developing performance assessment tasks
are very high (Quellmalz, 1984).
To meet the need for innovative approaches for sharing exemplary assessment
resources, to facilitate the development of new ones, and to further understand
how the use of standards-based performance assessment can advance educational
reform, SRI International is developing Performance Assessment Links in Science
(PALS), an on-line, standards-based, interactive resource bank of science
performance assessments. Coupled with the development of the resource bank is a
program of research on effective use of these resources.
This digest describes work-in-progress. SRI is currently seeking
organizations to participate in implementation studies involving the use of
PALS. The "ideal" professional development model would be for an organization or
group of schools to want to focus on classroom applications of science
performance assessment and to use the PALS resources in an initial professional
development institute, followed by several school-year meetings where teachers
would bring samples of student work produced in response to PALS investigations,
with subsequent work on designing additional classroom science assessments. For
the accountability studies, SRI will work with a few assessment programs from
states, districts, or specially funded programs that are interested in using the
on-line, secure tasks in the Accountability Pool. Assessment programs might use
PALS to search, select, and plan assessment administrations and/or conduct
on-line rater training and scoring sessions.
PERFORMANCE ASSESSMENT IN SCHOOL REFORM
programs have set out to develop alternative forms of student assessment that
call for students to construct rather than select responses. Performance
assessments are generally valued for testing students' deep understanding of
concepts and inquiry strategies, for making students' thinking visible, and for
measuring skills in communicating about their science knowledge. In addition,
performance assessments can present authentic, real-world problems that help
students to show they can apply academic knowledge to practical situations. On
the other hand, performance assessments are time-consuming and costly to
develop, logistically demanding, and of questionable utility if not developed
and scored according to sound measurement methods.
TECHNOLOGY IN ASSESSMENT REFORM
Technology offers a
powerful strategy for increasing the ease with which educators can access and
use standards-based student assessment. Technology can be used to efficiently
archive numerous assessments for ready browsing. Currently, some sets of
performance assessment tasks are distributed on CD-ROM, and there are plans to
place released test items on the Internet, although these resources are not yet
coordinated or easily accessible to programs and schools. Electronic networks
can go beyond storage by supporting the growth of a community of colleagues and
leveraging expertise, regardless of geographic location and institutional base.
Networks can offer templates and guidelines and support collaborative
development and on-line conversations about the alignment of tasks with
standards, quality of tasks and student work, on-line training, scoring, and
standards-setting sessions. Technology can also support simulated investigations
and data collection and analysis of student responses. The exponential advances
taking place in technology promise to revolutionize assessment practices and
education reform (Kozma &Schank, 1998; Quellmalz, 1999).
TECHNOLOGY IN PROFESSIONAL DEVELOPMENT
development models are needed to provide teachers with greater opportunity to
access, discuss, incorporate, and co-construct assessment resources and other
reform-based materials(Little, 1993).Current efforts find it difficult to reach
many teachers and to maintain discourse, and teachers have difficulty
implementing new ideas back at their schools. Technology can help provide
mechanisms for teachers to overcome their isolation and make more effective use
of their time spent on professional growth.
PERFORMANCE ASSESSMENT LINKS IN SCIENCE(PALS)
an on-line assessment resource library designed to serve two purposes and user
groups:(1) the accountability requirements os state education agencies and
specially funded programs, and (2) the professional development needs of
classroom teachers. SRI is developing PALS under a grant from the Instructional
Materials Development (IMD) program within the National Science Foundation with
two primary goals:
(1) To develop a two-tiered on-line performance assessment resource library
composed of performance assessment tasks for elementary, middle, and secondary
levels. One tier will be a password-protected, secure Accountability Pool of
science performance assessments for use by state assessment programs and
systemic reforms. The second tier is for use by teachers and professional
development organizations. The Professional Development Pool provides
performance-based science assessments that have been used successfully in
large-scale (state or national)assessment programs and have been released.
(2) To evaluate the effectiveness of policies, implementation models, and
technical quality requirements for the use of the two tiers of PALS.
In our design for PALS, experienced assessment programs can contribute
standards-based science assessment tasks with documented technical quality to
the PALS on-line library. The Accountability Pool will be composed of
password-protected, secure tasks accessible only to approved assessment program
staff. Assessment programs can thus share their resources and enjoy a much
larger pool of science performance assessments to use or adapt for their testing
administrations. The PALS resource library can provide large, continually
updated collections that can support efficient searching, selection, and
The Professional Development Pool contains resources that are of documented
technical quality and have been released for access by teachers and professional
development groups. Pre-service and in-service programs, for example, can reach
teachers in geographically distributed and remote locales, resulting in great
savings of travel and materials expenses. On-line guidelines and templates can
support classroom use of science performance assessments. Teachers may
administer the performance tasks as part of their classroom assessments, adapt
them, or use them as models for developing similar investigations.
To help users design test administration forms that cover important science
standards, the on-line system provides assessment planning charts
(Stiggins,1994).PALS automatically produces an assessment planning chart to
display coverage of standards by the performance tasks selected by the user.
The science performance assessment library includes the scoring rubrics
designed to judge the quality of student responses to a task or set of tasks. To
bring meaning to the scoring rubrics, a library of exemplars of scored student
work is included.
Rater training materials are not routinely published by assessment programs.
SRI has developed specifications for on-line rater training and scoring so that
each agency wishing to take advantage of the PALS system can convert
traditional, stand-up training procedures and calibration to written form,
assemble training packets, and test their effectiveness in on-line delivery.
An essential component of PALS is the documentation of the technical quality
of each task in the library. PALS contains the technical quality indicators
provided for the field-tested science tasks. Since the resource library will be
stocked only with tasks that have survived a systematic development process, the
tasks and rubrics will also have been subjected to content validity and bias
ISSUES AND EXPANSION OF THE SYSTEM
The project will be
addressing a number of issues. One will be the procedures for identifying the
science standards that tasks are designed to test and the nature of the groups
that will make the alignment judgments Another issue relates to policies for
accessing the resources that have been developed with funds from different
sources and that represent assessment materials being distributed by various
organizations. A third issue relates to the criteria and procedures for
reviewing tasks submitted for inclusion in the bank. Finally, the system must
develop strategies for operating, maintaining, and expanding the resources.
PALS uses technology to efficiently archive numerous assessments for ready
browsing by teachers, and goes beyond mere storage of assessments to support
cross-links with standards, tasks, scoring criteria, and annotated student work.
These resources can be shared by assessment programs, allowing them to have
access to a large pool of performance assessments to use or to adapt for their
testing administrations. Teachers can administer the tasks as part of their
classroom assessments, adapt them or use them as models for developing their own
investigations, and contribute their adaptations to the resource bank for other
teachers to use.
As an on-line library alone, PALS is a valuable set of resources. However,
the growing body of professional development research and the results of our
pilot study suggest many benefits of integrating more collaboration support into
PALS. SRI plans to integrate an online meeting/discussion component to help the
community members leverage expertise, regardless of geographic location. This
component could be used to support collaborative development and on-line
conversations about the alignment of tasks with standards, the quality of the
assessment tasks and student work, rater training and scoring, and standards
setting. The authors believe that, by taking advantage of new models of
professional development that include innovative digital technologies, PALS will
provide excellent professional development opportunities for teachers.
Kozma, R., & Schank, P. (1998). Connecting
with the twenty-first century: Technology in support of education reform. In D.
Palumbo and C. Dede (Eds.), Association for Supervision and Curriculum
Development 1998 yearbook: Learning with technology. Alexandria, Virginia:
Association for Supervision and Curriculum Development.
Lieberman, A.(1996, November).Creating intentional learning communities.
Educational Leadership, 51-55.
Little, J.(1993).Teachers' professional development in a climate of
educational reform. Educational Evaluation and Policy Analysis, 15(2), 129-151.
Quellmalz, E. S. (1984).Designing writing assessments: Balancing fairness,
utility, and cost. Educational Evaluation and Policy Analysis, 6, 63-72.
Quellmalz, E.S. (1999).The role of technology in advancing performance
standards in science and mathematics learning. In Kathleen Comfort (Ed.),
Advancing standards for science and mathematics learning: Views from the field.
Washington, DC: AAAS.
Stiggins, R. J. (1994).Student-centered classroom assessment. New York:
Macmillan College Publishing Company.