ERIC Identifier: ED458215
Publication Date: 2001-05-00
Author: Tienken, Christopher - Wilson, Michael
ERIC Clearinghouse on Assessment and Evaluation College Park MD.
State Standards, Assessments and Instruction. ERIC Digest.
Most states have put in place curriculum standards and state-developed
assessments to monitor the implementation of those standards. Most state
standards define expected outcomes, that is, what students need to know and be
able to do, but do not mandate specific strategies or pedagogy used by local
districts. As researchers Paul Black and Dylan William (1998) note, standards
are raised only by changing what happens in the classroom, beginning with
teachers and students. This Digest describes a program used by two educators to
help teachers improve instruction through a deeper understanding of state
standards and test specifications.
DEVELOPING A KNOWLEDGE BASE
standards-based state tests are constructed is the first step in being able to
use them to guide and improve instruction. A test is essentially a sample of
questions or activities that reflect a large body of knowledge and mental
processes associated with an academic subject area such as language arts,
mathematics, science, etc. School districts are under pressure to perform well
on state tests and often use a test preparation strategy of giving students
sample tests from commercially prepared workbooks or state-released items to get
ready for state tests. Although this strategy can be useful for providing
general information regarding student strengths and weaknesses as related to the
samples, it should not be the only method used by teachers because it does
little to educate them about how to understand and use state tests, standards,
and test specifications. This article recommends a three-part process of
delineation, alignment, and calibration for developing an understanding of state
assessments and using that understanding to improve instruction. The New Jersey
Core Curriculum Content Standards are used to model the process, but it can be
applied to any state's or district's standards.
Delineation is the process of identifying all
aspects or dimensions of a particular subject domain. It involves using state
testing documents that describe each content area of the assessment, including
test specifications, specific skill cluster information, subject area
frameworks, assessment examples and exemplars, and the state standards.
Delineation requires an examination of these documents for assessment dimensions
such as content, cognitive level and complexity. A thorough delineation might
also include analysis of additional factors such as the test format and
difficulty level of the questions.
The New Jersey Standards include macro, or big picture, statements and
cumulative progress indicators that provide details about general performance
expectations. The standards include knowledge specifications, which describe the
specific processes and content that all students must know by the end of fourth
grade (also known as content standards) as well as problem-solving
specifications, which describe what students should be able to do with the
content knowledge (also known as process standards). The following example is
excerpted from the 4th grade New Jersey mathematics standards and test
Macro Standard 4.1: All students will develop the ability to pose and solve
mathematical problems in mathematics, other disciplines, and everyday
Cumulative Progress Indicator 4.1.2: Recognize, formulate, and solve problems
arising from mathematical situations and everyday experiences. Test
Specification Manual - Cluster IV Discrete Mathematics: Knowledge (content
standards): Students should have a conceptual understanding of: Tree diagram
Problem Solving (process standards): In problem solving settings, students
should be able to: Draw and interpret networks and tree diagrams
After reviewing the entire body of standards and test specifications for
fourth-grade mathematics in New Jersey, a teacher would be able to identify
seven distinct mathematics strands or dimensions: Numeration and Number Theory,
Whole Number Operations, Fractions and Decimals, Measurement/Time/Money,
Geometry, Probability/ Statistics, and Pre-algebra. The test specifications for
that exam imply that the mathematics test questions are primarily composed of
problem-solving tasks. Therefore, it is safe to assume that test questions will
require thinking in the application, analysis, and perhaps synthesis and
evaluation levels of cognition.
Once teachers and administrators specify all of the subject area dimensions,
the following activities can begin:
Selecting and designing classroom assessments and practice questions.
Revising and designing curriculum that is congruent with the content identified
in the state standards and the district's delineation of the state-designed
Designing teacher training using instructional techniques that support these
During the alignment phase, administrators and
teachers work to identify, analyze, generalize, and describe the links between
the various elements associated with the subject area previously delineated and
the sample questions selected for practice or classroom activities to assess
student progress. The sample questions and student assessments can be derived
from several sources, including state-released test items, commercially
manufactured test preparation materials, or teacher-made activities. Teachers
and administrators examine linkages in the materials, organization, textbooks,
instructional strategies and other elements described in the curricula and used
in daily instructional activities to ensure consistency with the district's
delineation of the state assessment.
Using and understanding the test specifications become even more important at
this stage. Returning to the example above, a teacher seeking to ensure that
students would be able to understand tree diagrams and solve problems using them
would complete several alignment tasks:
1. Review classroom resources, curriculum, textbooks, teacher activities,
student thinking strategies, and tests to ensure that the test specifications
and macro standards are addressed on the knowledge and problem solving level. Do
the teacher resource materials and classroom instruction address the proper
2. Review the above factors to ensure congruency between the level o
difficulty required by the standards and specifications, and the difficulty of
the actual teacher resources and activities. Do the teacher's tests, lessons,
and activities match the difficulty level required by the standards and
3. Consider format. Although less important than skills and difficulty, the
teacher resources, activities, and tests should familiarize the students with
state test question formats. Teachers must align classroom assignments and
activities to the subject area delineation to ensure congruency.
After completing the delineation and beginning
the alignment processes, calibration begins. Calibration is the act of
conducting communications and interactions with teaching staff based on the
information identified in delineation and used in alignment. The calibration
process ensures that the conceptualization of content, cognitive process,
complexity, formats, etc., is consistently understood for each subject area.
Calibration, in its simplest form, is designing classroom instruction,
activities, and assessments that are congruent with content area delineation and
alignment. Using the mathematics vignette as an example, one can begin to see
how the process takes place.
Figure 1. Represents the sequence of events leading up to Calibration.
1. Teacher performs content area delineation and chooses a unit of focus. In
this case Standard 4.1, Cluster IV-Discreet Mathematics-tree diagrams
2. Teacher examines a compares classroom resources, local curriculum,
activities, skills level of difficulty, format, and tests to the Standards and
test specifications to ensure congruency.
3. Teacher uses and implements calibrated activities, tasks, resources and
4. Teacher designs lessons and activities, gathers resources, and creates
tests that are congruent with the skills and level of difficulty for the
Standards, test specifications and curriculum.
Figure 1. Delineation, Alignment, and Calibration Flow of Events
Matt has four channels on his television. He has channels 2,3,4 and 5. If
Matt watches only two channels each night, haw many different combinations of
channels can he watch? Show all your work, and explain your answer. Matt can
watch (2,3), (2,4), (2,5) or (3,2), (3,4), (3,5) etc.
A 4th grade teacher completing delineation and alignment and discovering that
her/his program was missing a unit on discrete mathematics would develop
objectives related to understanding, using, and interpreting tree diagrams.
Figure 2 is a sample activity/test question created by 4th grade teacher Terry
Maher to begin addressing the aspect of discrete math noted in the Cluster IV
Calibration is any action that helps teachers design activities and construct
assessments based on the dimensions of state assessments and standards. This
process helps to foster a collective understanding and agreement of the
dimensions and domains of each content area. It should be a team effort based on
USING SCORE REPORTS TO IMPROVE CALIBRATION
As teachers gain
a better understanding of how student work reflects the standards and test
specifications through delineation, alignment, and calibration, their efficiency
and accuracy at identifying which students are meeting the standards should
increase. Herein lies the usefulness of score reports sorting students into
categories of varying proficiency. A student who scores partially proficient,
proficient, or advanced proficient on a state language arts test may also show
some congruency in the level of achievement in his/her well-aligned school work
and classroom assessments. As teachers become better calibrated, they will be
able to answer questions such as: What level of proficiency does this student
show on class assessments? Is the difficulty level of the class work comparable
to the state exam? What can I do to help this student meet the state standards?
Is my program meeting the standards?
Teachers can reflect upon their level
of calibration accuracy by attempting to predict student results on state
assessments. Teachers should be aware, however, that 100% agreement should not
be expected between a student's performance on well-calibrated classroom tests
and on the state assessments, based on many factors of test design.
To begin the prediction process, the teacher uses a list of the students
taking the test. Beside each name, the teacher enters a predicted score level.
When the state assessment scores arrive, the teacher can compute the level of
accuracy as shown below.
Name Prediction Score
Allan Proficient Adv. Proficient
Ann Proficient Proficient
Tamika Adv. Proficient Proficient
Bronson Partial Proficient Partial Proficient
list above shows a 50% level of success in the predictions made.
The teacher making the predictions returns to each student's work and
compares the successful predictions with the unsuccessful ones to gain a better
idea of how the assessment performances reflect the aligned student work.
Student work associated with actual test scores can form the basis for
subsequent calibration discussions. Student work connected to state assessment
score levels can also function as scoring examples that students refer to when
judging their own work.
There is a distinct difference between
traditional notions of test preparation and aligning and calibrating instruction
and assessments with the content, cognition, difficulty, and format of state
assessment instruments, specifications, and standards. As educators, we are
trying to link the classroom activities to the standards and skills set by the
state. The aim is to ensure that teachers understand, and calibrate their
classrooms with respect to, the entire process and do not simply focus on how to
answer specific types of test questions. The questions will change, but the
underlying skills and concepts will not. One must be careful not to wallow in
the mire of test prep.
Delineation, alignment, and calibration are academic endeavors that demand
unending commitment. Do not expect to accomplish alignment or calibration at an
in-service day, or even during the course of a school year. The administration
must provide the time and resources to conduct frequent calibration meetings to
examine such things as classroom work and student assessment samples. Beware, it
is easy to fall out of alignment and calibration and into test prep.
Black, Paul, and Wiliam, D. ( 1998 ). Inside the
black box: Raising standards through classroom assessment, Phi Delta Kappan
October, pp. 139-148.