ERIC Identifier: ED446327
Publication Date: 2000-04-00
Author: Wall, Janet E.
Source: ERIC Clearinghouse on
Counseling and Student Services Greensboro NC.
Technology-Delivered Assessment: Guidelines for Educators
Traveling the Technology Highway. ERIC/CASS Digest.
The number of computers in schools for instructional use has increased
dramatically in recent years to nearly nine million nationwide (Anderson and
Ronnkvist, 1999). The ratio of students to computers is around 5 to 1. Internet
connectivity continues to be a powerful force in education with technology being
integrated more fully into the educational process (Market Data Retrieval,
1999). Though the statistics differ somewhat by grade, the vast majority (80-88
percent) of students report that they use computers at home or school to write
short stories or papers and to learn things (U.S. Department of Education,
1998). Educators are using technology for instructional purposes including
assessment and it is clear that the testing process can be enhanced through the
use of technology (Sampson, 2000).
Educators use assessment for a variety of reasons that range from individual
student assessment to program evaluation to system-wide educational
accountability (Wiggins, 1993). Good assessment provides objective information
that can assist in diagnosing problems and identifying curricular areas that
need improvement (Anastasi, 1988). Assessments can help an individual find his
or her way in the decision-making journey or help a school system travel the
road to educational excellence. To navigate the journey accurately and safely,
certain guidelines or rules of the road should be followed.
This paper highlights the rules or best practices that should be followed by
educators in evaluating, selecting, and using technology-delivered assessments.
The guidelines adhere to existing professional standards such as the Standards
for Educational and Psychological Testing (1999), Responsibilities of Users of
Standardized Tests (1985) and the Code of Fair Testing Practices in Education
(1988). The guidelines are presented in checklist form so that educators can use
it to perform an initial evaluation as to whether a technology-delivered
assessment is of sufficient quality that it can be accepted for a particular
educational use. Adherence to these guidelines is of vital importance in
reviewing, selecting, and using technology-delivered assessment instruments.
TECHNOLOGY-DELIVERED ASSESSMENT CHECKLIST
The test content has been described by the developer, reviewed by the user, and
matches the purpose of the testing. Assessment items are aligned with the
desired areas intended for assessment.
Clear and supportable statements are provided by the test developer about what
the test is intended to measure.
Evidence is provided by the test developer and has been reviewed by the test
user to determine if the assessment is appropriate for the test taker with
regard to age, membership in a subgroup, educational level, disability, and
There is ample evidence of comparability in the scores between the paper/pencil
and technology delivered versions of the same test; where necessary, appropriate
adjustments are made to scores to better assure comparability.
Evidence is provided to indicate the conditions under which the test results
have been found reliable. The strength of that reliability has been reported by
the test developers and examined by the individual who is considering using the
Validity evidence is provided by the test developer to substantiate suggested
uses and interpretations. Test users apply and interpret the test results for
purposes consistent with the validity evidence.
Limitations of the test and test results are clearly specified by the test
developer and examined by the test users prior to selection of the testing
instrument. No assessment is without limitations.
Published reviews of the assessments crafted by qualified persons have been
reviewed prior to test selection and use.
Test items and technical information have been examined to determine their
currency. Careful examination of the age of the test and its technical
information are considered before justifying the use of antiquated data or test
TEXT SCORE REPORTING AND INTERPRETATION
* Tests are
accurately scored and results have been returned as quickly as possible,
together with test interpretations that are comprehensive and accurate, given
the limitations of the test.
* Assessment results are not over- or under-interpreted. Care has been taken
not to develop interpretations or explanations that go beyond that which can be
supported by the reliability and validity evidence.
* Test reports and supporting manuals specify appropriate and inappropriate
uses of the assessment results.
* Reports include an indication of the degree of accuracy of the scores.
* Score reports specify which interpretations are supported by research and
which interpretations are based on professional opinion. Sufficient information
has been offered to allow the test taker to weigh the credibility of the
* Interpretations of test scores include the limitations of the test and test
results, with reference to common misinterpretations of the assessment results.
* If scores are used in high-stakes situations such as graduation, promotion,
college entrance, placement, and credentialing, an explanation on how the
passing or cutoff scores were set is available to the test user.
* The score and interpretation report reveal where further information can be
obtained about the test and score interpretation, and how a test taker can
verify or challenge the accuracy of the score.
* When interpreting a test score, relevant aspects of other information about
the individual, such as previous test scores, educational level, and other
performance indicators are used to augment the interpretation. This information
is used to help the test taker gain further insight into the meaning of his/her
* The test setting is comfortable,
quiet, and conducive to allowing the test taker to maximize performance. The
arrangement of the computers ensures privacy and comfort.
* Testing equipment is in good working order and the software and/or Internet
connectivity has been verified to be operating properly.
* A site administrator is available to troubleshoot problems that may occur
due to equipment, software, or other technology failures.
* Policies and procedures have been established, explained to the test taker,
and applied in cases of technology failure. For example, if there is a computer
crash or a power disruption, are the responses to the test items saved, or does
the test taker need to begin the task again?
* Test takers are comfortable with the test format and use of keyboard or
other equipment. If there is a question about the test takers' familiarity with
the technology, practice exercises or tutorials have been provided to enable
them to become facile with the equipment and the format, allowing the test taker
to focus on the assessment rather than the mode of delivery.
* Test items and answers are protected from compromise. Security of the
equipment and test items is critical to the fairness to test takers of current
and future test administrations.
* The identity of the test taker is verified, particularly in high-stakes
* Tests are administered according to the procedures specified by the test
developer, particularly in cases where standardization is important.
* Both test users and test takers are informed about on whether individual
score information is stored and, if so, where, and for how long. Periodic purges
of individual test results stored locally or centrally may be advantageous in
maintaining privacy. It may be more desirable for an individual to save test
results on a personal disk rather than on a server or local computer.
CREDIBILITY OF THE DEVELOPERS
* Information is available
for test users and test takers to determine the identity and professional
credibility of the test developer. A determination has been made of adequacy of
the test developer qualifications. This may be most important when assessments
are published by small companies or by individuals without a relevant history of
* Test developers provide information indicating whether they subscribe to
precepts of the various testing standards produced by professional
organizations. Professional test developers who pledge adherence to various
testing standards do attempt to produce high-quality assessments with sufficient
technical and research support.
ACCESS TO PROFESSIONALS
* Because it is unlikely that the
technology-delivered assessment programs will have information on the background
and experiences of the test taker, a professional educator or counselor is
available to provide value-added interpretations to the assessment information.
* Human judgment is provided where assessment results suggest specific
actions on the part of the test taker or the specification of interventions to
modify a situation and will never be replaced by technology alone.
STAYING CURRENT, KEEPING UPDATED
The speed of change in the
availability and use of technology in education continues to accelerate. Because
of these changes, it is important for professionals to keep tabs on the latest
developments in the assessment and technology areas. Information on web sites
frequently is expanded and updated faster than information in print format. For
these reasons, the following web sites should be checked periodically to
identify new and important information related to assessment and the use of
American Counseling Association (http://www.counseling.org)
American Educational Research Association (http://www.aera.net)
American Psychological Association in Science Directorate
American Speech Language Hearing Association (http://www.asha.org)
Association for Assessment in Counseling (http://www.aac.ncat.edu)
Association of Test Publishers(ATP) (http:/www.testpublishers.org)
Consortium for Equity in Standards and Testing (CTEST)
Council for Exceptional Children, Directory of Current Projects in Assessment
Education Commission of the States (http://www.ecs.org)
ERIC Clearinghouse on Assessment and Evaluation (http://www.ericae.net)
ERIC Clearinghouse on Counseling and Student Services
(http://www.uncg.edu/edu/ericass) * Joint Committee on Testing Practices
National Assessment of Educational Progress (http://www.nces.ed.gov/naep)
National Assessment of Educational Progress
National Association of School Psychologists (http://www.naspweb.org)
National Association of Test Directors (http://www.natd.org)
National Center on Educational Outcomes (http://www.coled.wmn.edu/nceo)
National Center for Education Statistics (http://www.nces.ed.gov/index.html)
National Center for Research on Evaluation, Standards, and Student Testing
National Council for Measurement in Education (http://www.ncme.org)
THE JOURNEY CONTINUES
Technology provides enabling tools
for educators and counselors in locating information on assessment, organizing
and maintaining test information, building relationships between and among test
results and counseling and guidance interventions, using alternative assessment
techniques, assessing difficult areas and higher order skills, introducing
efficiencies into the assessment situation, and reaching audiences that are not
normally accessible without the availability of certain technologies. Clearly
technology, used properly and creatively, can advance opportunities for
individuals and guide them through a journey where they can flourish.
American Association for Counseling and
Development. (1989). Responsibilities of Users of Standardized Tests.
Alexandria, VA: Author.
American Educational Research Association, American Psychological
Association, and National Council on Measurements in Education. (1999).
Standards for Educational and Psychological Testing. Washington, DC: American
Educational Research Association.
Anastasi, Anne (1988). Psychological Testing. New York, NY: Macmillan
Anderson, R.E. & Ronnkvist, A. (1998). The Presence of Computers in
American Schools, Center for Research on Information Technology and
Organizations, University of California, Irvine and The University of Minnesota,
Joint Committee on Testing Practices. (1988). Code of Fair Testing Practices
in Education. Washington DC: National Council on Measurement in Education.
Market Data Retrieval. (1999). Technology in Education, 1999. Shelton, CT:
Sampson, J.P., Jr., Using the Internet to Enhance Testing in Counseling,
Journal of Counseling and Development, Journal of Counseling and Development,
U.S. Department of Education, National Center for Education Statistics, The
Condition of Education 1998, NCES 1999-011. Washington. DC: U.S. Government
Grant P. (1993). Assessing Student Performance: Exploring the Purpose and Limits
of Testing. San Francisco, CA: Jossey-Bass Publishers.