ERIC Identifier: ED446325
Publication Date: 2000-02-00
Author: Wall, Janet E.
Source: ERIC Clearinghouse on
Counseling and Student Services Greensboro NC.
Technology-Delivered Assessment: Diamonds or Rocks? ERIC/CASS
There can be no doubt that technology has had a major impact on our daily
lives. We read the news online, we shop online, we communicate with each other
online, we are entertained on line, and we learn online. Education has been a
prime beneficiary of technology's power. More than 95 percent of all public
schools are equipped with computers, with a majority having Internet access
(Market Data Retrieval, 1999). Distance learning is becoming more and more
commonplace, not only in laboratories with computers connected to the Internet,
but also on our desktops at home and work. We can find information on nearly
everything at the stroke of the keyboard and listen and see events of historical
proportion that take place thousands of miles from our home.
Educators have long understood the potential of technology in supplementing
the instructional process, including the use of technology in the area of
assessment. Technology-delivered assessment is being implemented in a variety of
settings and grade levels for a variety of uses in education, business,
government, and the private sector (Sistek-Chandler, 2000). Under the right
conditions and with proper use, a marriage of technology and assessment can
shine like a diamond. Used without proper care or forethought, however, it can
be a rock.
DIAMONDS OF ASSESSMENT VIA TECHNOLOGY
Technology used with
good testing practices offers some capabilities that can add value to the
assessment process. This includes:
* accessibility: Individuals can take various tests by computer or on-line
for many purposes including college entrance, course placement, certifications
and licensure, career decision-making, academic preparation, military selection
and classification, and personality assessment. As a result, individuals can
have access to information that they may not have had before. Bill Gates, in his
book The Road Ahead (1995), suggested that individuals who are "wired" have
access to the same information, resulting in a virtual equity more achievable
than real-world equity.
* immediate feedback: The potential for immediate test scoring and feedback
is a key advantage of technology-delivered assessment which can be a significant
motivator for persons taking assessment instruments. Individuals can learn their
status on assessments quickly and use that information to make decision or take
* more efficient testing: The use of computer-adaptive testing, as opposed to
computer-administered testing, allows persons to take tests that are targeted
more accurately to their ability levels (Heubert & Hauser, 1999). Use of
technology in combination with the increasingly popular item response theory can
determine an individual's performance level using fewer questions. Under certain
conditions, significant time and money can be saved by using computer-adaptive
testing, particularly in large-scale assessment situations.
*judging portfolio information: The capability of placing ones work,
experiences and/or educational history on a floppy disk or on a website and
making that information available for review and judgment is empowered by
technology. Writing samples, art work, letters of recommendation, journals, test
results, transcripts, certificates and certifications, descriptions of community
service, listing of club memberships and activities, and special project work
can be saved electronically, transported easily (either physically or
electronically), and made readily available for evaluation by others.
* ability to assess higher order skills: Use of technology permits test
developers to use techniques and create situations that are difficult or
impractical to construct in regular paper/pencil assessments. Consequently, and
with some creativity through the use multi-media, interactivity, simulations,
etc., assessments can reflect more authentic conditions. Crafted carefully, the
assessments may provide the capability of tapping into higher order cognitive
skills than cannot be easily tested with a standard paper/pencil instrument
(Chung & Baker, 1997). Test developers can construct situations that
simulate the real world. For instance, test items on computer can be designed to
simulate events in biology, economics, architectural design, structural
engineering, etc., asking students to exhibit skills under real or more
realistic conditions. Language assessments can use audio and video to simulate
various situations that could be encountered in a foreign country.
* helping persons with disabilities: Use of assistive technology in
assessment can be significant in helping persons with disabilities. Text readers
can help persons with visual impairment gain access to testing situations.
Physically challenged persons can take advantage of voice recognition technology
to answer test items and even to dictate long responses to essay questions.
Those with difficulty controlling fine motor skills can use a touch screen or
smart board to respond to assessment items. Persons who are housebound or not
able to travel to a test site can take a test over the Internet from their home.
Technologies are being created that can respond to slight movements of the head
or eyes. Not only does technology provide testing accommodations, but access to
testing opportunities that were not available before.
ROCKS OF ASSESSMENT VIA TECHNOLOGY
Despite the positive
elements described above, there are potential problems that accompany the use of
technology in assessment. These include:
* lack of accessibility: The Department of Commerce has shown in its most
recent study that access to computers and the Internet is highly dependent on
income, racial and ethnic group, and urbanicity (U.S. Department of Commerce,
1999). Other studies indicate that access varies greatly by income, age,
education, and technology optimism. (Forrester Research, 2000). In addition to
basic computer access, persons may not be able to afford the cost of the tests
* test security problems: Not unlike paper/pencil instruments, those
available in electronic formats are vulnerable to compromise. Compromised tests,
can result in unfair advantage to test takers who might obtain the questions
prior to taking an exam or be able to find answers to questions on the Internet.
Solutions range from securing removable hard drives to tracking usage, and
blocking access to certain Internet addresses.
* concerns about test taker identity: Users of test results need to be sure
that the person taking the assessment has accurately represented his or her
identity. Special care must be taken to ensure that the person completing a
licensure test for credentialing, for example, is the person that is seeking
this certification. Solutions can range from desktop video teleconferencing to
fingerprint and retinal recognition systems.
issues of privacy/confidentiality: As with paper/pencil assessments,
information about an individual's technology-delivered scores and results should
be kept confidential and be made accessible only to those individuals who have a
rightful need to know. This is particularly critical for sensitive assessments
that are answered over the Internet
* lack of information on the quality of the instrument: It can be mistakenly
assumed that any test made available on computer or via the Internet meets
professional testing standards. This can be a risky assumption. It is quite
possible for the instrument to be deficient in the requisite technical
information necessary to show the degree of quality and suitability of the test.
In fact, many assessments have been posted for easy access via the Internet
where little or no technical information has been made available to the
* problems with test comparability: If an instrument is available in both
paper/pencil and computer-delivered format, it can be mistakenly assumed that
test developers have produced parallel forms that provide the same scores
regardless of administration format. It is not unusual for some high-quality
tests, offered in the paper/pencil format, to result in different scores when
those same items are administered via computer or Internet. Without assurance of
comparability, a test taker may have an advantage by using one format rather
than the other. Disparate results from noncomparable assessments can transpire
for any number of reasons including speededness, point size of the words,
monitor resolution, use of color, physical arrangement of the items, inability
to change completed items, comfort with the equipment, and response mode.
* gender, racial, and ethnic disparities: That females, persons of color, or
individuals of different ethnic backgrounds may be disadvantaged in certain
testing situations has been a long-standing concern in paper/pencil testing.
This can be exacerbated with tests delivered via computer or the Internet. If a
particular group has disproportionate access to computers and technology,
disparity could be created by the medium alone.
* reporting and interpretation: Immediate feedback on tests is clearly
desirable. Without appropriate interpretation, however, there is danger that the
test taker will take actions that are not warranted by the test results. The
potential exists for interpretations to appear to be so definitive that test
takers fail to understand the need for caution due to the lack of accuracy of
the scores. Also, feedback can be so extensive that the test results can
paralyze a person's actions.
* lack of human contact: With technology-delivered assessments, meaningful
human contact and intervention to assist with test score interpretation may be
lacking or unavailable. A skilled counselor can help a test taker to sort out
his or her results and use them in the context of other knowledge and
*issues of familiarity with technology: There are some studies that show that
students perform better on tests that use the same medium in which they were
taught. For example, if students use the computer to write essays and papers,
they tend to score higher on writing assessments that allow them to use that
technology to take writing assessments and more poorly if that technology is not
available for testing purposes. Therefore, students who learn on computers may
have an advantage on computerized assessments.
FOLLOWING ASSESSMENT STANDARDS AND PRACTICES
counselors and educators sort the diamonds from the rocks? They need to be aware
of the various issues that relate to the construction, production,
administration, and interpretation of tests delivered via the computer or
Internet. No compromise should be made on the quality of a test administered to
a client or student with either traditional or technology-delivered assessments.
Various agencies and organizations have produced policy statements and standards
for testing that are applicable to both paper/pencil and technology-delivered
assessment including the following:
American Educational Research Association, American Psychological Association,
and National Council on Measurements in Education. (1999). Standards for
Educational and Psychological Testing. Washington, DC: American Educational
American Association for Counseling and Development. (1989). Responsibilities of
Users of Standardized Tests. Alexandria, VA: Author.
American School Counselors Association and Association for Assessment in
Counseling. (1998). Competencies in Assessment and Evaluation for School
Counselors. Alexandria, VA: Author.
Committee on Professional Standards and Committee on Psychological Tests and
Assessment. (1985). Guidelines for Computer-Based Tests and Interpretations.
Washington, DC: American Psychological Association.
Dahir, C. A., Shelton, C. B., and Valiga, M. J. (1998). Vision Into Action:
Implementing the National Standards for School Counseling Programs. Alexandria,
VA: American School Counselor Association.
Joint Committee on Testing Practices. (1988). Code of Fair Testing Practices in
Education. Washington DC: National Council on Measurement in Education. This
document is being reviewed and should be revised for publication in 2001.
Joint Committee on Testing Practices. (2000). Rights and Responsibilities of
Test Takers: Guidelines and Expectations.
National Board for Certified Counselors. (1998). Standards for the Ethical
Practice of Web Counseling. Greensboro, NC: Author.
National Career Development Association. (1997). NCDA Guidelines for the Use of
the Internet for the Provision of Career Information and Planning Services.
Alexandria, VA: Author.
U.S. Department of Labor Employment and Training Administration. (1999). Testing
and Assessment: An Employer's Guide to Good Practices. Washington, DC: Author.
Chung, G.K.W.K & Baker, E.L. (1997). Year 1
Technology Studies: Implications for Technology in Assessment, CSE Technical
Report 459, National Center for Research on Evaluation, Standards, and Student
Testing (CRESST). University of California, Los Angeles.
Forrester Research. (2000) The Truth About the Digital Divide. Forrester.com.
Gates, W. (1995). The Road Ahead. New York: Penguin.
Heubert, J. P., & Hauser, R. M. (Eds.). (1999). High Stakes: Testing for
Tracking, Promotion, and Graduation. Washington, DC: National Academy Press.
Market Data Retrieval. (1999). Technology in Education, 1999. Shelton, CT:
Sistek-Chandler, C., Online Assessment: Changing the Way You Test, Converge,
U.S. Department of Commerce. (1999). Falling Through the Net: Defining the
Digital Divide. Washington, ED: Author.