Reliability, Validity, and Authenticity in American
Indian and Alaska Native Research. ERIC Digest.
by Lomawaima, K. Tsianina - McCarty, Teresa L.
The constructs used by scholars across the physical, natural, and social
sciences to evaluate research quality--as objective, reliable, valid, generalizable,
randomized, accurate, authentic--are not value-free or apolitical. They
all require the application of human judgment, which is inevitably affected
by cultural norms and values (Westmeyer, 1981). Consequently, the use of
these constructs to assess research focused on minority, marginalized populations
requires extraordinary judgment. Such assessments must be based not only
on Western notions of scientific quality but also, in the case of American
Indians and Alaska Natives (AI/ANs), on a separate set of criteria prescribed
in the interest of sovereignty. Sovereignty refers to the inherent, as
well as constitutionally recognized, rights of tribes to self-government,
self-determination, and self-education. Of course no contemporary polity--whether
American Indian tribe or U.S. federal government--exercises unlimited sovereignty:
"In the real world, sovereignty operates within constraints" (Wilkins &
Lomawaima, 2001, p. 5). This Digest addresses these issues as researchers
take on the challenges set forth by the "American Indian and Alaska Native
Education Research Agenda" (Research Agenda Working Group, Strang, &
von Glatz, 2001). The perspectives in this Digest are those of the authors
and do not necessarily represent the positions of the U.S. Department of
DEFINITION OF TERMS
In the sciences, the terms "accuracy," "reliability," and "validity"
are used to make evaluative judgments of measurement instruments (e.g.,
tests and surveys), or of data collection methods. An "accurate" measurement
is one that is precise and exact. Although commonly applied to the mechanical
measurements of the physical sciences, accuracy is also an issue in social
science, including qualitative research, where researchers seek to understand
complex, multiple perceptions of reality. A "reliable" method or instrument
gives consistent results in different applications. Consistency, however,
"says nothing about being right or wrong" (Kelly, 1996, p. 43). The "rightness"
or "truth value" of research is an issue of "validity." Validity is always
subject to human judgment: it asks the question, "Does this device [or
method] measure what it is said [or claims] to measure?" (Westmeyer, 1981,
The concept of validity has been contested and is subject to further
refinement. In experimental research, it is common to distinguish "internal
validity" (whether a proposed cause is actually the reason for an effect)
from "external validity" (whether research results are generalizable across
settings or populations) (Miller & Salkind, 2002). However, some qualitative
researchers reject validity as a criterion for judging research, arguing
that understanding is a more proper concern (Wolcott, 1994). Maxwell (1992)
argues that validity in qualitative research refers not to data or methods
but to accounts. "Descriptive validity" tests the factual accuracy of accounts;
"interpretive validity" asks whether an account represents research participants'
perspectives; "theoretical validity" refers to the explanatory power of
accounts; and "evaluative validity" questions the usefulness or applicability
of accounts. Parallels can be found in Banks' (1997) assessment of special
education needs among AI/AN children. She distinguishes "construct validity"
(the meaningfulness of research constructs or ideas), "treatment validity"
(whether research participants judge its objectives, instruments, and procedures
acceptable), and "ecological validity" (the extent to which research looks
beyond the individual to social-historical contexts).
THE REQUIREMENTS OF SOVEREIGNTY AND SELF-DETERMINATION
Judging research adequacy within the larger social-historical context
of AI/AN education involves answering questions critical to the exercise
of sovereignty: Why do the research? What factors motivate the researcher?
Who has set the research goals? Who has the "disciplinary authority" (Page,
2000, p. 23) to do the research? Who will be involved in conducting the
research? Whom does the research serve? How will it benefit the local community?
Until recently, Native communities were rarely involved in raising these
questions or in formulating, implementing, or evaluating studies. In the
past few decades, however, tribal governments have increasingly taken control
of research site access, instituting protocols and approval processes for
research projects (Lomawaima, 2000; see Romero, 1994, for an excellent
example of research working within protocols). For example, both the Tohono
O'odham and Pascua Yaqui language policies stipulate that tribal authorities
must approve all research about their people. The Yaqui policy lays out
further requirements for copyright of publications growing out of such
research (Zepeda, 1990). Additional sovereignty safeguards include requirements
that tribal representatives be involved in setting research standards and
evaluating projects against those standards. All these measures have grown
out of the recognition that local communities have as great a stake as
outside scholars in establishing what constitutes high-quality, useful
There are viable alternatives to conventional, colonizing research paradigms
(Smith, 1999). As a means for improving post-World War II intergroup relations,
sociologist Kurt Lewin (1946) introduced an action research cycle of problem
identification, fact finding, execution, and evaluation. A decade later,
Corey (1953) and Taba and Noel (1957) operationalized this cycle in education.
Now treated as synonymous with teacher research, action research is a "reemerging
tradition" in which those previously designated as subjects actively participate
in research processes that benefit them directly (Stringer, 1996, p. 7).
Recent examples of teacher/action research in AI/AN education include Lipka,
Mohatt, and the Ciulistet (teacher-leaders) Group's work on incorporating
Yup'ik knowledge into science and math instruction (Lipka et al., 1998);
collaborative research to revitalize Indigenous languages (McCarty, Watahomigie,
& Yamamoto, 1999); and research on alternative literacy assessment
and curriculum reform undertaken by Navajo teacher-researchers (McCarty
& Dick, 2003; McCarty, 2002; for an overview of teacher research methods,
see McCarty, 1997).
Are the quality and authenticity of research guaranteed by the involvement
of tribal members in design and evaluation? Some say it takes even more.
Swisher (1996) argues for the primacy of Native scholars in conducting
research on Native peoples and issues, attributing to them the benefits
of an insider's view and experiences, enhanced passion and commitment,
and the "authority to ask new and different questions" (p. 93). She calls
on non-Native researchers and mentors, whose expertise and high-quality
work she readily acknowledges, to step aside in favor of Native authors.
Many would agree with Swisher's call to increase the authority of, recognition
of, and publication by Native scholars and researchers, but not all are
willing to ask non-Native allies to step aside (see Mihesuah, 1996 for
a range of opinions). These observers suggest that the most productive
research will result from respectful collaboration that does not dichotomize
researchers as Native or non-Native but does make the contributions of
Native colleagues integral to the design and conduct of studies.
Even those who adhere to the "Native-only" requirement run up against
debates over who is "really Indian." Snipp (2000) productively discusses
the many criteria used to identify and assign "Indian-ness" and tribal
identity, and the political and other agendas attached to them. Snipp concludes
that contemporary criteria for tribal membership--however historically
situated and flawed--are the prerogatives of sovereign tribes.
Local evaluations of authenticity, whether dependent on research subject
or research investigator, should be respected in accord with the principle
of self-determination, even as researchers struggle to work out their often
anomalous positions as "insider," "outsider," or some combination of both
(Brayboy & Deyhle, 2000).
POLITICS AND SCHOLARSHIP
There have been many calls over the years for improved research in AI/AN
education. The most recent is Executive Order 13096 (Clinton, 1998), which
mandates the development and implementation of a national research agenda
in Indian education. Consultations and meetings with Native peoples across
the country culminated in the "American Indian and Alaska Native Education
Research Agenda" (Research Agenda Working Group, Strang, & von Glatz,
2001). The agenda identifies high-priority research interests and outcomes
for Native communities and clearly articulates the central roles that Native
scholars, parents, teachers, schools, and communities must play to ensure
high-quality research. The promise embedded in the research agenda, however,
may be threatened if limited notions of what constitutes high-quality research,
using criteria such as those outlined earlier, overtake the powers of Native
peoples to evaluate research in their communities and schools.
Science as practiced by many Western researchers is often invoked as
the answer to educational woes. This view tends to dismiss the power of
all human inquiry--including Indigenous knowledge systems--to observe,
analyze, theorize, and generate creative solutions. Research and development
projects designed within the Western mold--especially those meant to provide
guidance for classroom practice--often rely on standardized scripts to
instruct and standardized tests to assess effectiveness (Metcalf, 2002).
Using standardized tests with minority populations is a dubious practice
(Padilla & Lindholm, 1995, p. 97).
A much more promising line of research draws upon funds of knowledge
in Native communities to derive and apply research insights in education.
"Funds of knowledge" refers to the repositories of knowledge residing in
communities that are frequently overlooked or dismissed in conventional
Western science and school curricula (Moll et al., 1992). In a study of
the Yupiit (Alaska) Nation and school system, Yupiaq researcher Oscar Kawagley
shows how local scientific, mathematical, and technological knowledge can
be used to improve the school curriculum. Situating science within a Yupiaq
worldview and ecology of sustainability, Kawagley provides a rigorous and
authentic framework for research and "for rethinking what we teach in schools
and how we teach it, particularly as it relates to science" (1995, p.138).
Research is needed that attends to the great diversity among Nations
and the variability within local sites. Such research will most often be
qualitative in its methodology. Page (2000) points out that "Qualitative
research has challenged the 'science' in the social sciences, and in the
natural sciences, too, even though science has long been the gold standard
for knowledge and a source of disciplinary authority" (p. 23). She warns
"nostalgia for the large-scale, randomized experiments thought to provide
unbiased and exact answers" (p. 24). The quest for scientific truth in
such endeavors will likely prove elusive (Cassell, 2002) and could pose
a threat to AI/AN educational self-determination and sovereignty.
The political context of educational research can make the difference
between a project that serves Native community and strengthens sovereignty
or actively undermines both. None of the parties involved in educational
research is apolitical--not tribal communities, tribal governments, federal
education agencies, or academicians. We all work from particular positions,
but certain positions cannot continue to be privileged over others. Ultimately,
judgments about research quality in Indian education most productively
belong in Indian country.
Banks, S. R. (1997). Caregiver and professional perceptions of assessment
practices and validity for American Indian/Alaska Native families. Journal
of American Indian Education, 37(1), 16-44.
Brayboy, B. M., & Deyhle, D. (2000). Insider-outsider: Researchers
in American Indian communities. Theory into Practice, 39(3), 163-169.
Cassell, J. (2002). Perturbing the system: "Hard science," "soft science,"
and social science: The anxiety and madness of method. Human Organization,
Clinton, W. J. (1998, August 6). American Indian and Alaska Native education:
Executive order 13096. Federal Register, 63(154), 42681-84. Retrieved December
20, 2002, from http://www.niea.org/indedeo.htm.
Corey, S. M. (1953). Action research to improve school practices. New
York: Teachers College Press, Columbia University.
Kawagley, A. O. (1995). A Yupiaq worldview: A pathway to ecology and
spirit. Prospect Heights, IL: Waveland Press.
Kelly, K. T. (1996). The logic of reliable inquiry. New York: Oxford
Lewin, K. (1946). Action research and minority problems. Journal of
Social Issues, 2(4), 34-46.
Lipka, J., with Mohatt, G. V., & the Ciulistet Group. (1998). Transforming
the culture of schools: Yup'ik Eskimo examples. Sociocultural, political,
and historical studies in education. Mahwah, NJ: Lawrence Erlbaum Associates.
Lomawaima, K. T. (2000). Tribal sovereigns: Reframing research in American
Indian education. Harvard Educational Review, 70(1), 1-21.
Maxwell, J. A. (1992). Understanding and validity in qualitative research.
Harvard Educational Review, 62(3), 279-300.
McCarty, T. L. (1997). Teacher research methods in language and education.
In N. H. Hornberger & D. Corson (Eds.), Encyclopedia of language and
education: Vol. 8. Research methods in language and education (pp. 227-237).
Norwell, MA: Kluwer Academic Publishers.
McCarty, T. L. (2002). A place to be Navajo: Rough Rock and the struggle
for self-determination in Indigenous schooling. Sociocultural, political,
and historical studies in education. Mahwah, NJ: Lawrence Erlbaum.
McCarty, T. L., & Dick, G. S. (2003). Telling The People's stories:
Literacy practices and processes in a Navajo community school. In A. I.
Willis et al. (Eds.), Multicultural issues in literacy research and practice.
Mahwah, NJ: Lawrence Erlbaum Associates.
McCarty, T. L., Watahomigie, L. J., & Yamamoto, A. Y. (1999). Introduction:
Reversing language shift in Indigenous America: Collaborations and views
from the field [Theme issue], Practicing Anthropology, 21(2), 2-4.
Metcalf, S. (2002, January 28). Reading between the lines. The Nation.
Retrieved June 19, 2002, from http://www.thenation.com/doc.mhtml?I=20020128&x=metcalf.
Mihesuah, D. A. (Ed.) (1996). Writing about (writing about) American
Indians [Special issue]. American Indian Quarterly, 20(1).
Miller, D. C., & Salkind, N. J. (2002). Handbook of research design
and social measurement (6th ed.). Thousand Oaks & London: Sage Publications.
Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds
of knowledge for teaching: Using a qualitative approach to connect homes
and classrooms. Theory into Practice, 31(1), 132-141.
Padilla, A. M., & Lindholm, K. J. (1995). Quantitative educational
research with ethnic minorities. In J. A. Banks & C. A. Banks (Eds.),
Handbook of research on multicultural education (pp. 97-113). New York:
Macmillan Publishing. (ERIC Document Reproduction Service No. ED 382 701)
Page, R. N. (2000). The turn inward in qualitative research. Harvard
Educational Review, 70(1), 23-38.
Research Agenda Working Group, Strang, W., & von Glatz, A. (2001).
American Indian and Alaska Native education research agenda. Washington,
DC: U.S. Department of Education. (ERIC Document Reproduction Service No.
ED 462 216) Retrieved December 20, 2002, from http://www.indianeduresearch.net
Romero, M. K. (1994). Identifying giftedness among Keresan Pueblo Indians:
The Keres study. Journal of American Indian Education, 34(1), 35-58.
Smith, L. T. (1999). Decolonizing methodologies: Research and Indigenous
peoples. New York: Zed Books.
Snipp, C. M. (2000, May-June). Some alternative approaches to the classification
of American Indians and Alaska Natives. Paper presented at the American
Indian and Alaska Native Research Agenda Conference, Albuquerque, NM. Retrieved
June 19, 2002, from http://www.indianeduresearch.net/MSnipp.pdf.
Stringer, E. T. (1996). Action research: A handbook for practitioners.
Thousand Oaks: Sage Publications.
Swisher, K. G. (1996). Why Indian people should be the ones to write
about Indian education. American Indian Quarterly, 20(1), 83-90.
Taba, H., & Noel, E. (1957). Action research: A case study. Washington,
DC: Association for Supervision and Curriculum Development (NEA).
Westmeyer, P. (1981). A guide for use in planning and conducting research
projects. Springfield, IL: Charles C. Thomas.
Wilkins, D., & Lomawaima, K. T. (2001). Uneven ground: American
Indian sovereignty and federal law. Norman: University of Oklahoma Press.
Wolcott, H. F. (1994). On seeking--and rejecting--validity in qualitative
research. In H. Wolcott, Transforming qualitative data: Description, analysis,
and interpretation (pp. 337-373). Thousand Oaks, CA: Sage Publications.
Zepeda, O. (1990). American Indian language policy. In K. L. Adams &
D. T. Brink (Eds.), Perspectives on official English: The campaign for
English as the official language of the USA (pp. 247-256). Hawthorne, NY:
Walter de Gruyter & Co.