Reliability, Validity, and Authenticity in American Indian and Alaska Native Research. ERIC Digest.

by Lomawaima, K. Tsianina - McCarty, Teresa L.

The constructs used by scholars across the physical, natural, and social sciences to evaluate research quality--as objective, reliable, valid, generalizable, randomized, accurate, authentic--are not value-free or apolitical. They all require the application of human judgment, which is inevitably affected by cultural norms and values (Westmeyer, 1981). Consequently, the use of these constructs to assess research focused on minority, marginalized populations requires extraordinary judgment. Such assessments must be based not only on Western notions of scientific quality but also, in the case of American Indians and Alaska Natives (AI/ANs), on a separate set of criteria prescribed in the interest of sovereignty. Sovereignty refers to the inherent, as well as constitutionally recognized, rights of tribes to self-government, self-determination, and self-education. Of course no contemporary polity--whether American Indian tribe or U.S. federal government--exercises unlimited sovereignty: "In the real world, sovereignty operates within constraints" (Wilkins & Lomawaima, 2001, p. 5). This Digest addresses these issues as researchers take on the challenges set forth by the "American Indian and Alaska Native Education Research Agenda" (Research Agenda Working Group, Strang, & von Glatz, 2001). The perspectives in this Digest are those of the authors and do not necessarily represent the positions of the U.S. Department of Education. 

DEFINITION OF TERMS

In the sciences, the terms "accuracy," "reliability," and "validity" are used to make evaluative judgments of measurement instruments (e.g., tests and surveys), or of data collection methods. An "accurate" measurement is one that is precise and exact. Although commonly applied to the mechanical measurements of the physical sciences, accuracy is also an issue in social science, including qualitative research, where researchers seek to understand complex, multiple perceptions of reality. A "reliable" method or instrument gives consistent results in different applications. Consistency, however, "says nothing about being right or wrong" (Kelly, 1996, p. 43). The "rightness" or "truth value" of research is an issue of "validity." Validity is always subject to human judgment: it asks the question, "Does this device [or method] measure what it is said [or claims] to measure?" (Westmeyer, 1981, p.42). 

The concept of validity has been contested and is subject to further refinement. In experimental research, it is common to distinguish "internal validity" (whether a proposed cause is actually the reason for an effect) from "external validity" (whether research results are generalizable across settings or populations) (Miller & Salkind, 2002). However, some qualitative researchers reject validity as a criterion for judging research, arguing that understanding is a more proper concern (Wolcott, 1994). Maxwell (1992) argues that validity in qualitative research refers not to data or methods but to accounts. "Descriptive validity" tests the factual accuracy of accounts; "interpretive validity" asks whether an account represents research participants' perspectives; "theoretical validity" refers to the explanatory power of accounts; and "evaluative validity" questions the usefulness or applicability of accounts. Parallels can be found in Banks' (1997) assessment of special education needs among AI/AN children. She distinguishes "construct validity" (the meaningfulness of research constructs or ideas), "treatment validity" (whether research participants judge its objectives, instruments, and procedures acceptable), and "ecological validity" (the extent to which research looks beyond the individual to social-historical contexts). 

THE REQUIREMENTS OF SOVEREIGNTY AND SELF-DETERMINATION

Judging research adequacy within the larger social-historical context of AI/AN education involves answering questions critical to the exercise of sovereignty: Why do the research? What factors motivate the researcher? Who has set the research goals? Who has the "disciplinary authority" (Page, 2000, p. 23) to do the research? Who will be involved in conducting the research? Whom does the research serve? How will it benefit the local community? 

Until recently, Native communities were rarely involved in raising these questions or in formulating, implementing, or evaluating studies. In the past few decades, however, tribal governments have increasingly taken control of research site access, instituting protocols and approval processes for research projects (Lomawaima, 2000; see Romero, 1994, for an excellent example of research working within protocols). For example, both the Tohono O'odham and Pascua Yaqui language policies stipulate that tribal authorities must approve all research about their people. The Yaqui policy lays out further requirements for copyright of publications growing out of such research (Zepeda, 1990). Additional sovereignty safeguards include requirements that tribal representatives be involved in setting research standards and evaluating projects against those standards. All these measures have grown out of the recognition that local communities have as great a stake as outside scholars in establishing what constitutes high-quality, useful research. 

There are viable alternatives to conventional, colonizing research paradigms (Smith, 1999). As a means for improving post-World War II intergroup relations, sociologist Kurt Lewin (1946) introduced an action research cycle of problem identification, fact finding, execution, and evaluation. A decade later, Corey (1953) and Taba and Noel (1957) operationalized this cycle in education. Now treated as synonymous with teacher research, action research is a "reemerging tradition" in which those previously designated as subjects actively participate in research processes that benefit them directly (Stringer, 1996, p. 7). Recent examples of teacher/action research in AI/AN education include Lipka, Mohatt, and the Ciulistet (teacher-leaders) Group's work on incorporating Yup'ik knowledge into science and math instruction (Lipka et al., 1998); collaborative research to revitalize Indigenous languages (McCarty, Watahomigie, & Yamamoto, 1999); and research on alternative literacy assessment and curriculum reform undertaken by Navajo teacher-researchers (McCarty & Dick, 2003; McCarty, 2002; for an overview of teacher research methods, see McCarty, 1997). 

AUTHENTICITY

Are the quality and authenticity of research guaranteed by the involvement of tribal members in design and evaluation? Some say it takes even more. Swisher (1996) argues for the primacy of Native scholars in conducting research on Native peoples and issues, attributing to them the benefits of an insider's view and experiences, enhanced passion and commitment, and the "authority to ask new and different questions" (p. 93). She calls on non-Native researchers and mentors, whose expertise and high-quality work she readily acknowledges, to step aside in favor of Native authors. Many would agree with Swisher's call to increase the authority of, recognition of, and publication by Native scholars and researchers, but not all are willing to ask non-Native allies to step aside (see Mihesuah, 1996 for a range of opinions). These observers suggest that the most productive research will result from respectful collaboration that does not dichotomize researchers as Native or non-Native but does make the contributions of Native colleagues integral to the design and conduct of studies. 

Even those who adhere to the "Native-only" requirement run up against debates over who is "really Indian." Snipp (2000) productively discusses the many criteria used to identify and assign "Indian-ness" and tribal identity, and the political and other agendas attached to them. Snipp concludes that contemporary criteria for tribal membership--however historically situated and flawed--are the prerogatives of sovereign tribes. 

Local evaluations of authenticity, whether dependent on research subject or research investigator, should be respected in accord with the principle of self-determination, even as researchers struggle to work out their often anomalous positions as "insider," "outsider," or some combination of both (Brayboy & Deyhle, 2000). 

POLITICS AND SCHOLARSHIP

There have been many calls over the years for improved research in AI/AN education. The most recent is Executive Order 13096 (Clinton, 1998), which mandates the development and implementation of a national research agenda in Indian education. Consultations and meetings with Native peoples across the country culminated in the "American Indian and Alaska Native Education Research Agenda" (Research Agenda Working Group, Strang, & von Glatz, 2001). The agenda identifies high-priority research interests and outcomes for Native communities and clearly articulates the central roles that Native scholars, parents, teachers, schools, and communities must play to ensure high-quality research. The promise embedded in the research agenda, however, may be threatened if limited notions of what constitutes high-quality research, using criteria such as those outlined earlier, overtake the powers of Native peoples to evaluate research in their communities and schools. 

Science as practiced by many Western researchers is often invoked as the answer to educational woes. This view tends to dismiss the power of all human inquiry--including Indigenous knowledge systems--to observe, analyze, theorize, and generate creative solutions. Research and development projects designed within the Western mold--especially those meant to provide guidance for classroom practice--often rely on standardized scripts to instruct and standardized tests to assess effectiveness (Metcalf, 2002). Using standardized tests with minority populations is a dubious practice (Padilla & Lindholm, 1995, p. 97). 

A much more promising line of research draws upon funds of knowledge in Native communities to derive and apply research insights in education. "Funds of knowledge" refers to the repositories of knowledge residing in communities that are frequently overlooked or dismissed in conventional Western science and school curricula (Moll et al., 1992). In a study of the Yupiit (Alaska) Nation and school system, Yupiaq researcher Oscar Kawagley shows how local scientific, mathematical, and technological knowledge can be used to improve the school curriculum. Situating science within a Yupiaq worldview and ecology of sustainability, Kawagley provides a rigorous and authentic framework for research and "for rethinking what we teach in schools and how we teach it, particularly as it relates to science" (1995, p.138). 

Research is needed that attends to the great diversity among Nations and the variability within local sites. Such research will most often be qualitative in its methodology. Page (2000) points out that "Qualitative research has challenged the 'science' in the social sciences, and in the natural sciences, too, even though science has long been the gold standard for knowledge and a source of disciplinary authority" (p. 23). She warns against "nostalgia for the large-scale, randomized experiments thought to provide unbiased and exact answers" (p. 24). The quest for scientific truth in such endeavors will likely prove elusive (Cassell, 2002) and could pose a threat to AI/AN educational self-determination and sovereignty. 

The political context of educational research can make the difference between a project that serves Native community and strengthens sovereignty or actively undermines both. None of the parties involved in educational research is apolitical--not tribal communities, tribal governments, federal education agencies, or academicians. We all work from particular positions, but certain positions cannot continue to be privileged over others. Ultimately, judgments about research quality in Indian education most productively belong in Indian country. 

REFERENCES

Banks, S. R. (1997). Caregiver and professional perceptions of assessment practices and validity for American Indian/Alaska Native families. Journal of American Indian Education, 37(1), 16-44. 

Brayboy, B. M., & Deyhle, D. (2000). Insider-outsider: Researchers in American Indian communities. Theory into Practice, 39(3), 163-169. 

Cassell, J. (2002). Perturbing the system: "Hard science," "soft science," and social science: The anxiety and madness of method. Human Organization, 61(2), 177-185. 

Clinton, W. J. (1998, August 6). American Indian and Alaska Native education: Executive order 13096. Federal Register, 63(154), 42681-84. Retrieved December 20, 2002, from http://www.niea.org/indedeo.htm. 

Corey, S. M. (1953). Action research to improve school practices. New York: Teachers College Press, Columbia University. 

Kawagley, A. O. (1995). A Yupiaq worldview: A pathway to ecology and spirit. Prospect Heights, IL: Waveland Press. 

Kelly, K. T. (1996). The logic of reliable inquiry. New York: Oxford University Press. 

Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34-46. 

Lipka, J., with Mohatt, G. V., & the Ciulistet Group. (1998). Transforming the culture of schools: Yup'ik Eskimo examples. Sociocultural, political, and historical studies in education. Mahwah, NJ: Lawrence Erlbaum Associates. 

Lomawaima, K. T. (2000). Tribal sovereigns: Reframing research in American Indian education. Harvard Educational Review, 70(1), 1-21. 

Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279-300. 

McCarty, T. L. (1997). Teacher research methods in language and education. In N. H. Hornberger & D. Corson (Eds.), Encyclopedia of language and education: Vol. 8. Research methods in language and education (pp. 227-237). Norwell, MA: Kluwer Academic Publishers. 

McCarty, T. L. (2002). A place to be Navajo: Rough Rock and the struggle for self-determination in Indigenous schooling. Sociocultural, political, and historical studies in education. Mahwah, NJ: Lawrence Erlbaum. 

McCarty, T. L., & Dick, G. S. (2003). Telling The People's stories: Literacy practices and processes in a Navajo community school. In A. I. Willis et al. (Eds.), Multicultural issues in literacy research and practice. Mahwah, NJ: Lawrence Erlbaum Associates. 

McCarty, T. L., Watahomigie, L. J., & Yamamoto, A. Y. (1999). Introduction: Reversing language shift in Indigenous America: Collaborations and views from the field [Theme issue], Practicing Anthropology, 21(2), 2-4. 

Metcalf, S. (2002, January 28). Reading between the lines. The Nation. Retrieved June 19, 2002, from http://www.thenation.com/doc.mhtml?I=20020128&x=metcalf. 

Mihesuah, D. A. (Ed.) (1996). Writing about (writing about) American Indians [Special issue]. American Indian Quarterly, 20(1). 

Miller, D. C., & Salkind, N. J. (2002). Handbook of research design and social measurement (6th ed.). Thousand Oaks & London: Sage Publications. 

Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds of knowledge for teaching: Using a qualitative approach to connect homes and classrooms. Theory into Practice, 31(1), 132-141. 

Padilla, A. M., & Lindholm, K. J. (1995). Quantitative educational research with ethnic minorities. In J. A. Banks & C. A. Banks (Eds.), Handbook of research on multicultural education (pp. 97-113). New York: Macmillan Publishing. (ERIC Document Reproduction Service No. ED 382 701) 

Page, R. N. (2000). The turn inward in qualitative research. Harvard Educational Review, 70(1), 23-38. 

Research Agenda Working Group, Strang, W., & von Glatz, A. (2001). American Indian and Alaska Native education research agenda. Washington, DC: U.S. Department of Education. (ERIC Document Reproduction Service No. ED 462 216) Retrieved December 20, 2002, from http://www.indianeduresearch.net 

Romero, M. K. (1994). Identifying giftedness among Keresan Pueblo Indians: The Keres study. Journal of American Indian Education, 34(1), 35-58. 

Smith, L. T. (1999). Decolonizing methodologies: Research and Indigenous peoples. New York: Zed Books. 

Snipp, C. M. (2000, May-June). Some alternative approaches to the classification of American Indians and Alaska Natives. Paper presented at the American Indian and Alaska Native Research Agenda Conference, Albuquerque, NM. Retrieved June 19, 2002, from http://www.indianeduresearch.net/MSnipp.pdf. 

Stringer, E. T. (1996). Action research: A handbook for practitioners. Thousand Oaks: Sage Publications. 

Swisher, K. G. (1996). Why Indian people should be the ones to write about Indian education. American Indian Quarterly, 20(1), 83-90. 

Taba, H., & Noel, E. (1957). Action research: A case study. Washington, DC: Association for Supervision and Curriculum Development (NEA). 

Westmeyer, P. (1981). A guide for use in planning and conducting research projects. Springfield, IL: Charles C. Thomas. 

Wilkins, D., & Lomawaima, K. T. (2001). Uneven ground: American Indian sovereignty and federal law. Norman: University of Oklahoma Press. 

Wolcott, H. F. (1994). On seeking--and rejecting--validity in qualitative research. In H. Wolcott, Transforming qualitative data: Description, analysis, and interpretation (pp. 337-373). Thousand Oaks, CA: Sage Publications. 

Zepeda, O. (1990). American Indian language policy. In K. L. Adams & D. T. Brink (Eds.), Perspectives on official English: The campaign for English as the official language of the USA (pp. 247-256). Hawthorne, NY: Walter de Gruyter & Co. 

Please note that this site is privately owned and is in no way related to any Federal agency or ERIC unit.  Further, this site is using a privately owned and located server. This is NOT a government sponsored or government sanctioned site. ERIC is a Service Mark of the U.S. Government. This site exists to provide the text of the public domain ERIC Documents previously produced by ERIC.  No new content will ever appear here that would in any way challenge the ERIC Service Mark of the U.S. Government.

More To Explore