Incorporating a Theory of Action Into Validation

Thursday, June 20, 2013: 10:15 AM-11:45 AM
Maryland 1-2 (Gaylord National Resort and Convention Center)
Presentations
  • CCSSO CBAL action theory 6-13.pdf (326.0 kB)
  • NCSA 2013_Incorporating a TOA into Validation_Forte.pdf (213.2 kB)
  • Marion_Theory of Action and Interpretative Arguments_062113.pdf (88.3 kB)
  • sireci ccsso ncsa 2013 smarter balanced validation.pdf (1.5 MB)
  • Handouts
  • CBAL TOA journal article.pdf (2.7 MB)
  • toa for ncsa 2013.pdf (479.5 kB)
  • Content Strands:
    1. Balancing new assessment systems within a state
    2. Implementing state and federal programs and policy
    ABSTRACT:
    Incorporating a Theory of Action into Validation

    The Standards for Educational and Psychological Testing provide guidance for test development and for evaluating the use of a test for a particular purpose.  However, contemporary assessment programs, such as SBAC, PARCC, and NCSC go beyond simple statements of testing purpose to develop an assessment systemthat facilitates positive change for students, teachers, schools, and school systems.  Such goals are typically articulated in a Theory of Action (TOA), which should be the starting point for validation. In this symposium, measurement experts provide advice and examples of how to incorporate a TOA into a comprehensive validation plan. 

    The session features four presenters and a discussant; all of whom have extensive experience in test validation and are widely respected at the national level.  The first presenter will discuss the importance of incorporating TOA into a validation plan, the next two presenters will describe SBAC and PARCC's efforts in incorporating their TOA into the validation plan, and the fourth presenter will discuss how to document a comprehensive validity argument that focuses on a TOA.  The session also features a nationally recognized discussant who will comment on the presentations.  Brief descriptions of the presentations follow.

    Paper 1: Theory of Action in Educational Assessment:  This presentation will describe the notion of theory of action in educational assessment.  The idea is placed in the context of modern validity theory.  The importance of theory of action is emphasized for specifying how an assessment program is to be carried out and the impacts it is intended to achieve.  Finally, an example is given from a research program that uses theory of action as an underlying principle.

    Paper 2: Smarter, Balanced Validation:  Incorporating Systemic Objectives into a Validity Agenda:  The Standards for Educational and Psychological Testing (AERA et al., 1999) provide important guidance for test development and for evaluating the use of a test for a particular purpose.  However, modern assessment systems, such as the Smarter Balanced Assessment Consortium, have multiple purposes, many of which extend beyond the assessment to broader societal goals.  This presentation will address how the validation framework provided in the AERA et al. Standardscan be extended to incorporate the key ideas in a broad Theory of Action, using the SBAC Theory of Action and validation plan as an example.

    Paper 3:  Title: Validating Inferences from PARCC: A Theory of Action:  One approach to ensuring that inferences made from assessment results are reasonably supported by validation evidence is to build multiple opportunities for establishing such evidence throughout the design and development process. This presentation will highlight how evidence-centered design has created a framework for PARCC validation, and how validation is informed by PARCC theory of action.

    Paper 4: A Matrix Model for Gathering and Documenting Validity Evidence:  Using the traditional targets for validity evaluation, we describe the concepts underlying these major inferences as they apply across four critical validity themes: Content Coherence, Comparability, Accessibility and Fairness, and Consequences.  We then demonstrate how to address each of these themes by building upon an approach to technical documentation (Marion & Pellegrino, 2006) that organizes validity evidence across the operational phases of the testing process: test design and construction, administration, scoring and analysis, and reporting.

    References

    American Educational  Research Association, American Psychological Association, & National Council on Measurement in Education.  (1999).  Standards for educational and psychological testing.  Washington, D.C.:  American Educational Research Association.

    Marion, S. F., & Pellegrino, J. W. (2006).  A validity framework for evaluating the technical quality of alternate assessments.  Educational Measurement:  Issues and Practice, 25(4), 47-57.

    Official:
    Discussant: