From State-Wide to State Consortium Assessment Systems: Test Administration Lessons Learned From a Consortium Pilot

Saturday, June 22, 2013: 8:00 AM-9:30 AM
Chesapeake D-E (Gaylord National Resort and Convention Center)
  • NCSA presentation 6-22-2013.ppt (1.1 MB)
  • Administration Lessons Learned From the SBAC Pilot Test.pdf (521.0 kB)
  • NCSA SBAC Administration Lessons Learned_FINAL.ppt (1.8 MB)
  • Content Strands:
    1. Transitioning assessment systems
    The Smarter Balanced Assessment Consortium (SBAC) is one of two consortia developing and implementing test administration procedures to ensure the accuracy and comparability of computer adaptive assessments (CAA) in mathematics and language arts in 2015.    Member states are not only in the process of transitioning schools and students to the common core that inform the new tests, but also to new policies and procedures around the multi-state test administration in 25 states.  This symposium will report on member states’ pilot test experiences from a test administration perspective from the front lines as well as implications for transitioning from statewide to consortium assessments.

    According to the Standards for Educational and Psychological Testing (AERA, APA & NCME, 1999), without standardized test administrations and testing conditions, the accuracy and comparability of score interpretations as well as student opportunity to demonstrate skills and abilities could be diminished.   While the goal of standardized administrations is challenging with traditional paper-and-pencil large-scale tests, the task increases in complexity when computer delivered tests is added in the mix. This past spring, member states had a unique opportunity to preview consortium test administration policies and procedures in action and to glean how they might better transition states to a system that provides  in a large-scale technology-driven testing environment  “comparable measurements for everyone, and unfair advantage to no one” (AERA, APA & NCME, 1999, p. 61). 

    Reckase (2011) states that one of the biggest problems for a CAA is getting access to computers needed while maintaining test security.  He adds other issues include effective technical support and maintenance, system upgrades and replacement due to technological change.  Section 5 of the testing standards provides guidance for test administrations. However, can an assessment program of the magnitude of the consortia assessments succeed in meeting the goals of the standards?  The symposium is designed to present consortium test administration challenges and transition implications from the perspective of five states involved in the first consortium pilot in four areas of test administration:

    1. Communication with stakeholders (test directions, help desk support)
    2. Test Delivery technology readiness
    3. Accessibility and Accommodations (special populations)
    4. Test Security

    With the scope of the consortia assessments and the complexity of a development process that includes a multitude of contractors, state-led work groups, a high-stakes environment and psychometric challenges, it is incumbent on users to contribute to evidence of the validity of an assessment that is likely to change the face of student assessment across the nation.  An unsystematic and ineffective test administration could destroy an otherwise admirable test.  

    Standardized directions ensure that everyone understands the mechanics of test taking.  Presenters will detail consortium procedures to provide standardized directions and support during testing as well as transition implications.

    Technology readiness is one of the biggest challenges of a large-scale computerized assessment.  It is particularly challenging in an assessment and instructional world that remains largely paper-and-pencil based.  Presenters will describe how one state rose to the challenge and how it expects to shift all students entirely to a CAA environment.

    The uniform guidelines to test development eased the task of making assessment equally accessible to all students.  Nevertheless, ensuring students fair and equal opportunity to demonstrate what they know and can do remains thorny. Presenters will describe SBAC accessibility and accommodations policies and procedures as well as one state perspective of implications for the operational assessment.

    Designing and implementing effective test security is vital to a fraud-free test and is particularly demanding in technology-based assessments. Presenters will outline difficulties associated with preserving test security in CAAs and implications for the operational assessment.