Planning for Transition: Evolution of Next-Generation Assessment System Designs to Support Ongoing Improvements in Measuring Complex Skills and Constructs

Thursday, June 20, 2013: 8:30 AM-10:00 AM
National Harbor 4 (Gaylord National Resort and Convention Center)
  • PARCC NCSA session on assmt innovation and evolution 6 20 2013.pdf (1.2 MB)
  • Next-Generation Assessment Systems_McCall.pdf (1.4 MB)
  • ScaliseCCSSO_discussantSlidesCURR.pdf (2.4 MB)
  • CCSSO 6 20 2013.pdf (1.3 MB)
  • Content Strands:
    1. Transitioning assessment systems
    Common Core State Standards (CCSS) have brought hard-to-measure constructs to the fore and many have never been assessed in state summative assessments before, so challenges will arise (Scalise, 2012). Given that the Consortia are required to provide operational assessment systems by 2014-15, only moderate improvements can be expected in the systems when initially launched. This Session will focus on how to ensure that these assessments are designed to allow for continued incorporation of new technologies, data analytics and scoring methodologies that improve the measurement of important hard-to-measure constructs, and support innovation within an evolution of assessment ecosystems.

    A major challenge is the complexity of many of the constructs to be assessed in ELA and mathematics. Accurate and meaningful measurement will challenge existing state assessment systems, and many promising approaches are not yet ready for use within high-stakes assessments. As Member States in PARCC and Smarter Balanced develop their transition plans, they must address the deeper challenges emerging within the assessments themselves as the Consortia aggressively take on the task of measuring the hard-to-measure constructs being promulgated by the CCSS systems in authentic, engaging and psychometrically valid ways.

    Scalise (2012), who is an expert in technology-based assessment and has helped OECD and others incorporate new advances, provides a useful heuristic for examining the important dimensions of technology-based items and tasks to be brought online by the Consortia in school year 2014-15. An understanding of the extent to which the Consortia can incorporate “innovative item types” in their initial assessments and the challenges that must be addressed to ensure that these systems can evolve and improve over time, while still providing longitudinal trends, will be our focus.

    Teachers unpacking the CCSS often find themselves struggling to define what makes a hard-to-measure constructs, including (Scalise, 2012):

    • Test measures a trait that is difficult to define;
    • Trait is cross-cutting and must play out in a variety of contexts;
    • Knowledge, skills, and attitudes (KSAs) to be addressed involve interactions and dependencies;
    • Adequate construct coverage is difficult to achieve; and
    • Construct itself does not have sufficient stability.

    Scalise (2012) provides further guidance on what needs to happen to support the “evolution” of NGAS over time. Measurement challenges include:  

    1. Establishing forums where innovation can be adopted in an ongoing fashion within state assessment ecosystems and then migrated as desirable. Locking down around any particular era of technology will not be beneficial and should be avoided.
    2. Many of the assessment targets in CCSS will involve: more complexity due to cognitively complex domains and higher order thinking; assessments with more interactions among the KSAs; dependence in the observations, data density; and noisier data with more construct irrelevant variance.
    3. 21st century measurement thrives in generating subtle and complex inferences in the presence of abundant, rich data. If states can effectively capture this dual opportunity, a stronger signal likely can be generated from a less simplistic view of what students need to know and be able to do.

    The first two presentations by Dr. Jeff Nellhaus and Dr. Marty McCall will describe exemplars from PARCC and Smarter Balanced, respectively, of the emerging assessment development work underway to produce prototype items and performance tasks. Third, Dr. Kathleen Scalise will address the hurdles and challenges ahead of the Consortia if they are to avoid “locking in” on the 2014-15 designs and technologies, and instead are to support the ongoing evolution of NGAS over time. Sue Rigney, a leading USED assessment expert, will serve as the Discussant.


    Scalise, K. (2012). Center for K-12 Assessment & Performance Management at ETS, Using Technology to Assess Hard-to-Measure Constructs in the Common Core State Standards and to Expand Accessibility.