Continuous Flow Scoring of Prose Constructed Response: A Hybrid of Automated and Human Scoring

Wednesday, June 22, 2016: 10:00 AM-11:30 AM
Franklin Hall 8 (Philadelphia Marriott Downtown)
Presentations
  • Continuous Flow Scoring_CCSSO_Introduction_June20162.pdf (164.1 kB)
  • Continuous Flow Scoring of Prose Constructed Response_ CO.pdf (137.8 kB)
  • CCSSO_AS_Way_v1(06.15.16).pdf (219.5 kB)
  • CCSSO2016-Lochbaum.v5.pdf (1.3 MB)
  • Continuous Flow Scoring_NCSA_Discussant_June 2016.pdf (62.8 kB)
  • Content Strands:
    1. Assessment Delivery and Administration
    2. Test Results Analysis and Reporting
    SUMMARY:
    This session introduces the concept of continuous flow scoring of online constructed response items. Unlike most applications of automated scoring, where the computer either scores all responses or is used after humans score all responses, in continuous flow scoring the automated scoring engine is trained to estimate the confidence of the score given. In cases where there is less confidence in the automated score, it is automatically routed to human scoring. This session will have three presentations: (1) the concept of continuous flow scoring and infrastructure needed to support it; (2) a state perspective on what is needed for continuous flow scoring to succeed in a statewide testing program; and (3) results of research conducted on continuous flow scoring in the context of the PARCC assessment program. A state testing director experienced with automated scoring will serve as the session discussant and provide commentary on the presentations.
    Discussant:
    Moderator: