Standard Setting For The Brave New World

Saturday, June 22, 2013: 11:05 AM-12:00 PM
Maryland 3-4 (Gaylord National Resort and Convention Center)
  • CCSSO PPT 2013v2.pptx (976.4 kB)
  • Content Strands:
    1. Improving data analysis, use, and reporting
    2. Transitioning assessment systems
    Student scores must be reported in a very clear, structured manner to be of help to most educators and parents, and the only way to accomplish this is through a standard setting workshop. The purpose of this session is twofold: (1) Participants will learn about a relatively new approach to establishing cut scores using the Reasoned Integrated Judgment (RIJ) method of standard setting; and (2) participants will have the experience of serving as a panelist in a mock standard setting workshop.

    The RIJ method of standard setting involves aspects of the Reasoned Judgment and Integrated Judgment methods. Panelists first review and discuss the performance level descriptors (PLDs) in terms of what is expected from students to obtain a particular level of achievement. During a full-group, facilitated discussion, panelists talk through realistic expectations based on the PLDs and list those skills that are an absolute must for any student representing a specific PLD. The facilitator then asks panelists to limit this skills list to just those required of students moving from one level to the next. To complete Round 1, the facilitator asks panelists to determine what numerical point on the scale best represents this focused list of skills. Round 1 depends on skills used in the Reasoned Judgment method.

    Round 2 utilizes assessment items and student work to solidify this judgment, which is similar to the Integrated Judgment method. The facilitator leads participants through items, including summary data and/or student work as appropriate, to allow panelists an opportunity to clarify their expectations and understanding. Panelists review each item with a focus on item difficulty and the skills necessary to respond to each item. In particular, panelists are asked to consider if the skills necessary to respond to each item pertain to students just moving from one performance level to the next or if the skills more likely pertain to students solidly in the middle of a performance level. Following this discussion, panelists are allowed to adjust their cut scores as appropriate before commencing a final round, during which impact data is presented and reviewed.

    This workshop is designed to help participants get a clear understanding of what is involved for standard setting panelists. To accomplish this, participants will be trained on the RIJ method and learn about the necessary background work completed by program managers, assessment development staff, and psychometricians. Then, panelists will actually work through setting a cut score utilizing the RIJ method.

    Participants will serve as panelists for a mock standard setting workshop intent on establishing a passing point for a middle school math exam. The presenters, serving as facilitators, will lead panelists through the essential decision-making process to determine what skills are required to enter a particular level of performance before having them determine a numerical cut point. Panelists will then review items and summary data to provide background for discussing the skills necessary to answer a particular item. Following this item review and discussion, panelists will be asked to make any changes to their original cut scores and the facilitator will show mock impact data. 

    This session will give participants the tools to understand the intricacies of the standard setting process. Since many conference attendees are working to establish new tests or reworking old tests during this time of varying educational pressures, establishing cut scores will be required. This experience will provide participants with information on the prerequisites of standard setting and, in particular, the RIJ method.

    << Previous Proposal | Next Proposal >>