A Symposium On NAEP Grade 12 Preparedness and Use of Judgmental Standard Setting

Wednesday, June 22, 2011: 10:30 AM-12:00 PM
Plaza International J (Peabody Orlando)
Content Strands:
  1. Assessment Purposes and Types
Included in the priorities of the National Assessment of Educational Progress (NAEP) 12th Grade Preparedness Research is setting cut scores on the grade 12 reading and mathematics assessments indicating preparedness for college entry and entry in a job training course. In this session, the features of the judgmental standard setting process will be discussed and procedural results will be presented.

In 2009, the National Assessment Governing Board (NAGB) adopted a multifaceted program of research aimed at providing validity evidence for reporting preparedness on 12th grade NAEP. The program of research was formulated and recommended by a panel representing a variety of technical expertise related to preparedness for postsecondary education and job training. Judgmental standard-setting studies were recommended as one of several methods to identify points on the NAEP scale to reference preparedness for college credit-bearing, entry-level courses of the type that would fulfill general education requirements or preparedness for placement in a job training program for one of several occupational areas to be specified for this research (NAGB, 2009). In 2011, twelve operational standard setting studies will be implemented for the 2009 grade 12 NAEP in reading and mathematics. In this symposium, the steps in this innovative set of judgmental standard-setting studies will be traced and some preliminary results will be presented.

The first two presentations will provide the context for NAGB's 12th grade preparedness research studies. First, the role of these studies in NAGB’s 12th Grade NAEP Preparedness Research program will be presented. Second, information on the meaning of preparedness in the NAEP context will be presented and compared with other definitions currently under consideration.

The third presentation will describe the process for developing performance standards used for the judgmental standard setting and highlight key procedural results. To develop preparedness performance standards, panelists will be assembled online for webinars with the goal of drafting performance standards that they will subsequently finalize during the onsite standard setting panel meetings.

The fourth presentation will focus on the judgmental standard setting process to set the cut scores using the bookmark method. Several computer-based innovations of the method will be implemented in six judgmental standard setting workshops. A demonstration of some elements of the Computer Aided Bookmark (CAB) is included in the symposium. Preliminary results will also be presented.  

Lead Presenter: