- Implementing state and federal programs and policy
In February 2012, Measured Progress assisted that National Assessment Governing Board (NAGB) in setting achievement levels for the 2011 NAEP writing. The presentations will trace the achievement levels setting process from the development of the Achievement Levels Descriptions, setting the cut scores using the Body of Work Technological Integration and Enhancements (BoWTIE), and selecting student responses that are illustrative of student performance at the Basic, Proficient, and Advanced levels. Areas deemed to have contributed most significantly to the success of the achievement levels setting (ALS) process will be discussed. A demostration of BoWTIE will be provided to the participants.
Historical Context and Perspective
The National Assessment Governing Board (NAGB) is probably best known for setting achievement levels for NAEP. For the 2011 NAEP writing, the rigor and traditions of previous NAEP standard-setting meetings were upheld. To set achievement levels for the 2011 writing assessments, the first wholly computer-based NAEP, an integrated and technologically enhanced version of the Body of Work method implemented. The first presentation will provide a historical context and perspective for setting standards on NAEP followed by the different aspects of achievement levels-setting that contributed to the success of the achievement levels setting process.
Developing Achievement Levels Descriptions
The achievement levels descriptions (ALDs) for the 2011 NAEP writing were developed by a contractor to NAGB and were provisionally approved, with full approval contingent upon the results of the studies. After consideration of possible influences on the results of the pilot study, technical advisors recommended that the ALDs be revised. The content facilitators of the ALS process, who were also members of the NAEP Writing Framework Steering Committee, worked with NAGB staff to better align the ALDs with the policy definitions. The challenges in developing the final ALDs and how these challenges were overcome will be presented.
Thirty panelists for each grade level were planned for the NAEP achievement levels setting. The composition—two-thirds educators and one-third non-educators—has been deemed appropriate by NAGB since the beginning of its standard-setting initiative in 1992. According to NAGB policy, the panelists must represent a balance of demographic characteristics such as gender, race/ethnicity, and geographic region. Panelist recruitment proved to be more challenging than anticipated. This presentation will discuss how the challenges were overcome to yield an engaged and conscientious panel that was able to follow a rigorous process and make decisions and recommendations based on informed judgments.
Just as NAEP is considered the gold standard for assessment, NAEP ALS is considered the gold standard for standard setting, the process elements of which are based on highly regarded principles and traditions established by experts with a long history with NAEP and NAEP standard setting. The ALS process facilitators with experience in non-NAEP approaches to standard setting were instructed to make adjustments to successfully facilitate a NAEP ALS process. One of these process facilitators will present differences in facilitating standard settings for NAEP and those for other K-12 assessments, and discuss elements of NAEP standard setting that might be recommended for other K-12 assessments and vice versa.
With the 2011 NAEP ALS process, NAGB has substantially advanced the field of educational assessment by computerizing standard setting. Measured Progress implemented a technologically enhanced version of the Body of Work standard-setting method. Using the BoW method, ALS panelists examined student responses to two writing prompts and provided ratings by examining a student’s writing holistically and classifying it to achievement level. Classifications provided by the panelists were used to compute individual cut scores and the median was considered the cut score for each achievement level. All of the stages of the process were integrated in BoWTIE.