- Improving data analysis, use, and reporting
An important metric used in accountability systems are cut-scores derived from the application of standard-setting methods. Despite their common use, many standard-setting methods are often criticized because raters struggle to understand the relationships between different standard-setting data, may exhibit rater inconsistency, and may not understand score gaps that exist between items or scores. This session discusses a framework based on construct maps (Wilson, 2005) that can be used to make the relationships between different standard-setting data more transparent and can reduce the impact of score gaps and rater inconsistency.
The symposium is structured into four separate sections. In the first section of the symposium, an overview of construct maps and their critical features is provided. Construct maps are visual and graphical displays that show how the scores on an assessment are related to other examinee and item performance data. Specific discussion will be given to how several common standard-setting methods, including the Angoff (Angoff, 1971), Bookmark (Lewis, Mitzel, & Green, 1996), Briefing Book (Haertel, 2002), and Body of Work (Kingston, Kahl, Sweeney, & Bay, 2001) methods, fit into this framework. Discussion will also be given to how construct maps can be used to help improve standard setting by the making the process more transparent, reducing the impact of score gaps and rater inconsistency, enhancing validity arguments, and showing information to policy makers as they decide on final cut scores.
The second section of the symposium will discuss several standard-setting applications that employed construct maps as part of the standard-setting processes in state testing programs. This includes standard settings that employed the Body of Work and the Bookmark method. The applications discussed will include both general assessment and alternate assessment examples. Specific discussion will be given to the training and specific steps required to implement the construct maps as well as standard-setting results, panelist feedback, and validity evidence collected from the standard settings.
The third section of the symposium will discuss the policy implications of using the construct maps and how they can be used to improve score use and reporting from a state perspective. Specific consideration will be given to how construct maps can show not only data that has typically been displayed and considered in past standard settings, but also additional data, such as college and career readiness benchmarks, performance on other assessments, and cut scores from multiple states.
The fourth section of the symposium will engage the audience in a discussion and answer any questions that states, researchers, and practitioners may have about the use of construct maps in the context of future standard setting in state testing programs.
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed., pp. 508-597). Washington, DC: American Council on Education.
Haertel, E. H. (2002). Standard setting as a participatory process: Implications for validation of standards-based accountability programs. Educational Measurement: Issues and Practice, 21(1), 16-22.
Kingston, N. M., Kahl, S. R., Sweeney, K., & Bay, L. (2001). Setting performance standards using the Body of Work method. In G. J. Cizek (Ed.), Standard setting: Concepts, methods, and perspectives (pp. 219-248). Mahwah, NJ: Erlbaum.
Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996, June). Standard setting: A Bookmark approach. In D. R. Green (Chair), IRT-based standard setting procedures utilizing behavioral anchoring. Symposium presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment. Phoenix, AZ.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.