Technology Showcase: Usability Research: Lessons Learned For Digital Assessment Delivery

Friday, June 21, 2013: 10:00 AM-12:00 PM
Potomac C (Gaylord National Resort and Convention Center)
  • CWendler Session Overview.ppt (498.5 kB)
  • CMills CBT Assessment for Young Students.ppt (319.0 kB)
  • DPisacreta Adapting to User Supplied Hardware.ppt (960.0 kB)
  • DBrauner Accessiblity and Tablets.ppt (2.4 MB)
  • JSumpter K12 EL Identification tool.ppt (704.0 kB)
  • EZilbert CBT Science Tryout.ppt (304.0 kB)
  • Handouts
  • DBrauner Accessible Tablets in Mainstream Classrooms.pdf (316.1 kB)
  • DPisacreta Adapting to User Supplied Hardware Summary.pdf (436.5 kB)
  • Content Strands:
    1. Transitioning assessment systems
    2. Implementing state and federal programs and policy
    With the transitioning of assessment systems in the RTTA, we reflect upon the lessons learned over the years with the digital delivery of high stakes assessments. Students continue to be more tech-savvy and studies show that using technology for assessments increases their engagement while allowing for measurement innovations.  This session presents the results of two usability studies, Online Achievement Assessment of Young Students (Grades 1, 2 and 3) and Designing Items to Adapt to User Supplied Hardware,along with three cases of usability work – California STAR CBT Tryout, gaming approaches with ELs in Texas, and accessibility work on tablets.

    Online Achievement Assessment of Young Students (Grades 1, 2 and 3)Dr. Christine Mills presents three studies completed to support the transition of paper based achievement tests to a computer based format.  Two small usability studies assessing the ease of use for young students of the computer based (CBT) and paper based (PBT) versions, and the delivery of CBT audio delivered assessments (typically read aloud by the teacher) were completed along with one modality comparability study. 

    Designing Items to Adapt to User Supplied HardwareDebbie Pisacreta explores design issues involved in adapting computer delivered test items from laptop screens to the iPad.  Based on a literature review of presentation and interface design issues related to viewing content on different size screens and resolutions, a usability test was conducted to look at item designs.  In 2012, a second usability test was conducted to look at interface issues that might cause performance differences between students taking a test on a laptop vs. an iPad, such as typing on the soft keyboard and touching the screen instead of using the mouse.

    In Texas, Principal Robert Harris of John Jay High School will describe a pilot focused on helping schools to identify, screen, and appropriately place English Language Learners. The application is leveraging interactive scenarios to engage students and measure their abilities in a game-like atmosphere and is being delivered via tablets and PCs.  The goal of this work is to determine how students navigate and interact with the application across the different delivery platforms and determine the appropriate modes of delivery.

    Diane Brauner is a teacher of the visually impaired (TVI) and instructor in orientation and mobility (O&M) in North Carolina.  She will describe how tablets with built-in accessibility support have increased in the classroom and at home, and the potential for significant, if not revolutionary advances in how students with visual impairments and other disabilities, are enabled to independently interact with course materials and assessments.  Technologies, such as Bluetooth connected refreshable braille displays and innovative use of haptic feedback are two exciting areas with impact on assessments. While built-in accessibility features, such as Apple® VoiceOver®, are key enabling technologies, accessibility standards, such as those from the World Wide Web Consortium, play a critical role defining the technical requirements for making computer-based assessments work effectively with the assistive technologies.

    In the California study, a sample of several hundred students within a single school district take a version of the grade 8 science test that was administered as part of the STAR CBT Tryout, adapted for use on iPad tablets.  It is reasonable to expect that the future of CBT testing in California will include tablets so the goal was to observe how students interact with the tests through the touch screen iPad interface, which differs from the traditional keyboard-and-mouse testing interface.