Dissertations, Theses, and Capstone Projects

Date of Degree

9-2016

Document Type

Dissertation

Degree Name

Ph.D.

Program

Educational Psychology

Advisor

Jay Verkuilen

Committee Members

Howard Everson

Bruce Homer

Subject Categories

Cognitive Psychology | Educational Assessment, Evaluation, and Research | Educational Psychology

Keywords

Technology and Assessment, Validity, Cognitive Load Theory, Psychometrics

Abstract

With the advent of the newly developed Common Core State Standards and the Next Generation Science Standards, innovative assessments, including technology-enhanced items and tasks, will be needed to meet the challenges of developing valid and reliable assessments in a world of computer-based testing. In a recent critique of the next generation assessments in math (i.e., Smarter Balanced), Rasmussen (2015) observed that many aspects of the technology “enhancements” can be expected to do more harm than good as the computer interfaces may introduce construct irrelevant variance. This paper focused on issues surrounding the design of TEIs and how cognitive load theory (Miller, 1956) is a promising framework that can be applied to computer-based item design to mitigate the effects of computer interface usability. Two studies were conducted. In the first study I used multi-level modeling to assess the effect of item characteristics on examinees’ relative performance. I hypothesized that item level characteristics, namely response format, would significantly contribute to the amount of variance explained by item characteristics over and above student characteristics. In study two, I used two exemplar items to show how data concerning examinees’ actions—produced through latent class analyses—can be used as evidence in validity investigations. Results from study 1 suggested that item type does not explain the variation in student scores over and above examinee characteristics. Results from study two suggested that LCA is a useful tool for diagnosing potential issues in the design of items and the design of their scoring rubrics. Evidence provided from both studies illuminates the immediate need to further research computer-based items that are beginning to be used widely in high stakes, large-scale assessments. In an effort to move away from traditional multiple choice items and toward more authentic measurement by incorporating technology based item features, we may be affecting how examinees respond to the item due to inadvertent increases in cognitive load. Future research involving experimental manipulation is necessary for understanding how item characteristics impact how examinees responses.

Download

Share

COinS