Dissertations, Theses, and Capstone Projects

Date of Degree

2-2022

Document Type

Dissertation

Degree Name

Ph.D.

Program

Educational Psychology

Advisor

David Rindskopf

Committee Members

Keith Markus

Jay Verkuilen

Howard Everson

Morgan Polikoff

Subject Categories

Educational Assessment, Evaluation, and Research

Keywords

Alignment, Porter Alignment Index

Abstract

At the heart of educational systems worldwide is the idea that components of an educational system (i.e. standards, curricula, instruction, and assessments) need to be in alignment with one other for the system to produce the desired results of improving educational outcomes. Thus, a thorough and reliable methodology for measuring alignment is necessary especially given the escalating use of student assessment results as a determinant of effectiveness for districts, schools and teachers (Ananda, 2003a).

Today, there are two predominant methods for modeling and measuring that alignment: Webb’s methodology (Webb, 1997, 1999), and Porter’s methodology (Porter, 2002, 2006; McMaken & Porter 2012). Porter’s methodology has two distinct advantages: 1) it can measure alignment between any two (same or different) types of components, 2) it yields a single one-dimensional quantitative measure of alignment, Porter’s alignment index, hereafter PAI.

Administrators can select between competing standards, curricula, or assessments based on their PAI alignment with a given component they have in mind. For example, PAI can be useful most commonly in determining how well an assessment is aligned to a standard. The numeric value of PAI can tell educators whether that assessment is a relatively good match with that standard or not, in which case the assessment should be changed. But if the assessment should be changed, how should it be changed? PAI does not help educators answer this question. For that, Porter suggests a graphical representation he calls a heat map which shows the content distribution of each component in terms of high and low spots on a matrix. Educators would then have to compare heat maps and see where there is a deficit or excess of content on the assessment.

This dissertation built on the founding work of Professor Porter. I set forth to provide educators further insight into alignment supplementing PAI and heat maps. I argued that PAI is a measure of total alignment - with its numerical complement (i.e. 1-PAI) defined as a new measure I called total misalignment (or misalignment). I showed how misalignment can then be logically attributed to, and numerically apportioned into, exactly four distinct and additive types. That led me to creating four distinct indices of misalignment that specify the relative sizes of the four types of misalignment in any given Porter alignment calculation. I proposed how these four new indices of misalignment could be calculated and presented in a way that educators can reliably duplicate in their own alignment studies.

The motivation for this dissertation then revolved around two research questions both concerning how useful and worthwhile these misalignment indices were. More specifically, the first research question asked how informative or consequential to the conclusions were these misalignment indices when applied to real data sets from previously published alignment studies; while the second research question asked how sensitive or robust were these misalignment indices under different input parameters in Porter’s alignment methodology.

For the first research question, I calculated these four new indices of misalignment for 13 previously published PAI alignment studies that listed their raw data (i.e. their Porter’s matrices of proportions used to calculate their PAI values). For some, not all, of these studies, such calculations of misalignment indices were informative and possibly consequential to the conclusions. For those studies, researchers could use this new information to revisit conclusions about the alignment of components depending on context, perform various post hoc analyses based on varying the granularity of topics or levels of cognitive demand, or revise components in a strategic way to improve alignment if that was the goal. Such information could be helpful to educators in general because two components may have the same PAI value (i.e. the same total misalignment) to a third component in mind all while the nature of their misalignment to that component might differ in ways that the educators would care about.

For the second research question, I performed simulations in R to test the effects of varying input parameters (e.g. characteristics of an assessment such as number of items or number of topics) on the alignment of an assessment to a standard. I had some intuition about how PAI would respond to varying input parameters but not how the misalignment indices would. I wanted to see if PAI plus the four new indices of misalignment provided more insight into alignment that PAI alone. I wanted to see how sensitive or robust the misalignment indices were and how they changed compared to how PAI changed. For some, not all, of these simulations, misalignment indices did indeed give a more detailed view of alignment. Since alignment (as measured by PAI) and misalignment (as measured by the sum of the four misalignment indices) add up to one, a change in PAI can be traced back to a net change in one, two, three, or all four of the misalignment indices. Again, such detail helpful to educators in general because educators might have different preferences or tolerances for misalignment depending on context.

This dissertation advocates for researchers to not just report PAI values but also include their raw data of Porter matrices whenever conducting an alignment study using Porter’s methodology. This dissertation also advocates that researcher calculate the four indices of misalignment to report alongside PAI. This dissertation commends Porter’s methodology, calls for its continual use, and encourages expanding it to include a breakdown of misalignment into its four types. While PAI is the prevailing quantitative measure of alignment, and while it is based on a sound and practical methodology with the most logical assumptions, there is hopefully still some benefit in investigating and dissecting misalignment into its types towards the goal of further empowering educators when selecting or editing components based on alignment.

Share

COinS