Dissertations, Theses, and Capstone Projects

Date of Degree

2-2020

Document Type

Dissertation

Degree Name

Ph.D.

Program

Educational Psychology

Advisor

Keith Markus

Committee Members

David Rindskopf

Jay Verkuilen

Matthew Johnson

Charles Scherbaum

Subject Categories

Quantitative Psychology

Keywords

construct validity, construct-irrelevant variance, construct under-representation, test validity

Abstract

Test validity has commonly been defined in terms of a test measuring what it is supposed to measure, however there has been a century of discussion and debate on test validity and what qualifies as evidence of test validity. Some argue in favor of restricting test validity evidence to that which is internal to the test, such as the test items, while others argue that external information, such as the relationships with other variables, also offers evidence toward test validity. Cronbach and Meehl’s groundbreaking paper in 1955 introduced the concept of the nomological net, which is the construct’s lawful relationships with other variables, and they suggestedthat the nomological net provides validity evidence. Using a simulation study, I tested if relationships to other variables can offer information about test validity by looking at the ability to identify two sources of invalidity. Two simulation studies wereconducted to test the two types of construct invalidity proposed by Messick (1989): construct under-representation and construct-irrelevant variance. Each used a 4 × 4 × 4× 4×2 design, where four levels of factor loadings, structural coefficients, and sample size were analyzed along with four different mediation SEM models in two different validity conditions. Data was simulated using R to analyze whether we can detect construct invalidity using model fit. The results demonstrated that we can detect both construct under-representation and construct-irrelevant variance, particularly when structural coefficients and factor loadings were large. These findings were then illustrated using a real-data example, where the models again fit less well when sources of invalidity were deliberately introduced into the data. These findings support the broad view of test validity, which includes external evidence in the definition of test validity.

Download

Share

COinS