Jennifer Case PhD February 2020

Educational Psychology
Learning, Development, and Instruction

Test Validity, Social Skills Development

Dr Jennifer Case is the Associate Director of In Step, PC, a private mental health practice in Northern Virginia. In Step serves children and adults with a variety of mental health challenges from ADHD to BPD. Jennifer works to ensure clients' needs are met, while keeping In Step on the forefront of mental health treatment. This includes reviewing literature on treatment modalities to formulate new groups and conducting research on the efficacy of In Step's programs. Jennifer is a proud mother and active in her local community helping recovering addicts and the homeless..

Dissertation Title:

"Identifying Construct-Irrelevant Variance and Construct Under-Representation as Support of External Evidence of Validity"

Test validity has commonly been defined in terms of a test measuring what it is supposed to measure, however there has been a century of discussion and debate on test validity and what qualifies as evidence of test validity. Some argue in favor of restricting test validity evidence to that which is internal to the test, such as the test items, while others argue that external information, such as the relationships with other variables, also offers evidence toward test validity. Cronbach and Meehl’s groundbreaking paper in 1955 introduced the concept of the nomological net, which is the construct’s lawful relationships with other variables, and they suggested that the nomological net provides validity evidence. Using a simulation study, I tested if relationships to other variables can offer information about test validity by looking at the ability to identify two sources of invalidity. Two simulation studies were conducted to test the two types of construct invalidity proposed by Messick (1989): construct under-representation and construct-irrelevant variance. Each used a 4 × 4 × 4 × 4 × 2 design, where four levels of factor loadings, structural coefficients, and sample size were analyzed along with four different mediation SEM models in two different validity conditions. Data was simulated using R to analyze whether we can detect construct invalidity using model fit. The results demonstrated that we can detect both construct under-representation and construct-irrelevant variance, particularly when structural coefficients and factor loadings were large. These findings were then illustrated using a real-data example, where the models again fit less well when sources of invalidity were deliberately introduced into the data. These findings support the broad view of test validity, which includes external evidence in the definition of test validity.

Committee Members:

Dr. Keith Markus (chairperson); Dr. Jay Verkuilen; Dr. David Rindskopf; Dr. Matthew Johnson; Dr. Charles Scherbaum

Acknowledgements / Special Thanks:

It truly took a village to make this dissertation and degree possible. I would like to express my deepest appreciation to my advisor, Dr. Keith Markus. Without his unwavering support, guidance, and patience this would have remained a lost dream. I am also extremely grateful to my committee for the invaluable feedback and advice to make this the best possible research project.

I would like to extend my sincere thanks to my colleagues at In Step, who supported and nurtured me during this endeavor, especially Cathi Cohen. Her belief in me helped me believe in myself and her selfless support gave me strength along with a role model and friend. My colleagues, Bob, Jim, Constantina, Dale, Mary, Erika, and Ashleigh were the greatest cheerleaders.

I am blessed with the greatest family in the world. I cannot begin to express my thanks to my parents who supported me during this entire journey and spent countless hours babysitting. I am deeply indebted to my partner, Matthew Kocher, who picked up my slack around all areas of our lives while all my extra time and energy went into my dissertation. More importantly, his love and support gave me strength. I am extremely grateful to Kat Kocher for the serenity and so much more.

None of this would be possibly without the work of Jimmy K. His vision and footwork gave me a life I only dreamed of and introduced me to my closest friends and support. Valerie Hadeed mentored me through this process and never wavered in her belief of me; I would be lost without her. A special thank you to Melissa Hash for praying for me by name, I think it worked!

This is for you Clementine. You have given my life more love and purpose than I ever imagined.