Topic: Construct Validity and Machine Learning
Session Lead: Dr. Nigel Bosch
Time:  2022-10-05, Wednesday, 11 am – 12 pm (CDT)
Location: Zoom

Abstract: 
How do we know what a machine learning model predicts? The answer may seem obvious: it predicts whatever the labels are! However, the reality is more complicated. Machine learning models make predictions that are only an approximation of what we really want. Moreover, predictions may approximate things we do not want (e.g., demographic characteristics that are supposed to be orthogonal to predictions). Consequently, it is worthwhile to understand what models actually predict. Fortunately, there is well over 60 years of research on a closely related topic: how do we know what an educational test measures? In this talk, I will provide an overview of some epistemological, ethical, and legal issues stemming from educational testing and how they (very directly) apply to machine learning. I will also describe methodological techniques that can be applied to assess machine learning models and educational tests alike to examine the degree to which they predict what we want to predict — i.e., their “construct validity.”