Various methodological approaches are utilized in user engagement research, including self-report methods (e.g., questionnaires, verbal elicitations), neurophysiological methods (e.g., eye tracking, facial expressions), and observational methods of user behavior (e.g., mouse clicks, navigation patterns). Yet, seldom do we evaluate the methods and measures themselves. This talk will draw upon the User Engagement Scale (UES), a self-report measure developed by O’Brien, to focus on two intersecting and fundamental challenges: 1) How do we operationalize and measure multi-dimensional, complex, subjective concepts such as user engagement? and 2) How do we evaluate the robustness of such measures?