Evaluating Concepts, Evaluating Measures: The Case of User Engagement and the User Engagement Scale

Various methodological approaches are utilized in user engagement research, including self-report methods (e.g., questionnaires, verbal elicitations), neurophysiological methods (e.g., eye tracking, facial expressions), and observational methods of user behavior (e.g., mouse clicks, navigation patterns). Yet, seldom do we evaluate the methods and measures themselves. This talk will draw upon the User Engagement Scale (UES), a self-report measure developed by O’Brien, to focus on two intersecting and fundamental challenges: 1) How do we operationalize and measure multi-dimensional, complex, subjective concepts such as user engagement? and 2) How do we evaluate the robustness of such measures?

O’Brien will explore the UES’ reliability, validity and generalizability by examining its uptake and utilization to investigate user engagement with search, news, video, education, haptic, social networking, consumer, and video game applications. She will argue that evaluating concepts requires a parallel focus on the evaluation of measures designed to capture concepts. The question of whether we are measuring what we think we are measuring has serious implications for human computer interaction studies and phenomena in general, and user engagement more specifically, such as how we know users’ have experienced engagement and what system, user, and contextual factors precipitated or deterred it.

Heather L. O’Brien, PhD, is an assistant professor at the iSchool, University of British Columbia, Vancouver, Canada, where she teaches and researches in the areas of human information interaction, user experience and user engagement. Over the past ten years she has focused on the theoretical aspects of user engagement with computer-mediated search, news, shopping and education technologies. This research has grounded the concept of user engagement, and proposed and tested a Process Model of User Engagement. A second major stream of O’Brien’s research is the measurement of user engagement, specifically how to measure user engagement holistically using multiple subjective and objective approaches. She developed a self-report experiential questionnaire, the User Engagement Scale (UES), which has been adopted and utilized by multi-disciplinary researchers around the world. O’Brien is the author of multiple journal articles, conference papers, and the book Measuring User Engagement, published with Mounia Lalmas and Elad Yom-Tov in 2014 (Morgan Claypool). The book Why Engagement Matters: Cross-Disciplinary Perspectives and Innovations on User Engagement with Digital Media, a co- edited manuscript with Paul Cairns (Springer, in press), is the first comprehensive text on user engagement. More information about O’Brien’s research, teaching and publications can be found on her website.