Evaluating User Engagement with a Reminiscence App Using Cross-Comparative Analysis of User Event Logs and Qualitative Data

The aim of this study was to evaluate the usage of a reminiscence app by people living with dementia and their family carers, by comparing event log data generated from app usage alongside the qualitative experience of the process. A cross-comparative analysis of electronic event logging data with qualitative interview data was conducted. Electronic event logging data were obtained for 28 participating dyads (n = 56) and the interview sample comprised 14 people living with dementia and 16 family carers (n = 30). A thematic analysis framework was used in the analysis of interview transcripts and the identification of recurrent themes. The cross-comparison of electronic event log data and qualitative data revealed 25 out of 28 dyads regularly engaged with a reminiscence app, with the analysis of usage patterns revealing four clusters classifying different levels of user engagement. The cross-comparison of data revealed that the nature of the relationship was a significant factor in ongoing user engagement. The comparative analysis of the electronic event logs as "ground truth" in combination with the qualitative lived experience can provide a deeper understanding on the usage of a reminiscence app for those living with dementia and their family carers. This work not only shows the benefits of using automated event log data mining but also shows its clear limitations without using complementary qualitative data analysis. As such, this work also provides key insights into using mixed methods for evaluating human-computer interaction technologies.