Looking Inside - Mobile Screen Recordings as a Privacy Friendly Long-Term Data Source to Analyze User Behavior

Abstract

Mobile devices are ubiquitous in many societies and shape the way we interact with technology and each other. Research on how we use and perceive technology is essential to understand its impact. This work advances how we can follow user behavior on mobile devices. We combine the strength of two common data sources for tracking on mobile devices, log files, and screen recordings. Log files are suitable for long-term and privacy-friendly analyzation but provide rather general data (e.g. system log files) unless one has access to the source code of the applications or operating systems. Screen recordings are usually used for short-termed analysis (e.g. usability tests) because the analysis is time-consuming, but they provide all activities on the screen in high detail regardless of which application or operating system. This thesis combines both data sources and presents an approach to automatically generate log files from mobile screen recordings. The approach utilizes methods of computer vision and machine learning to automatically process screen recordings and extend their use. Screen recordings reveal virtually everything a user does with a device, making privacy important, especially in user studies. We present a privacy concept and implementation and show how the risk of exposing private data can be reduced, by processing all recordings locally on the mobile devices and anonymizing the resulting log files. In order to apply the developed method in practice, we carry out a study in the context of education and show how log files of screen recordings can complement and extend existing research in learning analytics. This thesis opens up novel perspectives on how we can look at human-computer interaction with mobile devices. We show how to generate long-term log data with high detail and accuracy from mobile screen recordings, in a privacy-friendly way, locally on mobile devices.