An Investigation of Metrics For the in-situ Detection of Software Expertise

AbstractTask-based analysis is a common and effective way to measure expertise levels of software users. However, such assessments typically require in-person laboratory studies, and inherently require knowledge of the user’s task. Today, there is no accepted method for assessing a user’s expertise levels outside of a lab, during a user’s own home or work environment activities. In this article, we explore the feasibility of software applications automatically inferring a user’s expertise levels, based on the user’s in-situ usage patterns. We outline the potential usage metrics that may be indicative of expertise levels, and then perform a study, where we capture such metrics, by installing logging software in the participants’ own workplace environments. We then invite those participants into a laboratory study, and perform a more traditional task-based assessment of expertise. Our analysis of the study examines if metrics captured in-situ, without any task knowledge, can be indicative of user expertise levels. The results show the existence of significant correlations between metrics calculated from in-situ usage logs, and task-based user expertise assessments from our laboratory study. We discuss the implications of the results, and how future software applications may be able to measure and leverage knowledge of the expertise of its users.