Investigations about replication of empirical studies in software engineering: findings from a mapping study

Abstract

Context: Two recent mapping studies which were intended to verify the current state of replication of empirical studies in Software engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994- 2012) and a second group of studies that are concerned with definitions, classifications, processes, guidelines, and other research topics or themes about replication work in empirical software engineering research (published between 1996- 2012). Objective: The goal of this work is to analyse and discuss the contents of the second set of studies about replications to increase our understanding of the current state of the work on replication in empirical software engineering research. Method: The systematic literature review method was applied to build a systematic mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by manual and automatic search procedures that collected articles published in 2013. Results: We analysed 37 papers reporting studies about replication published in the last 17 years. These papers explore different topics related to concepts and classifications, guidelines, and discuss theoretical issues that are relevant for our understanding of replication in our field. We also investigated how these 37 papers have been cited in the 135 replication papers published between 1994 and 2012. Conclusions: Replication in SE still lacks a set of standardized concepts and terminology, which has a negative impact on the replication work in our field. To improve this situation, it is important that the SE research community engage on an effort to create and evaluate taxonomy, frameworks, guidelines, and methodologies to fully support the development of replications.