Guidelines for Maintaining Systematic Literature Reviews in Software Engineering

Abstract

Systematic Literature Review (SLR) is a process in which a representative set of relevant available research about a research question, or topic, or phenomenon of interest is identified, evaluated, and interpreted through their individual studies, called primary studies, during SLR process. Kitchenham’s guidelines summarize systematic review process into three main phases: Planning, Conducting, and Reporting results. We argue SLR process should not end after reporting phase. The main purpose of maintain SLRs is to keep your evidence as up-to-date as possible. The main goal of this research is, based on researchers opinion who have already performed SLRs, to propose and evaluate a guideline to carry out systematic literature reviews maintenance in software engineering field. In this work we make a parallel with software maintenance concepts, adapting them to SLR concepts. Understanding what kind of changes can lead to a SLR maintenance, how to assess changes impact and how process should be conducted. Three steps were carried out: Investigation, Proposition, and Evaluation. In investigation step, two studies were conducted in parallel to serve as a basis for our work. First we conduct a systematic mapping (SM) to address the state-of-the-art on systematic literature reviews update, then we conduct a survey aimed to identify researchers opinion about maintenance of SLRs. In proposition step, we aim to define a guideline for conducting maintenance in systematic literature reviews. In evaluation step, to evaluate the proposed guideline, we conducted a series of semi-structured interviews. During SM, it was possible to check secondary studies updates since 2010, however if we compare with the amount of SLR studies already published, there is still a long way to go. We found that surveyed researchers have shown interest in keeping their SLRs up-to-date, but they have expressed concerns about the effort to be made to accomplish it. During interview the guideline was presented and process acceptance is perceived as basis for conducting SLR maintenance activities. Various contributions have been taken into account and incorporated into process. The guideline proposed was well accepted by the researchers, and besides the evaluation of its effectiveness as a guide for SLRs maintenance process, it was necessary to elaborate several discussions, involving the theme, for a better understanding about the topic. Important concepts were presented and validated during research. However, this is a first glimpse into SLRs maintaining process, and like any process formalization, still lacks depth. Some gaps in compliance or formalization may still be left blank. However, it is hoped with future community collaborations, this guideline may become a basis for conducting maintenance activities on SLRs.