A multimethod study of test smells : cataloging, removal, and new types

Abstract

Test smells are symptoms in the test code that indicate possible design or implementation problems. Their presence in automated test suites, along with their harmfulness, has already been demonstrated by previous research. Although test smells have been the subject of much gray and academic literature since their proposal in 2001, many questions regarding their ad- herence in the industry are yet to be clarified: concerning test smells — proposed by numerous studies and gray literature — no publicly available catalog aggregates them; considering the evolution of test frameworks and programming languages, there is no correspondence between the newly proposed features and their capability of refactoring or preventing test smells; finally, considering that test automation requires a significant initial investment not always available to software projects, little is known to the possibility of test smells’ existence in manual test suites, as well as how to identify and remove them. This work presents a multimethod study aimed at fulfilling these knowledge gaps in the test smells area, which comprises surveying state of the art on test smells and refactoring actions, the use of manual and automatic anal- yses of open-source repositories, the conduction of surveys with software testing professionals, the study of new test framework features and the proposition of test smell refactoring actions, and the submission of contributions to active and popular open-source software projects. The results present: (i) a catalog that unifies 127 primary studies and 480 distinct test smells in a previously unseen effort; (ii) the confirmation that new test framework features can refactor and prevent test smells, where we propose and evaluate new refactorings based on 7 JUnit 5 features intended for 13 test smells; (iii) the proposition of a catalog containing 8 new test smells specific to manual test suites, their identification strategies based on natural language processing, and their frequency in important government, industry and open-source systems. The findings of this work give directions for further development in various fronts of the test smells study area.