Sound test case generation for concurrent features combining test cases for individual features

Abstract

In the current landscape of software development, where applications are becoming increas- ingly intricate and designed to handle multiple tasks simultaneously, it is essential to validate reliability under concurrent conditions. Systems can exhibit a wide range of interactions and behaviours that are difficult to replicate, making the creation of effective testing strategies extremely challenging. Additionally, generating test cases for concurrent systems is demanding due to the lack of explicit descriptions of their concurrent behaviour in the typically captured natural language requirements. Our primary contribution involves an approach for generating consistent tests based on requirements expressed in natural language, with a particular focus on mobile device applications. This approach is enhanced with a dependency analysis strategy that ensures a consistent order of test steps execution, thereby eliminating incomplete test cases or those that cannot be executed due to unmet preconditions. Furthermore, we address the soundness of the proposed approach through the introduction of a new conformance relation, denoted as cspioq. This new relation effectively handles the absence of outputs (quiescence), aiming to ensure that the system can adequately handle scenarios where no further outputs or events are expected, which is a common characteristic in concurrent systems. When up- dated requirements are not available, which is often the case in an industrial context, a reverse engineering process is necessary to generate requirements from existing test cases, in order to allow the proof of soundness of test case generation from requirements. We explore this approach to define a sound test case generation strategy that considers quiescence. Neverthe- less, from a practical (implementation) point of view, this process is rather burdensome. As an alternative, we propose an optimised test generation strategy through the permutation of test steps, referred to as atoms. This strategy aims to simplify the process by directly extracting new test cases from existing ones without the need for a complex reverse engineering process. We also address the soundness of the optimised approach by demonstrating its connection with the original approach based on reverse engineering. We fully implemented tool support and conducted an empirical evaluation of the generated test effectiveness. We analysed test coverage and the number of bugs during the execution of tests created by engineers from our industrial partner, Motorola Mobility (a Lenovo company). The adopted metrics were then compared with those obtained from tests generated using the proposed approach. The results reveal that the test set produced by our approach exhibits significantly greater coverage and has the potential to identify more bugs compared to the set created by Motorola engineers.