The Impact of Computing Environment on Test Flakiness

Abstract

A test is said to be flaky when it non-deterministically passes or fails on the same code. This phenomenon undermines trust in testing and constitutes a long-standing problem in software engineering. The central thesis of this work is that flaky tests can emerge from broken assumptions that developers implicitly make about the execution environment. Tests may rely on assumptions about available resources, file system conventions, or platform behavior that are not consistently satisfied across environments. When those assumptions are violated, environmental mismatches trigger non-deterministic test outcomes. To investigate this thesis, this work considers two complementary perspectives on environmental variation. The first concerns variability in computational resources (CPU, memory, network, and disk) and its impact on test execution. The second concerns portability across platforms and operating systems, analyzing how differences in system behavior can induce flaky test outcomes. Together, these perspectives characterize how environmental assumptions may lead to flakiness through resource variability within a platform and behavioral differences across platforms. For resource variability, an empirical study across 52 Java, JavaScript, and Python projects shows that 46.5% (283/608) of flaky tests qualify as Resource-Affected Flaky Tests (RAFTs), whose non-deterministic behavior is caused by resource (un)-availability. CPU constraints show the strongest association with elevated failure rates, with memory constraints showing a secondary association, while disk and network constraints show limited association. Stricter configurations increase failure detection, although the most effective configuration varies across projects and often corresponds to intermediate setups (e.g., CPU 0.5/RAM 2GiB and CPU 2/RAM 4GiB). For cross-platform portability, cross-OS re-execution experiments across 500 projects (Linux, macOS, and Windows) reveal that 11.2% exhibit portability-related failures. Mining GitHub Issues identified additional cases in 95 projects, totaling 151 affected systems. From these observations, we derived a taxonomy comprising 7 categories and 24 subcategories, with file operations, process management, and library behavior being the most prevalent. Four general repair patterns were identified, with environment handling (35.7%) and defensive checks (32.2%) being the most common. In tool evaluation, static analyzers demonstrated limited coverage; large language models achieved detection accuracy between 40–79% and repair accuracy between 27–43% with generic prompts, improving to 50–77% with structured prompting. Based on these findings, this work introduces tools to support the detection, diagnosis, and mitigation of computing environment assumptions in testing. These tools assist practitioners and researchers in identifying resource-sensitive tests and cross-platform compatibility issues. The practical relevance of the work is demonstrated through 142 pull requests and issue reports submitted to open-source projects, of which 63 were accepted.