DeFlaker: automatically detecting flaky tests

摘要

One of the final steps in producing a new version of a software tool is to make sure the old functionality was not lost when the new functionality was added. A common approach to this problem is regression testing, that is, running the test cases used for testing the old functionality on the new version. If a regression test fails, further expensive and time-consuming investigation and/or additional testing may be needed to make sure that the old functionality is still viable. However, having one of the regression test cases fail may not actually mean that old functionality was lost. The test may be a flaky test, that is, a test case that "can non-deterministically pass or fail when run on the same version of the code."