Enshrining Good Enough: Code Regression Testing

May 7, 2020

One of DarkCyber’s research team forwarded me the PDF of “Measuring the Cost of Regression Testing in Practice: A Study of Java Projects Using Continuous Regression Integration.” Note: this may require you to pay to access this PDF.

The paper is narrow and written for engineers who have some involvement in making sure that the button one taps delivers the expected result. DarkCyber is deeply skeptical of software today. Consider the Windows 10 updates, the Apple OSX updates, and the less thrilling nuclear power plant issues; for example, “Most NE Nuclear Power Offline Due to Timing Fluke and Problem.”

The point is that software has bugs, and fixing these is expensive. Despite the claims of the artificial intelligence start ups, humans are needed. Machine intelligence is an admission that there are humans behind the scenes. Imagine that!

I want to cut to the conclusions of the study. Here’s an edited version:

Regression testing is widely used and widely studied. Despite this, it is not always clear that the benefits of having fewer faults in the program are outweighed by the cost of writing, maintaining, and executing regression tests….We studied 61 Java-based projects that use Travis CI. We found that 18% of test suite executions fail and that 13% of these failures are flaky. Of the non-flaky failures, only 74% were caused by a bug in the system under test; the remaining 26% were due to incorrect or obsolete tests. In addition, we found that, in the failed builds, only 0.38% of the test case executions failed and 64% of failed builds contained more than one failed test. Our findings contribute to a wider understanding of the unforeseen costs that can impact the overall cost effectiveness of regression testing in practice.

Translating: Don’t fix some bugs. Too expensive and the software will be “good enough.”

Net net: Get used to increasingly flawed software. For more coverage of “good enough” in the context of SEO, click this link.

Stephen E Arnold, May 6, 2020

Written by Stephen E. Arnold · Filed Under AI, News

Comments

Comments are closed.

Search the site
Subscribe to Beyond Search
Feature archive
News archive

Stephen E. Arnold monitors search, content processing, text mining and related topics from his high-tech nerve center in rural Kentucky. He tries to winnow the goose feathers from the giblets. He works with colleagues worldwide to make this Web log useful to those who want to go "beyond search". Contact him at sa [at] arnoldit.com. His Web site with additional information about search is arnoldit.com.