Web Analytics

Research Shows Strong Impact of Software Quality on Cost

September 2010
Software of low quality resists change. This resistance is felt when finding and fixing bugs and when implementing enhancements. If software is more complex, unstructured, or tangled, then more time and effort needs to be spent to make these changes. But how much time and effort can be saved by bringing quality at a higher level? In collaboration with the Technical University of Delft and the University of Amsterdam, the SIG research department has found answers to these questions.
The research has focused on the maintenance history of a large number of open source systems. We determined how many issues (defects and enhancements) the maintainers of these systems were able to resolve, how fast they did so, and with how much effort. Based on this, we calculated four indicators for issue handling performance from the extracted data:

Time   = how much time elapses between submission and resolution of issues?
Throughput   = how many issues are resolved per month per 100,000 lines of code?
Productivity   = how many issues are resolved per month per developer?
Efficiency   = how many of the issues that are resolved are of type enhancement?

For over 150 versions of the studied open source systems, the values of these indicators were calculated, together with their star ratings according to the SIG quality model. The amount of improvement that we found in each indicator for various quality levels is shown in five charts.

Let’s see what we can learn from these charts.

Chart 1 and 2
Issue resolution time increases exponentially as quality drops
Chart 1 and Chart 2 show that the issue resolution times for defects and enhancements get exponentially longer with each level of decrease in source code quality. For example, solving a defect in a 4-star system takes usually less then a week (6.5 days), but in a 2-star system more than three weeks (22.4 days).

Efficiency increases 13 percent points per quality level
Chart 3   Chart 3 shows that in systems of average quality (3 stars) developers or maintainers need to correct 3 bugs for every enhancement that they implement, which puts efficiency at 25%. When code quality slides down to 2 stars, efficiency drops to about 13% (almost 8 defects for every enhancement). For 4 star systems, on the other hand, efficiency shoots up to 39% (about 2.5 defects for every enhancement).

Throughput drops with quality, but bottoms out
Chart 4   Chart 4 shows that throughput drops steeply from 56 issues resolved per month, per 100,000 lines of code to about 9 resolved issues (6-fold decrease) as quality falls from 4 stars to 3 stars. When quality drops another star, throughput shrinks only marginally to about 8 issues.

Productivity increases significantly per quality level
Chart 5   Chart 5 shows that about 1.2 issues are solved per month per developer for systems that are rated with 3-star quality. For 4-star systems, productivity lies twice as high at about 2.3 issues per month. In a 2-star systems, about 1/5th of an issue is resolved per month, per developer (5-fold decrease). In other words, both productivity and throughput drop when going from 4 to 3 stars. But unlike throughput, productivity keeps falling when going from 3 to 2 stars.

The impact of demanding 4-star software
We recommend our clients to demand 4-star software from their development teams or external suppliers. This joint research with TU Delft and University of Amsterdam provides an indication of the impact that such quality requirements may have. In comparison to a system of below-average quality (2 stars):
  • Issues are resolved 3 times faster
  • Efficiency increases 26 percent points
  • Throughput increases 7-fold
  • Productivity increases almost 11-fold
Of course, many additional factors are at play and these may influence the exact gains for specific projects. We will keep you informed about our ongoing research to identify and qualify these factors. In any case, our research so far has shown that the star ratings of the SIG quality model have a strong predictive power for the maintenance burden that is associated to the system.

  • The Influence of Software Maintainability on Issue Handling, Bart Luijten, MSc thesis, Technical University Delft, 2010
  • Indicators of Issue Handling Efficiency and their Relation to Software Maintainability, Dennis Bijlsma, MSc thesis, University of Amsterdam, 2010

Copyright: © 2014 Software Improvement Group