Benchmark-based Aggregation of Metrics to Ratings
In this joint publication of SIG and Universidade do Minho, the SIG method for calibrating star rating thresholds is explained in full detail. This methodology, together with the methodology to derive metric thresholds is applied annually to update the SIG quality model against the industry reference data in the SIG benchmark repository.
Benchmark-based Aggregation of Metrics to Ratings
By: Tiago L. Alves (SIG & Universidade do Minho), José Pedro Correia (SIG) and Joost Visser (SIG)
Published in: The Joint Conference of the 21st International Workshop on Software Measurement (IWSM) and the 6th International Conference on Software Process and Product Measurement (Mensura), November 3-4, 2011.
Abstract:
Software metrics have been proposed as instruments, not only to guide individual developers in their coding tasks, but also to obtain high-level quality indicators for entire software systems. Such system-level indicators are intended to enable meaningful comparisons among systems or to serve as triggers for a deeper analysis.
Common methods for aggregation range from simple mathematical operations (e.g. addition and central tendency) to more complex methodologies such as distribution fitting, wealth inequality metrics (e.g. Gini coefficient and Theil Index) and custom formulae. However, these methodologies provide little guidance for interpreting the aggregated results or to trace back to individual measurements. To resolve such limitations, a two- stage rating approach has been proposed where (i) measurement values are compared to thresholds to summarize them into risk profiles, and (ii) risk profiles are mapped to ratings.
In this paper, we extend our approach for deriving met- ric thresholds from benchmark data into a methodology for benchmark-based calibration of two-stage aggregation of metrics into ratings. We explain the core algorithm of the methodology and we demonstrate its application to various metrics of the SIG quality model, using a benchmark of 100 software systems. We present an evaluation of the sensitivity of the algorithm to the underlying data.
Download publication: