In addition to defect removal tracking discussed in the previous section (1.2.2.4), other defect measures can be useful in assessing whether targets are met as well as facilitating software project and process management in general.  This section describes some measures and methods for selecting measures that can be useful in defect tracking against quality targets.

The Practical Software and Systems Measurement (PSM) Guide

     An excellent source of measures for software projects in general, including quality measures, is the Practical Software and Systems Measurement (PSM) Guide [Mcgarry, 2002].  Section 3 of this Guide explains the Issue-Category-Measure (ICM) approach to selecting measures for software projects.  The Guide lists more than 50 measures and provides a detailed description of each measure.  Table 16, from the PSM Guide, shows measures that can be used for selected issues and categories within issues.   Some of the measures that can be used in defect tracking against quality targets are now summarized.
 

1. Defects:  This refers to the number, status, and priority of defects reported.  It can include the average age of defects, which is related to the defect removal rate discussed earlier.  The defect status can include the closure rate of defects, also an indicator of defect removal rate.
2. Operator errors:  This measure is useful in assessing quality in the operational environment, after a system or program has been deployed.   It can be used to measure usability and quality of a system or program, and can be compared to any targets set in these areas.  The limitation of this measure is that it is only useful after a system has been deployed.
3. Failures:  This is a measure of the number, criticality, and time interval between failures.  It is useful in measuring software reliability in terms of mean-time-to-failure (MTTF) and can be compared to MTTF targets if they are established.  Like operator errors, this measure is useful only after system deployment.
4. Defect Containment:  As discussed in Sections 1.2.2.1 and 1.2.2.3, it is highly desirable to detect defects during the phase when they occur, or are inserted.  It is also highly desirable to remove, or correct defects during the phase when they are detected. The defect containment measure, as discussed in Section 3 of the PSM Guide, measures the number of defects found during a process (or phase) and the number of defects found after completion of a process, or phase, such as during a design review or inspection.  This measure can be used to determine the effectiveness of inspections and other defect detection methods used, and can be compared to any targets or goals set for these methods.  This measure could be expanded to include defect removal tracking, as discussed in Section 1.2.2.4.
5. Rework: This measure tracks the amount of effort required to fix, or remove defects.  An organization sometimes sets a budget for rework activities, and this measure can be used to compare actual rework effort to budgeted rework effort.  As discussed in previous sections, it is desirable to detect and remove defects as early as possible to reduce rework required.  This measure can show how well an organization is performing in this area.
6. Survey Results:  Survey Results track customer satisfaction ratings, from both acquiring agencies and users. Survey Results also measures progress in providing products and services that meet the customer’s needs and expectations.  One challenge with this metric is setting an objective scale for measurement.  Often, users are asked to rate a product on a scale of "1" to "5", or "Very Satisfied", "Satisfied", etc.  Even with a seemingly objective scale, users' ratings will often be based on subjective perceptions.  Also, this measure, like operator errors and failures, is not useful until after deployment.  Still, organizations sometimes set quality targets in terms of user satisfaction ratings, especially in the commercial environment, and this measure is useful in assessing successes in this area.

 Figure 11: Illustration of the GQM Approach [Ferens, 2010]

     Van Solingen [Van Solingen, 1999] provides an example of a quality goal that addresses software reliability.  His statement of the reliability goal is "Analyze the delivered product and development process for the purpose of characterizing with respect to reliability and its causes from the viewpoint of the software development team in the context of Project "A".  Van Solingen then proposes 18 questions and one or more metrics, or measures, for each question.  For example, Question 1 is "What is the composition of the product?", and three metrics are a list of life cycle documents, a list of subsystems, and a list of modules. 

     User satisfaction measures include ease of use, functionality, and support.  These are "soft" measures since they are based on user perspectives and obtained by surveys, usually taken once a year.  If user satisfaction measures are included in quality targets, these measures can be useful, since some definitions of quality include "high degrees of user satisfaction" and "ease of use".   One limitation of user satisfaction measures are they are not available until after a program is completed.

       Defect measures include defect volumes (e.g., number of defects per function point), defect severities, defect origins, defect causes, and (as discussed in Section 1.2.2.4) defect removal efficiency.  According to Capers Jones, defect removal efficiency is a measure used by leading edge companies.  The defect measures are "hard" measures that can be assessed objectively and more often (once a month) during a program.

     In addition to recommending quality measures to use, Capers Jones explains there are certain quality measures not to use!    Cost per defect is one example; it can actually penalize high quality products that have relatively fewer defects!  Table 17 shows a comparison of costs per defect for a low-quality "buggy" application and a high quality application of the same size.  The cost per defect is actually a combination of three costs: preparation costs, execution costs, and repair costs.  Preparation costs tend to be fixed; for example, the cost to create test cases for testing or reading specifications for a design review is the same no matter how many defects the product contains.  Execution costs, the costs of actually conducting a review, audit, or test, are "semi-variable"; there are certain costs associated with going through an exercise no matter how many defects are present, while there are costs such as recording defects found that increase with number of defects.  Lastly, repair costs are the costs of fixing or removing defects; these costs vary directly with the number of defects that must be corrected.  As shown in Table 17, for two similar sized programs, cost per defect looks better for the low quality program; however, the total costs are much lower for the higher quality program.

    Kan [Kan, 1995] provides a list of measures that resulted from studying quality efforts for several companies.  Kan's list contains 13 measures in three categories, as shown in Table 18.

      As explained by Fenton and Pfleeger [Fenton, 1997], most customers know about the area for which a software product is being developed, but little about compilers, programming languages, and other aspects of software development.  Therefore, measurements must be presented in a way that tells both customers and developers how a project is doing.  For any program or project, it is important to report progress of the effort, including quality concerns, to project or program managers, other senior-level managers, and other stakeholders in a program or project.  As explained by Schwalbe [Schwalbe, 2006], a stakeholder can be anyone who has an interest in a project, including a competitor.   (Of course, the information a company reports to competitors should be minimal!)  Schwalbe goes on to explain that different types and levels of reporting will be made to different stakeholders.  In fact, Schwalbe recommends developing a stakeholder matrix for a project, which lists key stakeholders and their personalities so reports can be tailored to individuals.  Although all stakeholders are important, this section concentrates on reports to management.

     We have already discussed some measures that are useful for Questions 3 and 4 above; many defect-related measures are useful in assessing code quality, and user satisfaction measures can be used to assess how content users are with products.  Some of the measures can also be used to answer the other questions.  (More information about process improvement is included in Section 1.3.3.)  This section addresses not only what measures should be reported to management, but how they should be reported.

     For the CMMI, one of the Process Areas, "Process and Product Quality Assurance", involves providing feedback to managers on the results of quality assurance activities  [Konrad, 2003].  Table 19 lists four specific practices within this Process Area, and some results of these practices that should be reported to management.  (More about the CMMI is in Section 4.0).

This section contains summary and detailed characteristics, and benefits that result from using this Gold Practice.

To effectively implement this Gold Practice, organizations must be capable of both setting quality targets (or identifying quality gates), and handling defect tracking. This includes detecting, analyzing, correcting, and reporting defects. The Software Engineering Institute (SEI) Capability Maturity Model Integration for Development (CMMI-DEV), discussed further in Section 4, includes some specific requirements in three Process Areas: Product and Process Quality Assurance, Quantitative Project Management, and Causal Analysis and Resolution [Konrad, 2003].  Chapter 11 of the Software Engineering Body of Knowledge (SWEBOK) explains prerequisites for an organization to properly address software quality [Abran, 2004]. 

“The cost impact on an individual software engineer during the development process is minimal. Typically, measured in the order of a few dozen keystrokes per defect to fill out one or two panels. The incremental time is probably negligible once one enters a tracking system to track the defect… There is an initial setup cost that involves education, tool changes, and process changes to get ODC started” [Bhandari, 1992]. 

However, the benefits usually outweigh the costs, both for current and future projects.  Current projects benefit from a positive return on investment (ROI), as explained in Section 1.3.3.1, while future projects will benefit from defect prevention measures, as explained in Section 1.3.3.2.

     In his book, "The ROI from Software Quality" [El Emam, 2005], the author, Khaled El Emam, compares the profits and losses from a company's releasing software with numerous defects.  The profit can come from some contractors being paid the same regardless of the reliability of the software, and maybe even being paid to later correct the defects.  Also, customers are more likely to purchase support contracts if software tends to fail, while they may not buy them if the software works well.  (An analogy may be extended warranties on cars and appliances, which customers are more likely to purchase for less reliable products.)  However, the consequences from releasing defective software probably outweigh the benefits.  The threat of costly litigation due to failed software products is increasing, and there is increasing concern that software defects can lead to security vulnerabilities.  Even if litigation is not a concern, the cost of rework, or fixing defects, can be very high.  Also, if a company delivers low quality software, the company's reputation will be tarnished, and future sales will be adversely affected.  People will not continue to buy products with many defects.  Despite these negative consequences, some organizations continue to develop defective software, and must be shown that the costs outweigh the benefits.  This section shows that return on investment (ROI) from defect tracking against quality targets will almost always be positive.

     Figure 12, taken from an article by Webb and Patton [Webb, 2008], shows the costs associated with good quality and poor quality.  The costs associated with good quality, or costs to achieve quality, are prevention and appraisal costs.   The costs of poor quality, or costs due to a lack of quality, are internal failure and external failure costs.  Prevention costs cover efforts to ensure quality and defect prevention activities.  Appraisal costs address efforts to discover the condition of the software quality.  Internal failure costs cover efforts to detect and correct problems prior to the user detecting them, while external failure costs cover the effort to correct problems discovered by users. Table 22, from a paper by Demirors and Yildiz [Demirors, 2000], provides examples of costs in each of the categories.

Figure 12: Cost of Quality [Webb, 2008]

      External failure costs can be particularly insidious.  The obvious costs are those associated with correcting defects, which, as shown previously, can be relatively high after software is deployed.  The costs of correcting defects, however, often pale in comparison to other costs.  Figure 13 [Mcgibbon, 2009] illustrates the potential costs to a customer that can result from defective software.

Figure 13: Costs of external failures [Mcgibbon, 2009]

      Some writers, such as Rex Black, classify prevention and appraisal costs as "costs of conformance", and internal and external failure costs as "costs of nonconformance" [Black, 2000]. Black provides a return on investment (ROI) analysis for use of formal testing .  Table 23 shows that the cost of conformance, which includes appraisal costs associated with manual and automated testing, are much less than the costs of nonconformance.  This assumes that bugs found during testing cost $10 to fix while bugs found by the customer (external failure costs) are $1,000.  While this example is hypothetical, Black cites an actual case where software systems costs were reduced by 50% by using appraisal techniques like formal testing.

Table 23: ROI Analysis of Testing [Polydys, 2009]

     While formal testing provides a positive ROI, inspections have been shown to be even more effective.  Gack [Gack, 2009] showed an ROI of 4.6 from using inspections and testing compared to using testing alone as an appraisal method.  El Emam [El Emam, 2005] showed that inspections are effective in that the effort needed to correct defects using design or code inspections is only 25% of the effort needed to correct defects during testing. The ROI from using inspections instead of testing alone, in terms of percentage savings, is 11% for pre-release costs, 26% for post-release costs, and 38% for customers. The use of inspections results in a 3% schedule savings, which shows that quality improvements from inspections do not add time to a software project. Freimut [Freimut, 2005] described the results of using inspections at Siemens AG, Germany. The cost of rework was a 67% mean decrease measured after the improvements for inspections performed in the analysis phase; the cost of rework was a 47% mean decrease measured after the improvements for inspections performed in the design phase; and the cost of rework was a 12% mean decrease measured after the improvements for inspections performed in the development, or coding phase.

     Table 24, from the DACS ROI Dashboard and included in the 2007 DACS report, "A Business Case for Software Process Improvement" [Ferens, 2007] summarizes the results from multiple resources for using software inspections as a software process improvement activity.  The average ROI was 6.84, and the minimum was 2.0, which shows that ROI will almost always be positive from using inspections.

     El Eman [El Emam, 2005] shows how ROI is usually positive from using other defect prevention and appraisal methods.  He shows how to compute ROI for various methods, including inspections.

    Most defect tracking and removal methods discussed so far address detecting and removing defects as soon as possible after they are injected into the current project during requirements, design, etc.  Defect prevention, according to  Tian [Tian, 2005], is to reduce the chances of defects being injected in the first place.  Soni states that "prevention is better than cure" is a saying that applies to defects in the software development cycle as well as illness in medical science (see Defect Prevention: Reducing Costs and Enhancing Quality).  Version 1.2 of the CMMI for Development (CMMI-DEV)  Process Area of Causal Analysis and Resolution has the purpose of identifying the causes of defects and other problems and take action to prevent them from happening in the future [Konrad, 2003].  It would appear that defect prevention can have even more positive impact than defect appraisal activities like walkthroughs and inspections.  According to Schulmeyer and McManus,  the defect prevention process has a 15-to-1 payback and avoids 50% of all defects normally injected during the first year and first project in which it is used, and 70% avoidance within 2 or 3 years [Mcmanus, 1999].

     The defect tracking and removal activities discussed throughout this document are prerequisite for defect prevention.  For example, the Causal Analysis and Resolution Process Area for the CMMI-DEV has the specific goals (SGs) and specific practices (SPs) that are shown in Table 25.  The table shows the SGs and SPs associated with this CMMI Process Areaa, and the subpractices associated with each practice:

          In Table 25, the first subpractice listed, "Gather relevant defect or problem data", implies that an organization is already performing the defect tracking activities discussed previously.  Also, SP 2.2, "Evaluate the effects of changes", implies that an organization will track defects in future projects to see if the defect prevention measures proposed are indeed effective.

     In the CMMI-DEV, Causal Analysis and Resolution is a "Maturity Level 5" Process Area.  Level 5 is the highest maturity level; it implies an organization is not only actively collecting quantitative data (such as defect data), but is using this data to continuously improve their processes.  Defect prevention, then, requires not only a high level of capability in other CMMI Process Areas, but also requires a high degree of commitment from an organization to perform defect prevention activities.  Table 25 shows that defect prevention can be very involved; it requires a lot of time and effort.  However, the rewards can be worth it.

     Version 1.2 of the CMMI for Acquisition (CMMI-ACQ) also has Causal Analysis and Resolution as a Maturity Level 5 Process Area for acquisition agencies.  According to the CMMI-ACQ,  acquirers use Causal Analysis and Resolution practices to guide identification of root causes of selected defects and other problems and taking action to prevent their reoccurrence [CMMI PRODUCT TEAM, 2007].   The same SGs and SPs apply, although the acquirer will focus on defects reported by customers, end users, and suppliers.  In either case, causal analysis is best performed by people who have the best understanding of the selected defect or problem being studied [Hofmann, 2007]. 

     If an organization desires to perform defect prevention activities, they must be ready.  In his "Quality Questionnaire", El Eman asks the following questions for defect prevention [El Emam, 2005]:

       Question 3 shows that defect prevention goes far beyond defect removal.  Defect removal involves removing current defects, while defect prevention removes the very causes of these defects.   In Figure 14, Soni illustrates the activities associated with defect prevention and compares them with activities associated with defect tracking and removal.  The area in gray represents those areas associated with "the current philosophy" of handling defects: detection, tracking/documenting and analysis of defects to arrive with quick, short-term solutions" (from Defect Prevention: Reducing Costs and Enhancing Quality).

 Figure 14:  A Defect Prevention Cycle (from Defect Prevention: Reducing Costs and Enhancing Quality)

     When an organization determines it is ready for a defect prevention program, the organization needs to plan and document its program.  There are several resources that explain how to set up a program and that provide examples.  For instance, Figure 14 illustrates some steps in implementing a defect prevention program.  Kan also lists three steps for a defect prevention program: Analyzing existing defects or errors to trace root causes; suggesting preventive actions to eliminate the defect root causes; and implementing the preventive actions [Kan, 1995].  This process consists of four elements:

1. Conduct causal analysis meetings, which are 2-hour brainstorming sessions at the end of each development phase (in the current project).  Suggestions are recorded in an action database for subsequent reporting and tracking.
2. Form an action team to screen, prioritize, and implement suggested actions from causal analysis meetings.  The action team is the "engine" of the process.
3. Stage kickoff meetings at the beginning of each development phase to determine an effective process, and tools and methods, to prevent defects during that phase.  The meetings serve both as a primary feedback mechanism for the defect prevention process, and as a preventative measure.
4. Track actions and collect data to prevent suggestions from being lost over time, aid action implementation, and enhance communication among groups [Kan, 1995].

     The Software Productivity Consortium has issued a Defect Prevention Guidebook which addresses a recommended defect prevention process that can be implemented on a single project or across the organization, the organizational and project roles and responsibilities, and a set of defect reduction strategies.  The Guidebook also provides information for process improvement change agents and project managers for classifying defects from inspections to provide causal analysis, performing causal analysis, identifying defect prevention elements with the highest ROI for defect prevention investment, and planning and implementing preventive actions [Brown, 1998].

     The Defect Prevention Guidebook explains all process elements in detail, devoting an entire chapter to each key area.  An example is a chapter on performing causal analysis, where the steps are: forming a causal analysis team; holding causal analysis meetings (like Kan's first step); and implementing defect prevention strategies.  Holding meetings is further divided into six steps: selecting problems; classifying problems; identifying systematic errors; determining principal causes; developing corrective action proposals; and documenting meeting results. An example of a tool that can be used in identifying systematic errors (the same defect being repeated on different occasions) is a Pareto Chart for coding defects, which is shown in Figure 15.  The Pareto chart shows the number of defects by cause in descending order, with the leftmost bars showing the most common causes, which subsequently should probably receive the most attention.  The Guide contains an appendix that describes several useful tools, including more information about Pareto charts. 

Figure 15:  Pareto Chart Used In Causal Analysis [Brown, 1998]

     While the Guidebook and other resources give useful information about setting up a defect prevention program, the project or organizational manager or team needs to determine the best processes to use in setting up a program for a specific project or organization.  Some resources do provide guidance in this area; for example, Capers Jones provides some useful guidance on which software development methods are most useful for which categories of defects, or errors.  One or more of these development methods can then be used in a defect prevention program.   According to Capers Jones, "defect prevention" means "the overall set of technologies that simplify complexity and minimize a natural human tendency to make errors while performing complex tasks" [Jones, 2008].  Table 26 shows some development tools and methods, such as Joint Application Design (JAD) and Quality Function Deployment (QFD), with their effectiveness for different categories of defects.

Table 26: Defect Prevention Methods [Jones, 2008]

This section describes the Gold Practices that are inputs to the Gold Practice, "Defect Tracking Against Quality Targets".  The two primary inputs are the identification of quality gates and methods for detect detection.  While information in both areas is included in this Gold Practice, the other Gold Practices shown as "inputs" in Figure 16 may be consulted for further insight into these input areas.  Table 27 describes the inputs from other Gold Practices, and how those Gold Practices contribute to these inputs.

This section describes the Gold Practices that use outputs from the Gold Practice, "Defect Tracking Against Quality Targets".  The three primary outputs are analyzing data and identifying defect causes and corrective action, implementing and verifying corrective action, and assessing and reporting quality improvements.  While information in these three areas is included in this Gold Practice, the other Gold Practices shown as "outputs" in Figure 16 may be consulted for further insight into these output areas.  Table 28 describes the outputs to other Gold Practices, and how those Gold Practices use these outputs.