|
Defect Tracking Against Quality Targets
Focus Area: Measurement
Definition and Summary: Defects should be tracked according to a planned, documented process, measured against established targets, and systematically tracked through to removal or resolution. Airlie Software Council Nine Best Practices DACS Focus Area: Quality
1.0 Description of the Practice
The following are summary and detailed descriptions of the Gold Practice, "Defect Tracking Against Quality Targets".
1.1 Summary Description
Defect tracking against quality targets entails recording defects in a database; following a documented process to analyze, resolve, and remove them; tracking the process; measuring defects against quality targets; and reporting metrics on the process to program management.
1.2 Detailed Description
Norm Brown, a member of the Software Program Managers Network (SPMN), explains this practice:
"Essential Elements. Implement practices to find defects when they occur. Establish a goal for delivered defects, including requirements problems, per unit of size. Use configuration management to report and track all defects found through all techniques. Also track the average and maximum time to close a defect after it's reported. Monitor the organization's defect removal efficiency by tracking the number of defects found and fixed during development and during the system's first year in the field." [Brown, 1996]
From a 1990s SPMN publication, Figure 1 shows a process description of the practice, while Table 1 provides a questionnaire further defining this practice.
![]() Figure 1: Process For Defect Tracking Against Quality Targets Table 1: Questionnaire To Determine If You Are Practicing Defect Tracking Against Quality Targets
Reproduced from Software Program Manager Network’s web site. Now available at SPMN Airlie Council Checklists
Defect tracking against quality targets requires that quality targets be set and that tracked defects be analyzed against those targets. The detailed description of this Gold Practice is now explained in depth in two areas: setting quality targets (or gates), and tracking defects. Suggestions for implementing tasks in these areas are included.
1.2.1 Setting quality targets (or gates)
This section describes some methods for setting quality targets, especially reliability, and processes for tracking defects.
Methods for Setting Quality Targets Quality targets, or gates, are quantitative values that are measures of product or process quality. According to the Practical Software and Systems Measurement Guide [Mcgarry, 2002], there are several quality measures that can be useful. Some of these measures, with examples of target values, are as follows:
The methods, or apprroaches discussed here use reliability as an example; however, some methods can also be used to set quality targets for other quality indicators. Two approaches to setting quality targets are user-oriented methods and parametric estimating methods. Each category of methods will now be discussed.
1. User-Oriented Measures: One set of approaches to setting quality targets is in terms of user-oriented measures such as reliability, availability, and maintainability. Three common methods for allocating reliability are by operational profile, architectural structure, and black box methods.
Allocating Reliability by Operational Profile
The operational profile of a system is the set of operations that the software can execute along with the probability with which they occur. It shows the user demands on a system.
Table 2: Procedure To Characterize Operational Profile of a System
System reliability, R, is the probability that the system will execute an operation correctly. System reliability is calculated as:
where pi is the probability of the ith operation being selected by the user, and Ri is the probability that the ith operation executes correctly. Each operation should be assigned a reliability such that the above sum equals the system reliability. Typically, operations that are executed more frequently are required to be more reliable.
Allocating Reliability by Architectural Structure
An architecture-based approach to reliability includes the construction of a model of a system. Typical models include:
In a sense, Markov chains subsume the other models mentioned above. A Markov chain is a random process where all information about the future is contained in the present state (i.e. one does not need to examine the past to determine the future). To be more exact, the process has the "Markov property", meaning that future states depend only on the present state and are independent of past states. In other words, the description of the present state fully captures all the information that could influence the future evolution of the process. Being a stochastic process means that all state transitions are probabilistic (determined by random chance and thus unpredictable in detail), though likely predictable in its statistical properties. Markov chains are often used in sciences like physics and chemistry to represent unknown or unmodelled details of the system where no history is available or is not relevant. They are also used in testing to generate efficient test strategies based on expected system usage patterns.
Markov chains can also be used in reliability allocation for a system or software program. Table 3 outlines the use of a Markov chain model to predict system reliability. Sensitivity analysis shows impact of model inputs on outputs. Sensitivity analysis can be used in allocating a system reliability requirement to target reliabilities for components.
Table 3: Markov Chain Model
Based on [Gokhale, 2002]
Black Box Methods
This practice calls for relating data collected during software development to quality targets, such as reliability. Table 4 summarizes some approaches that do not use information about the internal architectural structure of a subsystem.
Table 4: Black Box Approaches Relating Data To Reliability
Inspired by [Gokhale, 1998]
2. Estimating Methods: Another set of approaches is to use estimating models and methodologies. Two of these are analogy estimating, and parametric modeling.
Analogy Estimating This approach uses information or data from similar past programs. An organization can investigate what has occurred in the past, and set quality targets or goals based on what can be expected. Capers Jones, in his book "Applied Software Management: Third Edition" [Jones, 2008], presents several tables that show averages for defects. Table 5 here, an abbreviated version of one of his tables, shows U.S. averages for number of defects per function point (FP), which is a software size measure, for programs of various sizes and forms, or applications.
Table 5: U.S. Averages for Number of Defects Per FP [Jones, 2008]
While analogy methods can be used to determine reasonability of quality goals, they should not be the sole method used. First of all, values like those in Table 5 are strictly averages, and individual projects will vary, sometimes considerably, from these averages. Good quality targets for individual programs must consider the unique aspects of the project being developed.
Parametric Models Parametric models, which use design parameters of a project for estimating, have long been successfully used in estimating software cost and schedule, and sometimes software size. Some of the models also estimate defects based on the design parameters of the software. Some basic parametric models base defects on only one parameter or a few parameters. For example, Capers Jones proposes the following rule of thumb: [Jones, 2007]
Defects = FP 1.25
Of course, such models will only provide a ballpark estimate at best, and would not be used in accurate target setting. Some more complex software cost estimating tools, such as the Galorath SEER-SEM model and QSM Software Life Cycle Model (SLIM) use more than 20 design parameters to estimate defects. These models also estimate defects in categories, as will be explained in Section 1.2.2.2. A non-proprietary parametric estimating model is the Constructive Quality Model (COQUALMO), an extension of the popular Constructive Cost Model II (COCOMO II) software cost estimating model from the University of Southern California Center for Software Engineering (USC-CSE). Details of COQUALMO are published in the COCOMO II book [Abts, 2000] and on the COQUALMO page of the USC-CSE web site, and some of the features are summarized here.
COQUALMO is an estimation model that can be used for predicting number of residual defects/KSLOC (Thousands of Source Lines of Code) or defects/FP (Function Point) in a software product. It can be applied in the early activities such as analysis and design, as well as in the later stages for refining the estimate when more information is available. It enables "'what if" analyses that demonstrate the impact of various defect removal techniques and the effects of personnel, project, product and platform characteristics on software quality. It also provides insights into determining ship time, assessment of payoffs for quality investments and understanding of interactions amongst quality strategies. Table 6 shows the factors that COQUALMO uses to estimate defects; these parameters are also used in the COCOMO II to estimate software cost and schedule.
Table 6: COQUALMO Inputs [Abts, 2000]
Suggestions for Choosing an Approach Any of these approaches, or a combination of approaches, can be used depending on the situation. If relevant historical data is available, analogy or parametric models or methods are especially useful when they can be calibrated to the user's environment.
1.2.2 Processes For Defect Tracking
After quality targets are set and defect estimates are computed, it is essential to track defects to see if the quality goals, or targets are being met. Figure 2 provides an example of a defect tracking chart that may be used in tracking defects [Evans, 2006].
Figure 2: Defect tracking Chart Example [Evans, 2006] Defect tracking involves at least six processes. They are detecting defects, analyzing and classifying defects, resolving and removing defects, tracking defect removal, measuring defects, and reporting progress to program management. Each of the defect tracking processes will now be discussed in detail.
1.2.2.1 Detecting Defects
According to Capers Jones, there are at least five types of defects that can occur. These are requirements defects, design defects, code defects, documentation defects, and bad fixes, which are defects that are induced accidentally when correcting another defect [Jones, 2008]. The challenge to software engineers and managers is to detect defects in these areas as early as possible. According to El Emam, it is much less expensive to find and correct defects as early as possible when their costs are relatively low [El Emam, 2005].
Figure 3 is an example of a defect containment matrix [Campo, 2007]. Ideally, defects are detected and corrected in the phase in which they occur. If they are detected in a later phase, rework costs are incurred, and these costs will increase as the detected stage is later. For example, detecting and correcting a requirements defect will probably be a matter of changing a paragraph in a document, while detecting a requirements defect during testing will not only result in having to rewrite the requirement, but also having to redo portions of the design, rewrite code, and retest the software affected. Therefore, it is important to find (and correct) defects as early as possible.
Figure 3: Defect Detection Matrix [Campo, 2007] So, how do we detect defects as early as possible? Figure 4 from El Emam [El Emam, 2005] illustrates defect detection and correction, including bad fix removal, during the various phases of the software life cycle. Probably the most common, and proven technique, shown in Figure 4, is software inspections. Inspections can be performed during any phase of the software life cycle from requirements through maintenance.
Figure 4: [El Emam, 2005] There are several proven methods for detecting defects during all phases of the software life cycle. Some of these will now be discussed:
Inspections
One of the most popular and most effective methods is use of software inspections. The ideas behind design and code inspections were introduced by Fagan in 1976 [Fagan, 1976]. Since that time, many organizations have used inspections to detect defects early in a program. According to El Emam [El Emam, 2005] and others, inspections are much cheaper than some other commonly used methods of finding defects, such as testing and customer findings.
According to Fagan [Fagan, 1986], there are six steps in the inspection process:
Jones [Jones, 2008] describes three protocols necessary for the inspection process to succeed:
Reviews and Walkthroughs
As shown in Figure 5, reviews and walkthroughs are other methods for detecting defects. Although these methods are usually not as effective as inspections for detecting defects, they are useful and can be used in addition to formal inspections. As discussed by Jolene Hart [Hart, 1982] a walkthrough is a peer group review of any technical product. This means that the review involves other people working with you, as well as users, programmers, systems designers, operations people, and others who may be involved in various aspects of the project you are working on. But a walkthrough, under normal conditions, does not include your boss, or the head of the department, or the vice-president from the user organization. The primary reason for conducting a walkthrough is to spot errors as quickly and economically as possible. A walkthrough can take place at virtually any point in the development of a technical product. Design reviews are more formal reviews that are usually conducted at program milestones such as requirements reviews and critical design reviews. Reviews tend to be general in nature, so they will not catch as many problems as inspections. At the opposite end of the spectrum, peer reviews tend to be very informal, so they also will not usually be as effective as inspections at finding defects. A peer review technique that may be effective at discovering defects is pair programming, an agile software development practice [Lindvall, 2003] that can result in better software code than for programmers working alone.
Figure 5: Defect Detection Methods [Mcgibbon, 2009]
Testing
According to Tian, one of the purposes of testing is to detect and fix problems [Tian, 2005]. According to Kan, there are several levels of software testing, including unit testing, component testing, and system-level testing [Kan, 1995]. Testing is part of the software development process, so this method of defect detection should not require additional resources beyond what is expected during development. However, testing should be thorough and done correctly. There are plenty of resources available, such as the books by Tian [Tian, 2005] and well-known software engineering writer Roger Pressman [Pressman, 2005] that explain how to set up a viable testing strategy. Tian discusses different types of testing used at different levels, and Pressman describes a strategic approach which includes verification and validation.
A limitation of testing is that it does not discover defects, or errors, until later in the development process. Defects found in testing are generally require more effort to fix than those discovered during requirements and design. However, finding defects during testing is preferable to finding them after a product is deployed in the field!
Suggestions for choosing defect detection methods
The first consideration should be using inspections for the project being managed. Inspections are probably the most effective method for defect detection, but other methods can supplement inspections to detect defects. Testing, of course, should be a part of any software development endeavor. Whatever method or methods are used, the defects must be documented so further action can be taken.
1.2.2.2 Defect Analysis and Classification
A process for analyzing and classifying defects and problem reports is an important part of defect tracking against quality targets. Some classification schemes are described here: an IEEE standard classification categorization, an Orthogonal Defect Classification (ODC), and other classification schemes.
IEEE Standard Anomaly Classification
The Software Engineering Standards Committee of the IEEE Computer Society has created a standard classification for software anomalies [Konrad, 1997]. In the IEEE framework, an anomaly can be a failure occurring during run-time or a static fault located in the code or documentation. The committee chose the term “anomaly” to convey a more neutral connotation than some synonyms. Figure 6 shows the process for classifying anomalies. The steps for identifying the impact of each classification step results in the specification of the value of various categories. The standard defines two types of categories, mandatory and optional. The mandatory categories are shown in Table 7.
Figure 6: IEEE Standard Anomaly Classification Process
Table 7: IEEE Standard Mandatory Classifications
Orthogonal Defect Classification
Orthogonal Defect Classification (ODC) is a multidimensional in-process measurement system supporting sophisticated analyses. For example, the variation in the distribution of defect types by lifecycle phase can provide signatures of a process. ODC provides a means to compare observed distributions with expected distributions and thereby determine if a project’s logical location among phases by signature matches the scheduled location. Under ODC, defects are categorized:
"into classes that collectively point to the part of the process that needs attention, much like characterizing a point in a Cartesian system of orthogonal axes by its (x, y, z) coordinates... For classifications to be widely applicable, the classification must have consistency between the [lifecycle phase]. Without consistency, it is almost impossible to look at trends across stages. Ideally, the classification should be quite independent of the specifics of a product or organization" [Bhandari, 1992].
Early presentations of ODC [Bhandari, 1992] discussed two attributes of a defect: Type and Trigger. "A defect trigger is a condition that allows a defect to surface" [Bhandari, 1992]. For example, a defect of a certain type might not be visible until the software is run on a certain hardware platform. The hardware platform, then, is the trigger. A later IBM implementation had two additional attributes: Source and Impact [Lyu, 1996]. Tables 8 and 9 present how defects are classified in one implementation of ODC which has been recently described in the literature.
Table 8: Attributes Identified When Defect Uncovered
From [Bassin, 2001]
Table 9: Attributes Identified When Design Or Code Defect Fixed
From [Bassin, 2001] Other Classification Schemes There are some defect classification schemes that are less elaborate than the two methods explained above, but may be adequate in some situations. Putnam and Myers, who include defect rates as one of the five core metrics for successful software management [Myers, 2003], explain a defect severity ranking that is based on the time limit in which the defect must be fixed. The classification is as follows:
Fenton and Pfleeger [Fenton, 1997] also include a single classification scheme where defects are classified in one of five categories based on relative severity. The five categories are "major", "minor", "negligible", "documentation", or "unknown". Fenton and Pfleeger point out, however, that such a classification scheme is not truly orthogonal in that, for instance, a documentation error could be a major problem. They propose eight mutually-independent (or orthogonal) attributes of defects that, ideally, should be described:
El Emam [El Emam, 2005] proposes three schemes for classifying defects: priority, severity, and problem type. Each of the three schemes will now be explained:
Table 10: Defect Priority Classifications [El Emam, 2005]
Table 11: Defect Severity [El Emam, 2005]
Table 12: Problem Type [El Emam, 2005]
Suggestions for Defect Classification There are many different methods for classifying, or categorizing defects, a few of which are described here. The method or combination of methods chosen should be dependent on the particular needs of a project or program; however, the information presented here should be helpful.
1.2.2.3 Defect Resolution and Removal
After defects are detected , analyzed, and classified, it is important to remove them as soon as possible. Of course, they should be removed in the priority order set up in classification, but it is highly desirable to remove defects during the phase they occur to the maximum extent possible. Figure 7 shows a phase containment matrix [Ferens, 2008] which compares the phase in which defects are detected with the phase in which they are removed. (This is similar to Figure 3, which compares phases where defects are inserted and detected.) If defects are removed in the phase they are detected, this can be considered part of the normal development process; however, if they are removed in later phases, some degree of rework will be needed, and this can be costly. According to Kan [Kan, 1995] and others, removing defects during testing is more expensive than removing them in earlier phases. The red (or dark) areas on this chart illustrate out-of-phase removal. Generally, the later the defect is removed after it is detected, the greater degree of rework that will be required. Figure 8 is another illustration of the benefits of removing (or repairing) a defect during the phase it occurs. Here, the relative cost to remove a requirements defect is shown; it is much more expensive to correct during later phases of a program or project. During requirements analysis, a requirements defect or error correction will only involve something like a change to a paragraph in a requirements document. If this defect is repaired during testing, it will involve not only changing a paragraph in a document, but can involve redesigning certain components, and re-coding and re-testing them. Some techniques such as reviews, walkthroughs, inspections, and even testing are sometimes described as defect removal techniques. In reality, the purpose of these techniques is to discover or detect defects; defect removal itself may require additional techniques or methods. Table 13 illustrates the process for Fagan inspections, Note that it is the author's responsibility to fix (correct or remove) defects, and not the responsibility of the inspection team. Table 13: Fagan Inspection Activities [Gack, 2009]
Before defects can be removed, they must first be resolved; the cause of the defect must be determined in order to determine how to remove the defect. One Process Area of the Software Engineering Institute (SEI) Capability Maturity Model Integration (CMMI) is "Causal Analysis and Resolution" [Konrad, 2003]. Some of the "specific practices" for this Process Area are to "determine the cause of defects" and "analyze causes". If the cause of a defect can be removed, the defect itself may be removed. This Process Area is more associated with preventing future defects than correcting current ones; however, performing these two specific practices can aid in defect resolution and subsequent removal. According to Tian [Tian, 2005], inspections can be expanded to include causal analysis, which may later be used in defect prevention, but can also be used in defect resolution. Defect removal is sometimes included in the resolution process [Tian, 2005]; however, this Gold Practice will treat removal as a separate set of activities after the cause of a defect is resolved. As described in the CMMI, removal is often performed by a separate group from those discovering the defect, as is often the case in formal testing [Konrad, 2003]. In some cases, such as unit testing, the person discovering the defect also corrects it. According to Kan [Kan, 1995], defect removal is associated with activities at the end of development phases. Table 14 shows the activities associated with defect injection and removal. Table 14: Defect Injection and Removal Activities [Kan, 1995]
The specific processes used for removing defects will vary by phase and by project. In the case of requirements defects, the defect may be removed (or corrected) by the author merely rewriting a sentence or paragraph in a requirements specification or similar document. During design, removal may involve redoing a sequence diagram or a hierarchical chart. Coding errors, usually discovered during unit testing, are relatively easy to correct if they are syntax errors (such as spelling "class" as "glass") since compilers and interpreters usually identify these; however, logic errors (such as subtracting two numbers instead of adding them) can be quite arduous. Component and higher-level testing defects or errors are probably the most difficult to remove since they involve effort to isolate the defect, then retest the module where it occurred and possibly other modules to be sure they are unaffected by the defect removal. For detecting and removing testing defects, and some coding and design defects, automated tools may be useful or even necessary. (A listing of tools is included in the Appendix.)
Sometimes the removal process does not completely remove the defect or unintentionally introduces another defect. According to Capers Jones [Jones, 2008], these are sometimes called "bad fixes", new defects that are accidentally induced in an attempt to repair (or remove) previous defects. Capers Jones also uses the term "bad fix defect injection" to describe this, and states that about 7% of all defect repairs will exhibit this phenomenon. Jones says that this can rise to 20% for very complex and poorly structured applications [Jones, 2007].
Suggestions for Defect Analysis and Removal
Some methods for defect detection do not inherently result in defects being resolved and removed. For example, the purpose of some walkthroughs and inspections is only to identify defects; it is up to the software developers to correct them. For defects detected, the cause should be assessed, and the defect removed (or corrected) as soon as possible. Every effort should be made to remove defects in the same phase as they are detected; otherwise, rework will be needed, and the removal will probably cost much more. Also, realize that defects discovered later in the development process, such as testing, will be more expensive to remove than those discovered earlier, even if they are removed during the same phase.
1.2.2.4 Defect Removal Tracking
When defects are removed, or repaired, it is important to keep track of the removals as well as the occurrence of defects. Figure 9 is an example of a defect tracking chart where both incoming and corrected defects are plotted.
![]() Figure 9: Incoming and Corrected (Removed) Defects [Ferens, 2010]
One key measure of quality that can be obtained from keeping track of defect removals is "defect removal efficiency". According to Capers Jones, defect removal efficiency is "the percentage of defect potentials that will be removed before the software is delivered to its users or customers" [Jones, 2008]. Defect potentials, in turn, are the probable numbers of defects that will be found during the development of a software application. Table 15 is an illustration of defect removal efficiency, with the potentials measured in "defects per function point", a measure of software size.
Table 15: Defect Removal Efficiency [Jones, 2008]
The defect removal efficiency of 85% indicates that the end product delivered to the user will have 15% of the defects it would if removal did not take place. It should be noted that the 85% is an average for the development process. Some defects, like bad fixes, are harder to remove than others, like coding errors.
Kan discusses a similar term, defect removal effectiveness [Kan, 1995], which investigates the defect removal for each step, or phase, of the software development process. The defect removal effectiveness for each step is as follows:
Defect Removal Effectiveness = _________Defects Removed (at the step)______________ x 100% Defects existing on step entry + defects injected during the step
An example is, during high level design, Defects removed = 730, defects from previous phases = 122, and defects injected during this step = 859, then Defect Effectiveness = 730 / (122 + 859) x 100% = 74%. According to Kan, these measures are useful in quality planning and quality management. In the narrative for the CMMI Process Area "Organizational Process Performance", it is suggested that Defect Removal Effectiveness can be quite useful as a process performance measure [Konrad, 2003].
Suggestions for Defect removal tracking In addition to other quality measures (discussed in detail in Section 1.2.2.5), defect removal efficiency and effectiveness can be very useful in assessing quality targets and how well they are met. Collecting defect removal information during all development phases must be done if these measures are to be used.
1.2.2.5 Measuring Defects (Against Quality Targets)
In addition to defect removal tracking discussed in the previous section (1.2.2.4), other defect measures can be useful in assessing whether targets are met as well as facilitating software project and process management in general. This section describes some measures and methods for selecting measures that can be useful in defect tracking against quality targets. The Practical Software and Systems Measurement (PSM) Guide An excellent source of measures for software projects in general, including quality measures, is the Practical Software and Systems Measurement (PSM) Guide [Mcgarry, 2002]. Section 3 of this Guide explains the Issue-Category-Measure (ICM) approach to selecting measures for software projects. The Guide lists more than 50 measures and provides a detailed description of each measure. Table 16, from the PSM Guide, shows measures that can be used for selected issues and categories within issues. Some of the measures that can be used in defect tracking against quality targets are now summarized. Table 16: Issue-Category-Measure table [Mcgarry, 2002]
Of course, some of the other measures in Table 16 can be used in quality assessment. The six above are just provided as examples. As outlined in the PSM Guide, organizations can use Table 16 and the information in Section 3 to determine the specific issues and categories for their programs and choose the appropriate measures.
Goal-Question-Metric (GQM) Approach
As described by Ellen Walker as a DACS Gold Practice [Walker, 2005], and from the writings of Dr. V.R. Basili [Abran, 2004] and Rini van Solingen [Van Solingen, 1999], the GQM approach defines certain goals for a software project, refines these goals into one or more questions, then defines metrics (or measures) that provide information to answer these questions. Figure 10 shows the hierarchy of the GQM approach, as described by Dr. Basili [Abran, 2004]. It is possible, as illustrated in Figure 10, for a metric (or measure) to answer more than one question, and for a question to address one or more goals. Basili explains that the metrics represent a quantitative level; they answer questions in a quantitative way; however, they may be either objective or subjective. For example, number of lines of code in a program is an objective metric since it depends only on the object being measured (the code, in this case). Meanwhile, a user satisfaction rating from "1" to "5" is subjective since the user's viewpoint will affect the rating, even though the rating itself is quantitative, or numerical.
Figure 10: The GQM Hierarchy [Abran, 2004]
Figure 11 is an example of use of the GQM approach for quality measures. The goal is to minimize defects in a software product, and the relevant questions are "How likely are defects after release?", and "When are we creating the problems (like, during which stage of the development process)?". For each question, one metric, or measure, has been selected. (These metrics can be considered to be objective.) It is usually helpful to graph the measures as shown in Figure 10 for one of the measures.
Figure 11: Illustration of the GQM Approach [Ferens, 2010]
Van Solingen [Van Solingen, 1999] provides an example of a quality goal that addresses software reliability. His statement of the reliability goal is "Analyze the delivered product and development process for the purpose of characterizing with respect to reliability and its causes from the viewpoint of the software development team in the context of Project "A". Van Solingen then proposes 18 questions and one or more metrics, or measures, for each question. For example, Question 1 is "What is the composition of the product?", and three metrics are a list of life cycle documents, a list of subsystems, and a list of modules.
Capers Jones' Quality Measures In his book "Applied Software Measurement" [Jones, 2008], Capers Jones recommends quality
measures in two categories: user satisfaction measures and defect measures. A desirable goal for companies is, of course, to have
high levels of user satisfaction and low levels of defects. Specific values or ratings for these goals can be established using some
methods and ideas discussed in Section 1.2.1.
User satisfaction measures include ease of use, functionality, and support. These are "soft" measures since they are based on user perspectives and obtained by surveys, usually taken once a year. If user satisfaction measures are included in quality targets, these measures can be useful, since some definitions of quality include "high degrees of user satisfaction" and "ease of use". One limitation of user satisfaction measures are they are not available until after a program is completed. Defect measures include defect volumes (e.g., number of defects per function point), defect severities, defect origins, defect causes, and (as discussed in Section 1.2.2.4) defect removal efficiency. According to Capers Jones, defect removal efficiency is a measure used by leading edge companies. The defect measures are "hard" measures that can be assessed objectively and more often (once a month) during a program. In addition to recommending quality measures to use, Capers Jones explains there are certain quality measures not to use! Cost per defect is one example; it can actually penalize high quality products that have relatively fewer defects! Table 17 shows a comparison of costs per defect for a low-quality "buggy" application and a high quality application of the same size. The cost per defect is actually a combination of three costs: preparation costs, execution costs, and repair costs. Preparation costs tend to be fixed; for example, the cost to create test cases for testing or reading specifications for a design review is the same no matter how many defects the product contains. Execution costs, the costs of actually conducting a review, audit, or test, are "semi-variable"; there are certain costs associated with going through an exercise no matter how many defects are present, while there are costs such as recording defects found that increase with number of defects. Lastly, repair costs are the costs of fixing or removing defects; these costs vary directly with the number of defects that must be corrected. As shown in Table 17, for two similar sized programs, cost per defect looks better for the low quality program; however, the total costs are much lower for the higher quality program. Table 17: Cost-per-Defect Paradox [Jones, 2008]
Although Capers Jones does describe some measures that can and should be used in a program (and should not be used), determining which of these metrics to use for a specific program will depend on such factors as program characteristics, the user's environment, and the organization developing the software. For example, an inexperienced organization may not be able to report defect removal efficiency. Using approaches like PSM or GQM can help in this selection. It is also noted there are other references such as the CMMI [Konrad, 2003] that provide descriptions of quality measures that can be useful. Again, which ones to use depends on the nature of the program and, according to the CMMI, the maturity level of the organization developing the software.
Quality Measures Used by Corporations Sometimes companies develop "standard" quality measures that they use for their software development efforts. For example, as reported by Fenton and Pfleeger [Fenton, 1997], AT&T developed the following set of quality measures:
Kan [Kan, 1995] provides a list of measures that resulted from studying quality efforts for several companies. Kan's list contains 13 measures in three categories, as shown in Table 18. Table 18: Thirteen Quality Measures [Kan, 1995]
Lists of measures used by specific companies (or groups of companies) can be useful resources in finding suitable quality measures for a program. However, an approach such as PSM or GQM should be used to determine which measures are suitable for a specific project or program.
Suggestions for measuring defects This section describes many measures that can be used in assessing whether quality targets, or goals, are being met. The specific measures to be used, however, depend on the quality targets actually set (as explained in Section 1.2.1) and the characteristics of the project or program and the organization developing the software. Two methods that can help in selecting suitable measures are PSM and GQM; however, there are other methods that may be used to supplement these methods, such as the "Factor-Criteria-Metric" method described by Fenton and Pfleeger [Fenton, 1997]. Another useful source is information on past programs from different corporations; however, the measures and methods that should be used will vary by program or project.
1.2.2.6 Reporting Progress to Management
As explained by Fenton and Pfleeger [Fenton, 1997], most customers know about the area for which a software product is being developed, but little about compilers, programming languages, and other aspects of software development. Therefore, measurements must be presented in a way that tells both customers and developers how a project is doing. For any program or project, it is important to report progress of the effort, including quality concerns, to project or program managers, other senior-level managers, and other stakeholders in a program or project. As explained by Schwalbe [Schwalbe, 2006], a stakeholder can be anyone who has an interest in a project, including a competitor. (Of course, the information a company reports to competitors should be minimal!) Schwalbe goes on to explain that different types and levels of reporting will be made to different stakeholders. In fact, Schwalbe recommends developing a stakeholder matrix for a project, which lists key stakeholders and their personalities so reports can be tailored to individuals. Although all stakeholders are important, this section concentrates on reports to management. In a section of their book "Software Metrics" [Fenton, 1997], Fenton and Pfleeger compare the concerns of managers with those of developers (like analysts and engineers), and the information needed by both. The five main concerns of management, according to Fenton and Pfleeger, are:
We have already discussed some measures that are useful for Questions 3 and 4 above; many defect-related measures are useful in assessing code quality, and user satisfaction measures can be used to assess how content users are with products. Some of the measures can also be used to answer the other questions. (More information about process improvement is included in Section 1.3.3.) This section addresses not only what measures should be reported to management, but how they should be reported. For the CMMI, one of the Process Areas, "Process and Product Quality Assurance", involves providing feedback to managers on the results of quality assurance activities [Konrad, 2003]. Table 19 lists four specific practices within this Process Area, and some results of these practices that should be reported to management. (More about the CMMI is in Section 4.0). Table 19: Specific Practices for CMMI Process Area "Process and Product Quality Assurance" [Konrad, 2003]
Several other CMMI Process Areas include reporting quality information to managment. The Integrated Project Management Porcess Area includes a Specific Practice of reporting defects to relevant stakeholders. The Organizational Process definition Process Area includes a Specific Practice of establishing a measurement repository to include quality measures such as defect density and number of defects found. The Requirements Development Process Area includes requirements defects as an item to be reported for the Specific Practice of Analyzing Requirements. The Quantitative Process Management Process Area includes assessing process performance, which is characterized by both process measures (e.g., effort, cycle time, and defect removal efficiency) and product measures (e.g., reliability, defect density, and response time).
The CMMI also has a Generic Practice, "Review Status With Higher Management" (GP 2.10) that applies to all Process Areas. This is a "Level 2" Process Area, which means that organizations must be proficient at this area to be recognized at Level 2, which is, in effect, "basic project management". This Generic Practice involves periodically reviewing activities, status, and results of the software development process with higher level management, and resolving issues. This will include quality considerations.
For communications issues in general, including reporting progress to management, The Project Management Body of Knowledge (PMBOK) treats Project Communications Management as one of nine "knowledge areas" for project management [Project Management Institute, 2008]. The PMBOK lists five processes to insure timely and appropriate generation, collection, distribution, storage, retrieval, and ultimate disposition of project information. The five steps are:
In her book, "Information Technology Project management" [Schwalbe, 2006], Kathy Schwalbe provides an example of a stakeholder analysis that can be used to determine the level of reporting and best method for distributing reports, and an example of a stakeholder communications analysis to show the preferred method and frequency of communication. Table 20 is an example of a combination of these analyses that can be used in defect reporting.
Table 20: Communication Analysis
Suggestions for Reporting Progress to Management There are many defect measures that can be reported to management, and different ways of reporting this information. What information should be reported, and to what level of detail, will depend on who is receiving the data. Although Fenton and Pfleeger, and the CMMI provide some general guidelines, reporting to management and other stakeholders is highly dependent on the project and the nature of stakeholders. Having a communications management plan early in a project and a stakeholder analysis can help determine these reporting requirements.
1.3 Characteristics of Implementation
This section contains summary and detailed characteristics, and benefits that result from using this Gold Practice.
1.3.1 Summary Characteristics
To effectively implement this Gold Practice, organizations must be capable of both setting quality targets (or identifying quality gates), and handling defect tracking. This includes detecting, analyzing, correcting, and reporting defects. The Software Engineering Institute (SEI) Capability Maturity Model Integration for Development (CMMI-DEV), discussed further in Section 4, includes some specific requirements in three Process Areas: Product and Process Quality Assurance, Quantitative Project Management, and Causal Analysis and Resolution [Konrad, 2003]. Chapter 11 of the Software Engineering Body of Knowledge (SWEBOK) explains prerequisites for an organization to properly address software quality [Abran, 2004].
1.3.2 Detailed Characteristics
Table 21 lists the characteristics, or practice essentials, for an organization to implement this Gold Practice. These are from Version 5.2 of the SPMN Guide 16 Critical Software Practices (SPMN).
Table 21: Defect Tracking Practice Essentials
Reproduced from SPMN 16 Critical Software Practices Version 5-2
1.3.3 Anticipated Benefits from Implementation
Defect tracking against quality targets provides benefits to both the project(s) currently being managed and future projects. These benefits are gained at the expense of training costs, installation of appropriate tools, and introduction of processes. These costs can be seen in reports on experiences with Orthogonal Defect Collection (ODC), one method of defect tracking:
“The cost impact on an individual software engineer during the development process is minimal. Typically, measured in the order of a few dozen keystrokes per defect to fill out one or two panels. The incremental time is probably negligible once one enters a tracking system to track the defect… There is an initial setup cost that involves education, tool changes, and process changes to get ODC started” [Bhandari, 1992]. However, the benefits usually outweigh the costs, both for current and future projects. Current projects benefit from a positive return on investment (ROI), as explained in Section 1.3.3.1, while future projects will benefit from defect prevention measures, as explained in Section 1.3.3.2.
1.3.3.1 Return on Investment for Current Projects
In his book, "The ROI from Software Quality" [El Emam, 2005], the author, Khaled El Emam, compares the profits and losses from a company's releasing software with numerous defects. The profit can come from some contractors being paid the same regardless of the reliability of the software, and maybe even being paid to later correct the defects. Also, customers are more likely to purchase support contracts if software tends to fail, while they may not buy them if the software works well. (An analogy may be extended warranties on cars and appliances, which customers are more likely to purchase for less reliable products.) However, the consequences from releasing defective software probably outweigh the benefits. The threat of costly litigation due to failed software products is increasing, and there is increasing concern that software defects can lead to security vulnerabilities. Even if litigation is not a concern, the cost of rework, or fixing defects, can be very high. Also, if a company delivers low quality software, the company's reputation will be tarnished, and future sales will be adversely affected. People will not continue to buy products with many defects. Despite these negative consequences, some organizations continue to develop defective software, and must be shown that the costs outweigh the benefits. This section shows that return on investment (ROI) from defect tracking against quality targets will almost always be positive. Figure 12, taken from an article by Webb and Patton [Webb, 2008], shows the costs associated with good quality and poor quality. The costs associated with good quality, or costs to achieve quality, are prevention and appraisal costs. The costs of poor quality, or costs due to a lack of quality, are internal failure and external failure costs. Prevention costs cover efforts to ensure quality and defect prevention activities. Appraisal costs address efforts to discover the condition of the software quality. Internal failure costs cover efforts to detect and correct problems prior to the user detecting them, while external failure costs cover the effort to correct problems discovered by users. Table 22, from a paper by Demirors and Yildiz [Demirors, 2000], provides examples of costs in each of the categories.
Figure 12: Cost of Quality [Webb, 2008] Table 22: Costs of Software Quality Categories [Demirors, 2000]
External failure costs can be particularly insidious. The obvious costs are those associated with correcting defects, which, as shown previously, can be relatively high after software is deployed. The costs of correcting defects, however, often pale in comparison to other costs. Figure 13 [Mcgibbon, 2009] illustrates the potential costs to a customer that can result from defective software.
Figure 13: Costs of external failures [Mcgibbon, 2009]
Some writers, such as Rex Black, classify prevention and appraisal costs as "costs of conformance", and internal and external failure costs as "costs of nonconformance" [Black, 2000]. Black provides a return on investment (ROI) analysis for use of formal testing . Table 23 shows that the cost of conformance, which includes appraisal costs associated with manual and automated testing, are much less than the costs of nonconformance. This assumes that bugs found during testing cost $10 to fix while bugs found by the customer (external failure costs) are $1,000. While this example is hypothetical, Black cites an actual case where software systems costs were reduced by 50% by using appraisal techniques like formal testing.
While formal testing provides a positive ROI, inspections have been shown to be even more effective. Gack [Gack, 2009] showed an ROI of 4.6 from using inspections and testing compared to using testing alone as an appraisal method. El Emam [El Emam, 2005] showed that inspections are effective in that the effort needed to correct defects using design or code inspections is only 25% of the effort needed to correct defects during testing. The ROI from using inspections instead of testing alone, in terms of percentage savings, is 11% for pre-release costs, 26% for post-release costs, and 38% for customers. The use of inspections results in a 3% schedule savings, which shows that quality improvements from inspections do not add time to a software project. Freimut [Freimut, 2005] described the results of using inspections at Siemens AG, Germany. The cost of rework was a 67% mean decrease measured after the improvements for inspections performed in the analysis phase; the cost of rework was a 47% mean decrease measured after the improvements for inspections performed in the design phase; and the cost of rework was a 12% mean decrease measured after the improvements for inspections performed in the development, or coding phase. Table 24, from the DACS ROI Dashboard and included in the 2007 DACS report, "A Business Case for Software Process Improvement" [Ferens, 2007] summarizes the results from multiple resources for using software inspections as a software process improvement activity. The average ROI was 6.84, and the minimum was 2.0, which shows that ROI will almost always be positive from using inspections. Table 24: ROI from Inspections [Ferens, 2007]
El Eman [El Emam, 2005] shows how ROI is usually positive from using other defect prevention and appraisal methods. He shows how to compute ROI for various methods, including inspections. Suggestions for realizing ROI from current projects
The primary benefit from using defect tracking methods such as testing and inspections is the positive return on investment. However, there are some benefits that are hard to quantify, yet will result in a positive return. One area is schedules; there will be less time required for nonconformance activities; hence, less time required for the overall effort. Customer satisfaction will be greater when prevention and appraisal activities are emphasized since products will be of higher quality. Also, there is a positive impact on team morale with higher quality products, which can reduce turnover rates and increase productivity in general. Try to implement processes such as inspections and formal testing which almost certainly will result in a high positive ROI. As shown earlier, these methods are effective at finding defects during the phase in which they occur, which will reduce rework (and frustration).
1.3.3.2 Defect Prevention for Future Projects
Most defect tracking and removal methods discussed so far address detecting and removing defects as soon as possible after they are injected into the current project during requirements, design, etc. Defect prevention, according to Tian [Tian, 2005], is to reduce the chances of defects being injected in the first place. Soni states that "prevention is better than cure" is a saying that applies to defects in the software development cycle as well as illness in medical science (see Defect Prevention: Reducing Costs and Enhancing Quality). Version 1.2 of the CMMI for Development (CMMI-DEV) Process Area of Causal Analysis and Resolution has the purpose of identifying the causes of defects and other problems and take action to prevent them from happening in the future [Konrad, 2003]. It would appear that defect prevention can have even more positive impact than defect appraisal activities like walkthroughs and inspections. According to Schulmeyer and McManus, the defect prevention process has a 15-to-1 payback and avoids 50% of all defects normally injected during the first year and first project in which it is used, and 70% avoidance within 2 or 3 years [Mcmanus, 1999]. The defect tracking and removal activities discussed throughout this document are prerequisite for defect prevention. For example, the Causal Analysis and Resolution Process Area for the CMMI-DEV has the specific goals (SGs) and specific practices (SPs) that are shown in Table 25. The table shows the SGs and SPs associated with this CMMI Process Areaa, and the subpractices associated with each practice: Table 25: Causal Analysis and Resolution Process Area [Konrad, 2003]
In Table 25, the first subpractice listed, "Gather relevant defect or problem data", implies that an organization is already performing the defect tracking activities discussed previously. Also, SP 2.2, "Evaluate the effects of changes", implies that an organization will track defects in future projects to see if the defect prevention measures proposed are indeed effective. In the CMMI-DEV, Causal Analysis and Resolution is a "Maturity Level 5" Process Area. Level 5 is the highest maturity level; it implies an organization is not only actively collecting quantitative data (such as defect data), but is using this data to continuously improve their processes. Defect prevention, then, requires not only a high level of capability in other CMMI Process Areas, but also requires a high degree of commitment from an organization to perform defect prevention activities. Table 25 shows that defect prevention can be very involved; it requires a lot of time and effort. However, the rewards can be worth it. Version 1.2 of the CMMI for Acquisition (CMMI-ACQ) also has Causal Analysis and Resolution as a Maturity Level 5 Process Area for acquisition agencies. According to the CMMI-ACQ, acquirers use Causal Analysis and Resolution practices to guide identification of root causes of selected defects and other problems and taking action to prevent their reoccurrence [CMMI PRODUCT TEAM, 2007]. The same SGs and SPs apply, although the acquirer will focus on defects reported by customers, end users, and suppliers. In either case, causal analysis is best performed by people who have the best understanding of the selected defect or problem being studied [Hofmann, 2007]. If an organization desires to perform defect prevention activities, they must be ready. In his "Quality Questionnaire", El Eman asks the following questions for defect prevention [El Emam, 2005]:
Question 3 shows that defect prevention goes far beyond defect removal. Defect removal involves removing current defects, while defect prevention removes the very causes of these defects. In Figure 14, Soni illustrates the activities associated with defect prevention and compares them with activities associated with defect tracking and removal. The area in gray represents those areas associated with "the current philosophy" of handling defects: detection, tracking/documenting and analysis of defects to arrive with quick, short-term solutions" (from Defect Prevention: Reducing Costs and Enhancing Quality).
Figure 14: A Defect Prevention Cycle (from Defect Prevention: Reducing Costs and Enhancing Quality) When an organization determines it is ready for a defect prevention program, the organization needs to plan and document its program. There are several resources that explain how to set up a program and that provide examples. For instance, Figure 14 illustrates some steps in implementing a defect prevention program. Kan also lists three steps for a defect prevention program: Analyzing existing defects or errors to trace root causes; suggesting preventive actions to eliminate the defect root causes; and implementing the preventive actions [Kan, 1995]. This process consists of four elements:
The Software Productivity Consortium has issued a Defect Prevention Guidebook which addresses a recommended defect prevention process that can be implemented on a single project or across the organization, the organizational and project roles and responsibilities, and a set of defect reduction strategies. The Guidebook also provides information for process improvement change agents and project managers for classifying defects from inspections to provide causal analysis, performing causal analysis, identifying defect prevention elements with the highest ROI for defect prevention investment, and planning and implementing preventive actions [Brown, 1998]. The Defect Prevention Guidebook explains all process elements in detail, devoting an entire chapter to each key area. An example is a chapter on performing causal analysis, where the steps are: forming a causal analysis team; holding causal analysis meetings (like Kan's first step); and implementing defect prevention strategies. Holding meetings is further divided into six steps: selecting problems; classifying problems; identifying systematic errors; determining principal causes; developing corrective action proposals; and documenting meeting results. An example of a tool that can be used in identifying systematic errors (the same defect being repeated on different occasions) is a Pareto Chart for coding defects, which is shown in Figure 15. The Pareto chart shows the number of defects by cause in descending order, with the leftmost bars showing the most common causes, which subsequently should probably receive the most attention. The Guide contains an appendix that describes several useful tools, including more information about Pareto charts.
Figure 15: Pareto Chart Used In Causal Analysis [Brown, 1998]
While the Guidebook and other resources give useful information about setting up a defect prevention program, the project or organizational manager or team needs to determine the best processes to use in setting up a program for a specific project or organization. Some resources do provide guidance in this area; for example, Capers Jones provides some useful guidance on which software development methods are most useful for which categories of defects, or errors. One or more of these development methods can then be used in a defect prevention program. According to Capers Jones, "defect prevention" means "the overall set of technologies that simplify complexity and minimize a natural human tendency to make errors while performing complex tasks" [Jones, 2008]. Table 26 shows some development tools and methods, such as Joint Application Design (JAD) and Quality Function Deployment (QFD), with their effectiveness for different categories of defects. Suggestions for Defect Prevention
Defect prevention can greatly enhance quality and ROI for projects beyond what defect appraisal methods, including tracking and removal methods, alone can do. However, defect prevention can only be undertaken for projects and organizations that are already effective at defect appraisal activities, including defect tracking and removal activities discussed thus far. The CMMI-DEV Process Area of Causal Analysis and Resolution summarizes the goals, practices, and methods of associated with defect prevention. There are several resources, including the Defect Prevention Guidebook [Brown, 1998], that can help an organization or project manager establish a defect prevention program, and other resources, such as Capers Jones' book, "Applied Software Measurement" [Jones, 2008] can help determine specific strategies or ideas for a particular project or organization.
2.0 Relationships to Other Practices
A Best Practice is rarely successful as a stand-alone process. The DACS Gold Practice concept extends beyond typical Best Practice implementation by emphasizing the integration of other implicit or explicit practice/process concepts that may influence, or be influenced by, the Gold Practice. Figure 16 is a high-level process architecture for the practice of Defect Tracking Against Quality Targets, depicting relationships to this practice and the nature of the influences on the practice. Defect Tracking Against Quality Targets uses as inputs information from Gold Practices associated with setting quality gates and tracking defects, while the outputs of Defect Tracking Against Quality Targets are information for other Gold Practices which are associated with analyzing data and identifying defect causes and corrective action, implementing and verifying corrective action, and assessing and reporting quality improvements. Thus, Figure 16 shows feedback. These relationships are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may influence (or be influenced by) the ability to successfully implement other practices. Section 2.1 discusses the inputs to this Gold Practice, while Section 2.2 discusses outputs. ![]() Figure 16: Gold Practice "Defect Tracking Against Quality Targets" Relationship to other Gold Practices
2.1 Inputs
This section describes the Gold Practices that are inputs to the Gold Practice, "Defect Tracking Against Quality Targets". The two primary inputs are the identification of quality gates and methods for detect detection. While information in both areas is included in this Gold Practice, the other Gold Practices shown as "inputs" in Figure 16 may be consulted for further insight into these input areas. Table 27 describes the inputs from other Gold Practices, and how those Gold Practices contribute to these inputs. Table 27: Inputs to Gold Practice, "Defect Tracking against Quality Targets"
2.2 Outputs
This section describes the Gold Practices that use outputs from the Gold Practice, "Defect Tracking Against Quality Targets". The three primary outputs are analyzing data and identifying defect causes and corrective action, implementing and verifying corrective action, and assessing and reporting quality improvements. While information in these three areas is included in this Gold Practice, the other Gold Practices shown as "outputs" in Figure 16 may be consulted for further insight into these output areas. Table 28 describes the outputs to other Gold Practices, and how those Gold Practices use these outputs. Table 28: Outputs From Gold Practice "Defect tracking Against Quality Targets"
3.0 Definitions
Defect
Both Capers Jones and the SPMN define a defect as “a problem or 'bug' that, if not removed, could cause a program to either produce erroneous results or otherwise fail” [Jones, 2008] [SPMN, 2000]. According to Pressman [Pressman, 2005], the terms defect, bug, error, and fault are usually synonymous; however, Pressman himself uses the term “error” to describe quality problems found before a product is released and “defect” to describe problems after release. Schulmeyer and McManus also differentiate between errors and defects in that errors are deviations from requirements that arise during a process while defects are specific kinds of unwanted conditions delivered to the customer [Mcmanus, 1999].
In IEEE Std 1044-1993(R2002), [IEEE STANDARDS BOARD, 2002], the term anomaly is preferred to the term defect because an anomaly may not represent a problem, but a condition which can be enhanced. (For the purposes of this Gold Practice, the SPMN / Jones definition will be used.).
According to the IEEE Standard Glossary of Software Engineering Terminology [IEEE STANDARDS BOARD, 1990], a fault is an incorrect step, process, or data definition in a computer program. Although this term is often used synonymously with errors or bugs, it is the term used in the fault tolerance disciplines. Also, IEEE gives other definitions for the term “error”. Fault tolerance is the ability of a system or component to continue normal operation despite the presence of faults. Another view of fault tolerance is that it prevents a fault, or error, from resulting in a failure, which IEEE defines as “the inability of a system or component to perform its required functions within specified performance requirements” [IEEE STANDARDS BOARD, 1990]. However, as discussed in Chapter 11 of the SWEBOK, different cultures and different standards may use several different meanings to define these terms [Abran, 2004].
Defect Tracking
One definition of defect tracking by Tian is monitoring and recording what happens to each defect after its initial discovery until its final resolution [Tian, 2005]. Here, defect tracking is listed as one activity for defect handling; the others are defect resolution and defect logging.
Quality Targets
Version 1.2 of the CMMI for Development, as explained in [Chrissis, 2007], defines quality and process-performance objectives as “Objectives and requirements for product quality, service quality, and process performance. Process-performance objectives include quality; however, to emphasize the importance of quality in the CMMI Product Suite, the phrase quality and process-performance objectives is used rather than just process-performance objectives”. Tian [Tian, 2005] describes the process of setting quality goals as part of a quality planning process. Part of a quality assessment and improvement process is determining whether the goals are satisfied.
4.0 Sources
The following paragraphs summarize the chronology/history of defect tracking against quality targets and highlight a few key elements.
Any organization that staffs a call center to which users can report problems will track problems, so defects have been tracked to some extent for decades. Processes that use tracked defects to quantitatively assess the quality of software processes and products are of comparatively more recent origin.
Developments in processes for tracking software defects parallel developments in hardware processes, often with lags. Hardware engineers have developed a variety of reliability models, methods for allocating system reliability to components, procedures for accelerated life testing, and techniques to conduct formal statistical tests of reliability [Mann, 1974]. Researchers in software engineering modified this technology, developing families of software reliability growth models and techniques for using those models to specify and measure reliability [Lyu, 1996].
Hardware systems are often developed and supported with a Failure Reporting Analysis and Corrective Action System (FRACAS). An effective FRACAS system captures:
Software organizations also track defects. The IEEE has developed a standard [IEEE STANDARDS BOARD, 2002] and a guide [IEEE STANDARDS BOARD, 1996] for classifying anomalies. The guide states that immature organizations typically only “record the facts and impressions about an anomaly when it is encountered so that it can be understood…at a later date”. The information described in the standard “will facilitate the discovery of common errors made, most effective activities to find anomalies, the place where anomalies were first introduced and other telling facts about your development process.” Analysis of the data provided through use of the standard “can be used for project management, process improvement, and product assessment” [IEEE STANDARDS BOARD, 1996].
Other techniques have been developed for using software defect data for more than qualitative causal analysis. Orthogonal Defect Classification (ODC) is one such technique. ODC is a systematic framework enabling in-process feedback to software developers, the evaluation of software processes, and other measurement and analysis techniques [Bhandari, 1992].
4.1 Recommending Sources
This section identifies segments from specific DoD, industry, or academic sources that recommend, specify or otherwise support adoption of the subject practice either directly, or as part of a broader best practice initiative, or a practice with a different name.
1. Capability Maturity Model Integration for Development (CMMI-DEV)
Elements of defect tracking against quality targets are spread across several Key Process Areas in the CMMI-DEV [Paulk, 1994].
Table 29: Defect Tracking in the CMMI-DEV
2. Turner, R., “Implementation of Best Practices in US Department of Defense Software-Intensive Systems”, Dissertation, George Washington University, January, 2002 [Turner, 2002]
Dr. Richard Turner, under the sponsorship of the Software-Intensive Systems Office of the Deputy Under Secretary of Defense (at the time, Science and Technology), surveyed 14 military software centers. Seven responded, with data on 150 software acquisition programs (53% Air Force, 26% Army, 21% Navy). Turner found that 68% of these programs had partially implemented defect tracking against quality targets, while 31% had full implementations. The table shows the perceived effectiveness of defect tracking against quality targets, according to these survey respondents.
Table 30: Turner Survey Results
Turner conducted a second survey of experts from academia, industry, and government. The 35 respondents to this survey ranked defect tracking against quality targets thirteenth of 32 in terms of perceived effectiveness.
3. Cleanroom Software Engineering
Cleanroom processes have common measurement elements. These measurement elements include defect tracking:
-Measure the Process
Measure the effectiveness of a review in terms of the percentage of all defects originating prior to the review that are found in the review. These percentages are determined, of course, after execution testing.
-Measure the Product
Measure the quality of work products that define the software in terms of the percentages of execution failures that are traced to defects in the work products. These percentages are determined, of course, after execution testing” [Abran, 2004].
4. Nine Best Practices: The Software Management Framework, 3 June 1999
The Navy’s Software Program Manager’s Network (SPMN) convened a group of experts in 1995 at the Airlie Center outside Warrenton, Virginia (the Airlie Council). The Airlie Council originally identified nine best practices, of which defect tracking against quality targets was one. The council provided a rationale for defect tracking against quality:
"In many cases there is no defect tracking mechanism at the project level. Keeping bug reports is not sufficient, the metrics of who, what, where, when, how, and why are as important as the existence of a simple bug list. By tracking these metrics, basic indicators relating to defects, schedule, cost, requirements, documentation and staff can be made visible."
Appendices
Bibliography
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||










