GP 28 - Defect Tracking Against Quality Targets 1.2

A process for analyzing and classifying defects and problem reports is an important part of defect tracking against quality targets. Some classification schemes are described here: an IEEE standard classification categorization, an Orthogonal Defect Classification (ODC), and other classification schemes.

IEEE Standard Anomaly Classification

The Software Engineering Standards Committee of the IEEE Computer Society has created a standard classification for software anomalies [Konrad, 1997]. In the IEEE framework, an anomaly can be a failure occurring during run-time or a static fault located in the code or documentation. The committee chose the term �anomaly� to convey a more neutral connotation than some synonyms. Figure 6 shows the process for classifying anomalies. The steps for identifying the impact of each classification step results in the specification of the value of various categories. The standard defines two types of categories, mandatory and optional. The mandatory categories are shown in Table 7.

Figure 6: IEEE Standard Anomaly Classification Process

Table 7: IEEE Standard Mandatory Classifications

RECOGNITION
Category	Classification
Project Activity	Analysis, Review, Audit, Inspection, Code/Compile/Assemble, Testing, Validation/Qualification Testing, Support/Operational, Walk-through
Project Phase	Requirements, Design, Implementation, Test, Operation and Maintenance, Retirement
Symptom	Operating System Crash, Program Hang-up, Program Crash, Input Problem, Output Problem, Failed Required Performance, Perceived Total Product Failure, System Error Message, Other
INVESTIGATION
Category	Classification
Actual cause	Product, Test System, Platform, Outside Vendor/Third Party, User, Unknown
Source	Specification, Code, Database, Manual and Guides, Plans and Procedures, Reports, Standards/Policies
Type	Logic Problem, Computation Problem, Interface/Timing Problem, Data Handling Problem, Data Problem, Documentation Problem, Document Quality, Enhancement
ACTION
Category	Classification
Resolution	Immediate, Eventual, Deferred, No Fix
IMPACT
Category	Classification
Severity	Urgent, High, Medium, Low, None
Project Schedule	High, Medium, Low, None
Project cost	High, Medium, Low, None
DISPOSITION
Category	Classification
Disposition	Closed, Deferred (Reference), Merged with Another Problem (Reference), Referred to Another Project (Reference)

Orthogonal Defect Classification

Orthogonal Defect Classification (ODC) is a multidimensional in-process measurement system supporting sophisticated analyses. For example, the variation in the distribution of defect types by lifecycle phase can provide signatures of a process. ODC provides a means to compare observed distributions with expected distributions and thereby determine if a project�s logical location among phases by signature matches the scheduled location. Under ODC, defects are categorized:

"into classes that collectively point to the part of the process that needs attention, much like characterizing a point in a Cartesian system of orthogonal axes by its (x, y, z) coordinates... For classifications to be widely applicable, the classification must have consistency between the [lifecycle phase]. Without consistency, it is almost impossible to look at trends across stages. Ideally, the classification should be quite independent of the specifics of a product or organization" [Bhandari, 1992].

Early presentations of ODC [Bhandari, 1992] discussed two attributes of a defect: Type and Trigger. "A defect trigger is a condition that allows a defect to surface" [Bhandari, 1992]. For example, a defect of a certain type might not be visible until the software is run on a certain hardware platform. The hardware platform, then, is the trigger. A later IBM implementation had two additional attributes: Source and Impact [Lyu, 1996]. Tables 8 and 9 present how defects are classified in one implementation of ODC which has been recently described in the literature.

Table 8: Attributes Identified When Defect Uncovered

Activity and Triggers				Customer Impact
Inspection	Unit Test	Function Test	System Test	Customer Impact
Design conformance	Simple path	Coverage	Workload/stress	Installability
Logic/data flow	Complex path	Variation	Startup/restart	Standards
Lateral compatibility		Sequencing	Recovery/exception	Standards
Backward compatibility		Interaction	Hardware configuration	Integrity/Security
Language dependency			Software configuration	Migration
Concurrency			Blocked test	Reliability
Internal Document				Performance
Side effects				Documentation
Rare Situations				Requirements
				Maintenance
				Usability
				Accessibility
				Capability

From [Bassin, 2001]

Table 9: Attributes Identified When Design Or Code Defect Fixed

Target	Defect Type	Qualifier	Source	Age
Design/Code	Assignment/initialization	Missing	Developed in-house	Base
	Checking	Incorrect	Reused from library	New
	Algorithm/method	Extraneous	Outsourced	Rewritten
	Function/class/object		Ported	Refixed
	Timing/serialization
	Interface/OO messages
	Relationship

From [Bassin, 2001]

Other Classification Schemes

There are some defect classification schemes that are less elaborate than the two methods explained above, but may be adequate in some situations. Putnam and Myers, who include defect rates as one of the five core metrics for successful software management [Myers, 2003], explain a defect severity ranking that is based on the time limit in which the defect must be fixed. The classification is as follows:

Critical: The program fails to execute or it leads to a morass no one wants to enter. These defects must be corrected before the program can be executed again.
Serious: Some results are wrong, or performance is substantially degraded, but the user can temporarily make allowances for the deficiencies. These defects should be fixed "soon".
Moderate: The program executes, but it contains obvious deficiencies. Users can live with them temporarily. These should be fixed in the current release.
Cosmetic: Flaws do not affect program performance or results, such as a misspelled table heading. These should be fixed in the next release.

Fenton and Pfleeger [Fenton, 1997] also include a single classification scheme where defects are classified in one of five categories based on relative severity. The five categories are "major", "minor", "negligible", "documentation", or "unknown". Fenton and Pfleeger point out, however, that such a classification scheme is not truly orthogonal in that, for instance, a documentation error could be a major problem. They propose eight mutually-independent (or orthogonal) attributes of defects that, ideally, should be described:

Location: Where did the problem occur?
Timing: When did it occur?
Symptom: What was observed?
End result: What consequences resulted?
Mechanism: How did it occur?
Cause: Why did it occur?
Severity: How much was the user affected?
Cost: How much did it cost?

El Emam [El Emam, 2005] proposes three schemes for classifying defects: priority, severity, and problem type. Each of the three schemes will now be explained:

1. Defect Priorities: This scheme allows the software developers to focus on the most urgent defects first. Table 10 shows two possible classifications for defect priority.

Table 10: Defect Priority Classifications [El Emam, 2005]

Defect Priority Classification I:

Immediate fix: blocks further testing, very visible
Must fix the defect before release
Should fix as time permits
Would like to fix, but can release as is

Defect Priority Classification II

Fix immediately
Fix before alpha release
Fix before beta release
Fix before general availability
Fix before next release

2. Defect Severity: This relates to how a defect affects the usability and utility of a product. Unlike priority, which may change during a project, severity levels tend to remain the same throughout the project. Table 11 shows a defect severity scheme.

Table 11: Defect Severity [El Emam, 2005]

Corruption or Loss of Data
- - Inability to meet response or capacity constraints
- - Unintentional halt or failure
- - Failure that does not return to service within a mean time to repair
System Inadequate
- - Non-compliance of a main function within specifications
- - Any critical fault that occurs within system testing
Unsatisfactory
- - Non-compliance of an auxiliary function to specifications
- - Non-compliance with standards for deliverable items
- - Layout or format defects on presentation that do not affect system operational integrity
- - Inconsistency or omission within a document
Suggestion
- - Suggestion for improving the software or documentation

3. Problem Type: This looks at the type of problem by defect that is fixed. According to El Emam [El Emam, 2005], Table 12 is a problem type scheme that has been shown to be reliable in that different developers are expected to assign the same defects to the same categories. For each type, a brief description is included, and some questions which, if answered "no", would indicate a defect in that category. This classification can be useful in process monitoring and improvement.

Table 12: Problem Type [El Emam, 2005]

Documentation
- - Description: Comments and messages, Inability to meet response or capacity constraints
- - Questions:
- - Is the function described adequately at the top of the file?
- - Are variables described when declared?
- - Does the function documentation describe its behavior properly?
Interface
- - Description : Procedure calls and references, I/O, user formats, declarations
- - Questions:
- - Does the library interface correctly divide the functions into their different types?
- - Do the functions allow the proper object access rules?
- - Are the declared and expected interface signatures the same?
Function
- - Description: Logic, pointers, loops, recursion, computation
- - Questions:
- - Are all branches handled correctly?
- - Are pointers declared and used as pointers?
- - Are arithmetic expressions evaluated as specified?
Build/Package/Merge
- - Description: Change management, library, version control
- - Questions:
- - Is there a version number defined for this file?
- - Are the correct versions of functions included in the build?
Assignment
- - Description: Declaration, duplicate names, scope, limits
- - Questions:
- - Are variables initialized properly?
- - Are all library variables that capture a characteristic or state of the object defined?
- - Are all return values that are special cases (e.g., an error return) really invalid values (i.e., would never occur unless there was an error)?
Memory
- - Definition: Memory allocation, leaks
- - Questions:
- - Are objects instantiated before being used?
- - Do all objects register their memory usage?

Suggestions for Defect Classification

There are many different methods for classifying, or categorizing defects, a few of which are described here. The method or combination of methods chosen should be dependent on the particular needs of a project or program; however, the information presented here should be helpful.

1.2.2.3 Defect Resolution and Removal

After defects are detected , analyzed, and classified, it is important to remove them as soon as possible. Of course, they should be removed in the priority order set up in classification, but it is highly desirable to remove defects during the phase they occur to the maximum extent possible. Figure 7 shows a phase containment matrix [Ferens, 2008] which compares the phase in which defects are detected with the phase in which they are removed. (This is similar to Figure 3, which compares phases where defects are inserted and detected.) If defects are removed in the phase they are detected, this can be considered part of the normal development process; however, if they are removed in later phases, some degree of rework will be needed, and this can be costly. According to Kan [Kan, 1995] and others, removing defects during testing is more expensive than removing them in earlier phases. The red (or dark) areas on this chart illustrate out-of-phase removal. Generally, the later the defect is removed after it is detected, the greater degree of rework that will be required.

Figure 7: Phase Containment Matrix [Ferens, 2008]

Figure 8 is another illustration of the benefits of removing (or repairing) a defect during the phase it occurs. Here, the relative cost to remove a requirements defect is shown; it is much more expensive to correct during later phases of a program or project. During requirements analysis, a requirements defect or error correction will only involve something like a change to a paragraph in a requirements document. If this defect is repaired during testing, it will involve not only changing a paragraph in a document, but can involve redesigning certain components, and re-coding and re-testing them.

Figure 8: Relative Cost to Remove a Requirements Defect [Ferens, 2008a]

Some techniques such as reviews, walkthroughs, inspections, and even testing are sometimes described as defect removal techniques. In reality, the purpose of these techniques is to discover or detect defects; defect removal itself may require additional techniques or methods. Table 13 illustrates the process for Fagan inspections, Note that it is the author's responsibility to fix (correct or remove) defects, and not the responsibility of the inspection team.

Table 13: Fagan Inspection Activities [Gack, 2009]

Stage	Activities
Plan	Select work product (250-300 lines of code) Select inspectors (3-5) Conduct orientation
Prepare	About 2 hours Each inspector, individually; frame questions
Meet	About 2 hours "Reader" paraphrases each line Author logs defects
Fix	Author corrects defects
Close	Author + Moderator Re-inspect if indicated
Report	Monitor "x's" Document benefits

Before defects can be removed, they must first be resolved; the cause of the defect must be determined in order to determine how to remove the defect. One Process Area of the Software Engineering Institute (SEI) Capability Maturity Model Integration (CMMI) is "Causal Analysis and Resolution" [Konrad, 2003]. Some of the "specific practices" for this Process Area are to "determine the cause of defects" and "analyze causes". If the cause of a defect can be removed, the defect itself may be removed. This Process Area is more associated with preventing future defects than correcting current ones; however, performing these two specific practices can aid in defect resolution and subsequent removal. According to Tian [Tian, 2005], inspections can be expanded to include causal analysis, which may later be used in defect prevention, but can also be used in defect resolution.

Defect removal is sometimes included in the resolution process [Tian, 2005]; however, this Gold Practice will treat removal as a separate set of activities after the cause of a defect is resolved. As described in the CMMI, removal is often performed by a separate group from those discovering the defect, as is often the case in formal testing [Konrad, 2003]. In some cases, such as unit testing, the person discovering the defect also corrects it. According to Kan [Kan, 1995], defect removal is associated with activities at the end of development phases. Table 14 shows the activities associated with defect injection and removal.

Table 14: Defect Injection and Removal Activities [Kan, 1995]

Development Phase	Defect Injection	Defect Removal
Requirements	Requirements gathering and development of functional specifications	Requirements analysis and review
High-level design	Design work	High-level design inspections
Lower-level design	Design work	Low-level design inspections
Code implementation	Coding	Code inspections
Integration / build	Integration and build process	Build verification test
Unit test	Bad fixes	Testing itself
Component test	Bad fixes	Testing itself
System itself	Bad fixes	Testing itself

The specific processes used for removing defects will vary by phase and by project. In the case of requirements defects, the defect may be removed (or corrected) by the author merely rewriting a sentence or paragraph in a requirements specification or similar document. During design, removal may involve redoing a sequence diagram or a hierarchical chart. Coding errors, usually discovered during unit testing, are relatively easy to correct if they are syntax errors (such as spelling "class" as "glass") since compilers and interpreters usually identify these; however, logic errors (such as subtracting two numbers instead of adding them) can be quite arduous. Component and higher-level testing defects or errors are probably the most difficult to remove since they involve effort to isolate the defect, then retest the module where it occurred and possibly other modules to be sure they are unaffected by the defect removal. For detecting and removing testing defects, and some coding and design defects, automated tools may be useful or even necessary. (A listing of tools is included in the Appendix.)

Sometimes the removal process does not completely remove the defect or unintentionally introduces another defect. According to Capers Jones [Jones, 2008], these are sometimes called "bad fixes", new defects that are accidentally induced in an attempt to repair (or remove) previous defects. Capers Jones also uses the term "bad fix defect injection" to describe this, and states that about 7% of all defect repairs will exhibit this phenomenon. Jones says that this can rise to 20% for very complex and poorly structured applications [Jones, 2007].

Suggestions for Defect Analysis and Removal

Some methods for defect detection do not inherently result in defects being resolved and removed. For example, the purpose of some walkthroughs and inspections is only to identify defects; it is up to the software developers to correct them. For defects detected, the cause should be assessed, and the defect removed (or corrected) as soon as possible. Every effort should be made to remove defects in the same phase as they are detected; otherwise, rework will be needed, and the removal will probably cost much more. Also, realize that defects discovered later in the development process, such as testing, will be more expensive to remove than those discovered earlier, even if they are removed during the same phase.

1.2.2.4 Defect Removal Tracking

When defects are removed, or repaired, it is important to keep track of the removals as well as the occurrence of defects. Figure 9 is an example of a defect tracking chart where both incoming and corrected defects are plotted.

Figure 9: Incoming and Corrected (Removed) Defects [Ferens, 2010]

One key measure of quality that can be obtained from keeping track of defect removals is "defect removal efficiency". According to Capers Jones, defect removal efficiency is "the percentage of defect potentials that will be removed before the software is delivered to its users or customers" [Jones, 2008]. Defect potentials, in turn, are the probable numbers of defects that will be found during the development of a software application. Table 15 is an illustration of defect removal efficiency, with the potentials measured in "defects per function point", a measure of software size.

Table 15: Defect Removal Efficiency [Jones, 2008]

Defect Origin	Defect Potential	Removal Efficiency	Defects Remaining
Requirements Defects	1.00	77%	0.23
Design Defects	1.25	85%	0.19
Coding Defects	1.75	95%	0.09
Documentation Defects	0.60	80%	0.12
Bad Fixes	0.40	60%	0.12
Total	5.00	85%	0.75

The defect removal efficiency of 85% indicates that the end product delivered to the user will have 15% of the defects it would if removal did not take place. It should be noted that the 85% is an average for the development process. Some defects, like bad fixes, are harder to remove than others, like coding errors.

Kan discusses a similar term, defect removal effectiveness [Kan, 1995], which investigates the defect removal for each step, or phase, of the software development process. The defect removal effectiveness for each step is as follows:

Defect Removal Effectiveness = _________Defects Removed (at the step)______________ x 100%

Defects existing on step entry + defects injected during the step

An example is, during high level design, Defects removed = 730, defects from previous phases = 122, and defects injected during this step = 859, then Defect Effectiveness = 730 / (122 + 859) x 100% = 74%. According to Kan, these measures are useful in quality planning and quality management. In the narrative for the CMMI Process Area "Organizational Process Performance", it is suggested that Defect Removal Effectiveness can be quite useful as a process performance measure [Konrad, 2003].

Suggestions for Defect removal tracking

In addition to other quality measures (discussed in detail in Section 1.2.2.5), defect removal efficiency and effectiveness can be very useful in assessing quality targets and how well they are met. Collecting defect removal information during all development phases must be done if these measures are to be used.

1.2.2.5 Measuring Defects (Against Quality Targets)

In addition to defect removal tracking discussed in the previous section (1.2.2.4), other defect measures can be useful in assessing whether targets are met as well as facilitating software project and process management in general. This section describes some measures and methods for selecting measures that can be useful in defect tracking against quality targets.

The Practical Software and Systems Measurement (PSM) Guide

An excellent source of measures for software projects in general, including quality measures, is the Practical Software and Systems Measurement (PSM) Guide [Mcgarry, 2002]. Section 3 of this Guide explains the Issue-Category-Measure (ICM) approach to selecting measures for software projects. The Guide lists more than 50 measures and provides a detailed description of each measure. Table 16, from the PSM Guide, shows measures that can be used for selected issues and categories within issues. Some of the measures that can be used in defect tracking against quality targets are now summarized.

Table 16: Issue-Category-Measure table [Mcgarry, 2002]

Issue - Category - Measure Mapping
Common Issue Area	Measurement Category	Measures
Schedule and Progress	Milestone Performance Work Unit Progress Incremental Capability	Milestone Dates Critical Path Performance Requirements Status Problem Report Status Review Status Change Request Status Component Status Test Status Action Item Status Increment Content - Components Increment Content - Functions
Resources and Cost	Personnel Financial Performance Environment and Support Resources	Effort Staff Experience Staff Turnover Earned Value Cost Resource Availability Resource Utilization
Product Size and Stability	Physical Size and Stability Function Size and Stability	Database Size Components Interfaces Lines of Code Physical Dimensions Requirements Functional Change Workload Function Points
Product Quality	Functional Correctness Support-Maintainability Efficiency Portability Usability ependability-Reliabilty	Defects Technical Performance Time To Restore Cyclomatic Complexity Maintenance Actions Utilization Throughput Timing Standard Compliance Operator Errors Failures Fault Tolerance
Process Performance	Process Compliance Process Efficiency Process Effectiveness	Reference Model Rating Process Audit Findings Productivity Cycle Time Defect Containment Rework
Technology Effectiveness	Technology Suitability Impact Technology Volatility	Requirements Coverage Technology Impact Baseline Changes
Customer Satisfaction	Customer Feedback Customer Support	Survey Results Performance Rating Requirements For Support Support Time

Defects: This refers to the number, status, and priority of defects reported. It can include the average age of defects, which is related to the defect removal rate discussed earlier. The defect status can include the closure rate of defects, also an indicator of defect removal rate.
Operator errors: This measure is useful in assessing quality in the operational environment, after a system or program has been deployed. It can be used to measure usability and quality of a system or program, and can be compared to any targets set in these areas. The limitation of this measure is that it is only useful after a system has been deployed.
Failures: This is a measure of the number, criticality, and time interval between failures. It is useful in measuring software reliability in terms of mean-time-to-failure (MTTF) and can be compared to MTTF targets if they are established. Like operator errors, this measure is useful only after system deployment.
Defect Containment: As discussed in Sections 1.2.2.1 and 1.2.2.3, it is highly desirable to detect defects during the phase when they occur, or are inserted. It is also highly desirable to remove, or correct defects during the phase when they are detected. The defect containment measure, as discussed in Section 3 of the PSM Guide, measures the number of defects found during a process (or phase) and the number of defects found after completion of a process, or phase, such as during a design review or inspection. This measure can be used to determine the effectiveness of inspections and other defect detection methods used, and can be compared to any targets or goals set for these methods. This measure could be expanded to include defect removal tracking, as discussed in Section 1.2.2.4.
Rework: This measure tracks the amount of effort required to fix, or remove defects. An organization sometimes sets a budget for rework activities, and this measure can be used to compare actual rework effort to budgeted rework effort. As discussed in previous sections, it is desirable to detect and remove defects as early as possible to reduce rework required. This measure can show how well an organization is performing in this area.
Survey Results: Survey Results track customer satisfaction ratings, from both acquiring agencies and users. Survey Results also measures progress in providing products and services that meet the customer�s needs and expectations. One challenge with this metric is setting an objective scale for measurement. Often, users are asked to rate a product on a scale of "1" to "5", or "Very Satisfied", "Satisfied", etc. Even with a seemingly objective scale, users' ratings will often be based on subjective perceptions. Also, this measure, like operator errors and failures, is not useful until after deployment. Still, organizations sometimes set quality targets in terms of user satisfaction ratings, especially in the commercial environment, and this measure is useful in assessing successes in this area.

Of course, some of the other measures in Table 16 can be used in quality assessment. The six above are just provided as examples. As outlined in the PSM Guide, organizations can use Table 16 and the information in Section 3 to determine the specific issues and categories for their programs and choose the appropriate measures.

Goal-Question-Metric (GQM) Approach

As described by Ellen Walker as a DACS Gold Practice [Walker, 2005], and from the writings of Dr. V.R. Basili [Marciniak, 1994] and Rini van Solingen [Van Solingen, 1999], the GQM approach defines certain goals for a software project, refines these goals into one or more questions, then defines metrics (or measures) that provide information to answer these questions. Figure 10 shows the hierarchy of the GQM approach, as described by Dr. Basili [Marciniak, 1994]. It is possible, as illustrated in Figure 10, for a metric (or measure) to answer more than one question, and for a question to address one or more goals. Basili explains that the metrics represent a quantitative level; they answer questions in a quantitative way; however, they may be either objective or subjective. For example, number of lines of code in a program is an objective metric since it depends only on the object being measured (the code, in this case). Meanwhile, a user satisfaction rating from "1" to "5" is subjective since the user's viewpoint will affect the rating, even though the rating itself is quantitative, or numerical.

Figure 10: The GQM Hierarchy [Marciniak, 1994]

Figure 11 is an example of use of the GQM approach for quality measures. The goal is to minimize defects in a software product, and the relevant questions are "How likely are defects after release?", and "When are we creating the problems (like, during which stage of the development process)?". For each question, one metric, or measure, has been selected. (These metrics can be considered to be objective.) It is usually helpful to graph the measures as shown in Figure 10 for one of the measures.

Figure 11: Illustration of the GQM Approach [Ferens, 2010]

Van Solingen [Van Solingen, 1999] provides an example of a quality goal that addresses software reliability. His statement of the reliability goal is "Analyze the delivered product and development process for the purpose of characterizing with respect to reliability and its causes from the viewpoint of the software development team in the context of Project "A". Van Solingen then proposes 18 questions and one or more metrics, or measures, for each question. For example, Question 1 is "What is the composition of the product?", and three metrics are a list of life cycle documents, a list of subsystems, and a list of modules.

Capers Jones' Quality Measures

In his book "Applied Software Measurement" [Jones, 2008], Capers Jones recommends quality measures in two categories: user satisfaction measures and defect measures. A desirable goal for companies is, of course, to have high levels of user satisfaction and low levels of defects. Specific values or ratings for these goals can be established using some methods and ideas discussed in Section 1.2.1.

User satisfaction measures include ease of use, functionality, and support. These are "soft" measures since they are based on user perspectives and obtained by surveys, usually taken once a year. If user satisfaction measures are included in quality targets, these measures can be useful, since some definitions of quality include "high degrees of user satisfaction" and "ease of use". One limitation of user satisfaction measures are they are not available until after a program is completed.

Defect measures include defect volumes (e.g., number of defects per function point), defect severities, defect origins, defect causes, and (as discussed in Section 1.2.2.4) defect removal efficiency. According to Capers Jones, defect removal efficiency is a measure used by leading edge companies. The defect measures are "hard" measures that can be assessed objectively and more often (once a month) during a program.

In addition to recommending quality measures to use, Capers Jones explains there are certain quality measures not to use! Cost per defect is one example; it can actually penalize high quality products that have relatively fewer defects! Table 17 shows a comparison of costs per defect for a low-quality "buggy" application and a high quality application of the same size. The cost per defect is actually a combination of three costs: preparation costs, execution costs, and repair costs. Preparation costs tend to be fixed; for example, the cost to create test cases for testing or reading specifications for a design review is the same no matter how many defects the product contains. Execution costs, the costs of actually conducting a review, audit, or test, are "semi-variable"; there are certain costs associated with going through an exercise no matter how many defects are present, while there are costs such as recording defects found that increase with number of defects. Lastly, repair costs are the costs of fixing or removing defects; these costs vary directly with the number of defects that must be corrected. As shown in Table 17, for two similar sized programs, cost per defect looks better for the low quality program; however, the total costs are much lower for the higher quality program.

Table 17: Cost-per-Defect Paradox [Jones, 2008]

Program factor	Low Quality Program	High Quality Program
Size, in Lines of Code	10,000	10,000
Defects Found	500	5
Preparation Costs	$5,000	$5,000
Execution Costs	$25,000	$2,500
Defect repair Costs	$20,000	$200
Total Defect Removal Costs	$50,000	$7,700
Cost Per Defect	$100	$1,540

Although Capers Jones does describe some measures that can and should be used in a program (and should not be used), determining which of these metrics to use for a specific program will depend on such factors as program characteristics, the user's environment, and the organization developing the software. For example, an inexperienced organization may not be able to report defect removal efficiency. Using approaches like PSM or GQM can help in this selection. It is also noted there are other references such as the CMMI [Konrad, 2003] that provide descriptions of quality measures that can be useful. Again, which ones to use depends on the nature of the program and, according to the CMMI, the maturity level of the organization developing the software.

Quality Measures Used by Corporations

Sometimes companies develop "standard" quality measures that they use for their software development efforts. For example, as reported by Fenton and Pfleeger [Fenton, 1997], AT&T developed the following set of quality measures:

- Cumulative fault density - faults found internally
- Cumulative fault density - faults found by customers
- Total "serious" faults found
- Mean time to close serious faults
- Total field fixes
- High level design review errors per thousand lines of code
- Low level design errors per thousand lines of code
- Code inspection errors per inspected thousand lines of code
- Development test and integration errors found per thousand lines of code
- System test problems found per developed thousand lines of code
- First application test errors found per developed thousand lines of code
- Customer found problems per developed thousand lines of code

Kan [Kan, 1995] provides a list of measures that resulted from studying quality efforts for several companies. Kan's list contains 13 measures in three categories, as shown in Table 18.

Table 18: Thirteen Quality Measures [Kan, 1995]

Category	Metrics (Measures)
Product Quality	Mean Time To Failure
	Defect Density
	Customer-Reported Problems
	Customer Satisfaction
In-Process Quality Measures	Defect Removal Effectiveness
	Phase-based Defect Removal Program
	Defect Density During Formal Machine testing
	Defect Arrival Pattern During Formal Machine Testing
Maintenance Quality Measures	Fix Backlog
	Backlog Management Index
	Fix Response Time
	Percent Delinquent Fixes
	Defective Fixes

Lists of measures used by specific companies (or groups of companies) can be useful resources in finding suitable quality measures for a program. However, an approach such as PSM or GQM should be used to determine which measures are suitable for a specific project or program.

Suggestions for measuring defects

This section describes many measures that can be used in assessing whether quality targets, or goals, are being met. The specific measures to be used, however, depend on the quality targets actually set (as explained in Section 1.2.1) and the characteristics of the project or program and the organization developing the software. Two methods that can help in selecting suitable measures are PSM and GQM; however, there are other methods that may be used to supplement these methods, such as the "Factor-Criteria-Metric" method described by Fenton and Pfleeger [Fenton, 1997]. Another useful source is information on past programs from different corporations; however, the measures and methods that should be used will vary by program or project.

1.2.2.6 Reporting Progress to Management

As explained by Fenton and Pfleeger [Fenton, 1997], most customers know about the area for which a software product is being developed, but little about compilers, programming languages, and other aspects of software development. Therefore, measurements must be presented in a way that tells both customers and developers how a project is doing. For any program or project, it is important to report progress of the effort, including quality concerns, to project or program managers, other senior-level managers, and other stakeholders in a program or project. As explained by Schwalbe [Schwalbe, 2006], a stakeholder can be anyone who has an interest in a project, including a competitor. (Of course, the information a company reports to competitors should be minimal!) Schwalbe goes on to explain that different types and levels of reporting will be made to different stakeholders. In fact, Schwalbe recommends developing a stakeholder matrix for a project, which lists key stakeholders and their personalities so reports can be tailored to individuals. Although all stakeholders are important, this section concentrates on reports to management.

In a section of their book "Software Metrics" [Fenton, 1997], Fenton and Pfleeger compare the concerns of managers with those of developers (like analysts and engineers), and the information needed by both. The five main concerns of management, according to Fenton and Pfleeger, are:

What does each process cost?
How productive is the staff?
How good is the code being developed?
Will the user be satisfied with the product?
How can the process be improved?

We have already discussed some measures that are useful for Questions 3 and 4 above; many defect-related measures are useful in assessing code quality, and user satisfaction measures can be used to assess how content users are with products. Some of the measures can also be used to answer the other questions. (More information about process improvement is included in Section 1.3.3.) This section addresses not only what measures should be reported to management, but how they should be reported.

For the CMMI, one of the Process Areas, "Process and Product Quality Assurance", involves providing feedback to managers on the results of quality assurance activities [Konrad, 2003]. Table 19 lists four specific practices within this Process Area, and some results of these practices that should be reported to management. (More about the CMMI is in Section 4.0).

Table 19: Specific Practices for CMMI Process Area "Process and Product Quality Assurance" [Konrad, 2003]

CMMI Specific Practice	Management Reporting
SP 1.1 Objectively Evaluate Processes	Evaluation reports Non-compliance with standards and procedures Lessons learned for future projects
SP 1.2 Objectively Evaluate Work Processes and Services	Evaluation reports Non-compliance with Standards and procedures Lessons learned for future projects
SP 2.1 Communicate and Ensure Resolution of Non-Compliance Issues	Results of evaluations and quality trends Tracking of resolution of non-compliance issues
SP 2.2 Establish Records	Evaluation reports Quality assurance reports Status reports of corrective actions Reports of quality trends

Several other CMMI Process Areas include reporting quality information to managment. The Integrated Project Management Porcess Area includes a Specific Practice of reporting defects to relevant stakeholders. The Organizational Process definition Process Area includes a Specific Practice of establishing a measurement repository to include quality measures such as defect density and number of defects found. The Requirements Development Process Area includes requirements defects as an item to be reported for the Specific Practice of Analyzing Requirements. The Quantitative Process Management Process Area includes assessing process performance, which is characterized by both process measures (e.g., effort, cycle time, and defect removal efficiency) and product measures (e.g., reliability, defect density, and response time).

The CMMI also has a Generic Practice, "Review Status With Higher Management" (GP 2.10) that applies to all Process Areas. This is a "Level 2" Process Area, which means that organizations must be proficient at this area to be recognized at Level 2, which is, in effect, "basic project management". This Generic Practice involves periodically reviewing activities, status, and results of the software development process with higher level management, and resolving issues. This will include quality considerations.

For communications issues in general, including reporting progress to management, The Project Management Body of Knowledge (PMBOK) treats Project Communications Management as one of nine "knowledge areas" for project management [Project Management Institute, 2008]. The PMBOK lists five processes to insure timely and appropriate generation, collection, distribution, storage, retrieval, and ultimate disposition of project information. The five steps are:

Identify stakeholders: identify all people and organizations impacted by the project and document information relevant to their interests. This may involve creating a stakeholder analysis matrix, which can also identify particular characteristics of each stakeholder (e.g., hates detail, prefers e-mail to phone calls, etc.)
Plan communications: Determine information needs and a communication approach. The PMBOK recommends developing a communications management plan for a project. This plan would include reporting defect information to management.
Distribute information: Make relevant information available to stakeholders as planned (in the communications management plan). It involves preparing and distributing performance reports and giving presentations.
Manage stakeholder expectations: Communicate with and work with stakeholders to meet their needs and address issues as they occur. Part of this process is updating an issue log where new issues are identified and current issues are resolved. Defect removal could be part of an issue log.
Report performance: Collect and distribute performance information, including status reports, progress measurements, and forecasts. These reports must be provided at the appropriate level for each audience. For defect tracking, this could range from a monthly summary report to a detailed analysis of defects discovered and removed depending of the interest level of the stakeholder. This is why it is important to perform a stakeholder analysis early in a program.

In her book, "Information Technology Project management" [Schwalbe, 2006], Kathy Schwalbe provides an example of a stakeholder analysis that can be used to determine the level of reporting and best method for distributing reports, and an example of a stakeholder communications analysis to show the preferred method and frequency of communication. Table 20 is an example of a combination of these analyses that can be used in defect reporting.

Table 20: Communication Analysis

Stakeholder	Andrea	Brandon	Cameron	Dylan
Role on Project	Sponsor	Customer	Project Manager	Programmer/Analyst
Characteristics	Likes summary information but weekly, also likes hardcopies, is not a technical person	Likes electronic communication, does not get too involved	Likes electronic communication, has degree in engineering, did not like business courses	Detail person, Likes reading reports, does not like e-mail
Document	Weekly 1-page status report with defect summaries	Monthly summary status report with only product defects reported	Weekly status report with some detailed defect information	Daily or weekly report with details
Communication Method	Hard copy	E-mail attachment	E-Mail attachment and hardcopy	Hardcopy
Frequency	Weekly	Monthly	Weekly	Daily

Suggestions for Reporting Progress to Management

There are many defect measures that can be reported to management, and different ways of reporting this information. What information should be reported, and to what level of detail, will depend on who is receiving the data. Although Fenton and Pfleeger, and the CMMI provide some general guidelines, reporting to management and other stakeholders is highly dependent on the project and the nature of stakeholders. Having a communications management plan early in a project and a stakeholder analysis can help determine these reporting requirements.

1.3 Characteristics of Implementation

This section contains summary and detailed characteristics, and benefits that result from using this Gold Practice.

1.3.1 Summary Characteristics

To effectively implement this Gold Practice, organizations must be capable of both setting quality targets (or identifying quality gates), and handling defect tracking. This includes detecting, analyzing, correcting, and reporting defects. The Software Engineering Institute (SEI) Capability Maturity Model Integration for Development (CMMI-DEV), discussed further in Section 4, includes some specific requirements in three Process Areas: Product and Process Quality Assurance, Quantitative Project Management, and Causal Analysis and Resolution [Konrad, 2003]. Chapter 11 of the Software Engineering Body of Knowledge (SWEBOK) explains prerequisites for an organization to properly address software quality [Abran, 2004].

1.3.2 Detailed Characteristics

Table 21 lists the characteristics, or practice essentials, for an organization to implement this Gold Practice. These are from Version 5.2 of the SPMN Guide 16 Critical Software Practices (SPMN).

Table 21: Defect Tracking Practice Essentials

1. All programs need to have pre-negotiated quality targets, which is an absolute requirement to be met prior to acceptance by the customer.

2. Programs should implement practices to find defects early in the process and as close in time to creation of the defect as possible and should manage this defect rate against the quality target.

3. Metrics need to be collected as a result of the practices used to monitor defects, which will indicate the number of defects, defect leakage, and defect removal efficiency.

4. Quality targets need to be redefined and renegotiated as essential program conditions change or customer requirements are modified.

5. Compliance with quality targets should be reported to customers on a frequent and regular basis, along with an identification of the risk associated with meeting these targets at delivery.

6. Meeting quality targets should be a subject at every major program review.

7. Product quality goals for delivered products should be unambiguously defined at project inception:

Understandability
Modularity
Defect density
May have different goals for different types of applications or different parts oft the same application

8. Source code cyclomatic complexity < 15 for code units should be one understandability goal.

9. Task products should be inspected against task exit criteria and defects fixed or formally deferred before earned value credit is given for the task.

10. From project inception:

Defects should include defects in requirements
Classify defects by level of severity
Unambiguous definitions of severity levels at project inception
Rules about when fixes must be made for each of the severity levels

11. Customer, project managers, and supplier/developer executives achieve a culture that rewards, not penalizes, finding defects when made.

12. Defect metrics should be collected and periodically reported, including:

Number of defects found in each inspection and each test
For each defect:
- When each defect was created
- When each defect was found
The number of inspections in which the defect was present but not found
The number of defects closed and currently open by category of defect: requirements, architecture, detailed design, code, and documentation

13. CM should manage defects:

All verified defect reports turned over to CM
CM maintains status of defect closure
CM performs version control of products in the developmental and operational baselines

14. Baseline of customer-accepted deliverables is made.

15. Developmental baseline for stability of products shared among the development team is made.

16. Treat tools as a subset of CM (addressed in CMM).

17. Automated tools solve project problems more efficiently than manual techniques.

Reproduced from SPMN 16 Critical Software Practices

1.3.3 Anticipated Benefits from Implementation

Defect tracking against quality targets provides benefits to both the project(s) currently being managed and future projects. These benefits are gained at the expense of training costs, installation of appropriate tools, and introduction of processes. These costs can be seen in reports on experiences with Orthogonal Defect Collection (ODC), one method of defect tracking:

“The cost impact on an individual software engineer during the development process is minimal. Typically, measured in the order of a few dozen keystrokes per defect to fill out one or two panels. The incremental time is probably negligible once one enters a tracking system to track the defect… There is an initial setup cost that involves education, tool changes, and process changes to get ODC started” [Bhandari, 1992].

However, the benefits usually outweigh the costs, both for current and future projects. Current projects benefit from a positive return on investment (ROI), as explained in Section 1.3.3.1, while future projects will benefit from defect prevention measures, as explained in Section 1.3.3.2.

1.3.3.1 Return on Investment for Current Projects

In his book, "The ROI from Software Quality" [El Emam, 2005], the author, Khaled El Emam, compares the profits and losses from a company's releasing software with numerous defects. The profit can come from some contractors being paid the same regardless of the reliability of the software, and maybe even being paid to later correct the defects. Also, customers are more likely to purchase support contracts if software tends to fail, while they may not buy them if the software works well. (An analogy may be extended warranties on cars and appliances, which customers are more likely to purchase for less reliable products.) However, the consequences from releasing defective software probably outweigh the benefits. The threat of costly litigation due to failed software products is increasing, and there is increasing concern that software defects can lead to security vulnerabilities. Even if litigation is not a concern, the cost of rework, or fixing defects, can be very high. Also, if a company delivers low quality software, the company's reputation will be tarnished, and future sales will be adversely affected. People will not continue to buy products with many defects. Despite these negative consequences, some organizations continue to develop defective software, and must be shown that the costs outweigh the benefits. This section shows that return on investment (ROI) from defect tracking against quality targets will almost always be positive.

Figure 12, taken from an article by Webb and Patton [Webb, 2008], shows the costs associated with good quality and poor quality. The costs associated with good quality, or costs to achieve quality, are prevention and appraisal costs. The costs of poor quality, or costs due to a lack of quality, are internal failure and external failure costs. Prevention costs cover efforts to ensure quality and defect prevention activities. Appraisal costs address efforts to discover the condition of the software quality. Internal failure costs cover efforts to detect and correct problems prior to the user detecting them, while external failure costs cover the effort to correct problems discovered by users. Table 22, from a paper by Demirors and Yildiz [Demirors, 2000], provides examples of costs in each of the categories.

Figure 12: Cost of Quality [Webb, 2008]

Table 22: Costs of Software Quality Categories [Demirors, 2000]

Prevention	Appraisal	Internal Failure	External Failure
Prototyping	Inspections	Fixing errors	Technical support for defect complaints
Requirements reviews	Testing	Corrective rework	Maintenance and release due to defects
Quality planning	Verification and Validation (V&V) activities	Updating documents due to changes	Penalties
Quality Training	Quality audits	Integration	Products returned due to defects
Reuse libraries	Field performance trials		Service agreement claims
Configuration Management (CM) Administration
QA Administration

External failure costs can be particularly insidious. The obvious costs are those associated with correcting defects, which, as shown previously, can be relatively high after software is deployed. The costs of correcting defects, however, often pale in comparison to other costs. Figure 13 [Ferens, 2009] illustrates the potential costs to a customer that can result from defective software.

Figure 13: Costs of external failures [Ferens, 2009]

Some writers, such as Rex Black, classify prevention and appraisal costs as "costs of conformance", and internal and external failure costs as "costs of nonconformance" [Black, 2000]. Black provides a return on investment (ROI) analysis for use of formal testing . Table 23 shows that the cost of conformance, which includes appraisal costs associated with manual and automated testing, are much less than the costs of nonconformance. This assumes that bugs found during testing cost $10 to fix while bugs found by the customer (external failure costs) are $1,000. While this example is hypothetical, Black cites an actual case where software systems costs were reduced by 50% by using appraisal techniques like formal testing.

Table 23: ROI Analysis of Testing [Polydys, 2009]
roi spreadsheet

While formal testing provides a positive ROI, inspections have been shown to be even more effective. Gack [Gack, 2009] showed an ROI of 4.6 from using inspections and testing compared to using testing alone as an appraisal method. El Emam [El Emam, 2005] showed that inspections are effective in that the effort needed to correct defects using design or code inspections is only 25% of the effort needed to correct defects during testing. The ROI from using inspections instead of testing alone, in terms of percentage savings, is 11% for pre-release costs, 26% for post-release costs, and 38% for customers. The use of inspections results in a 3% schedule savings, which shows that quality improvements from inspections do not add time to a software project. Freimut [Freimut, 2005] described the results of using inspections at Siemens AG, Germany. The cost of rework was a 67% mean decrease measured after the improvements for inspections performed in the analysis phase; the cost of rework was a 47% mean decrease measured after the improvements for inspections performed in the design phase; and the cost of rework was a 12% mean decrease measured after the improvements for inspections performed in the development, or coding phase.

Table 24, from the DACS ROI Dashboard and included in the 2007 DACS report, "A Business Case for Software Process Improvement" [Ferens, 2007] summarizes the results from multiple resources for using software inspections as a software process improvement activity. The average ROI was 6.84, and the minimum was 2.0, which shows that ROI will almost always be positive from using inspections.

Table 24: ROI from Inspections [Ferens, 2007]

Metric	Data Points	Minimum	Maximum	Median	Mean	Standard Deviation	25th Percentile	75th Percentile
ROI	15	2 ratio	30 ratio	4.8 ratio	6.84 ratio	6.62 ratio	4.23 ratio	7 ratio
Productivity	2	37% increase	350% increase	193.5% increase	193.5% increase	221.32% increase	N/A	N/A
Project Cost	1	8% decrease	8% decrease	8% decrease	8% decrease	N/A	N/A	N/A
Improvement Cost	1	3% of total effort	3% of total effort	3% of total effort	3% of total effort	N/A	N/A	N/A
Quality (% defect reduction)	5	63% defect reduction	100% defect reduction	99% defect reduction	86.6% defect reduction	18.12% defect reduction	67% defect reduction	100% defect reduction
Quality (% of defects found)	6	50% defects found	100% defects found	86% defects found	77.5% defects found	22.07% defects found	50% defects found	93% defects found
Rework	4	12% decrease	67% decrease	39% decrease	39.25% decrease	23.39% decrease	21.5% decrease	57% decrease

El Eman [El Emam, 2005] shows how ROI is usually positive from using other defect prevention and appraisal methods. He shows how to compute ROI for various methods, including inspections.

Suggestions for realizing ROI from current projects

The primary benefit from using defect tracking methods such as testing and inspections is the positive return on investment. However, there are some benefits that are hard to quantify, yet will result in a positive return. One area is schedules; there will be less time required for nonconformance activities; hence, less time required for the overall effort. Customer satisfaction will be greater when prevention and appraisal activities are emphasized since products will be of higher quality. Also, there is a positive impact on team morale with higher quality products, which can reduce turnover rates and increase productivity in general. Try to implement processes such as inspections and formal testing which almost certainly will result in a high positive ROI. As shown earlier, these methods are effective at finding defects during the phase in which they occur, which will reduce rework (and frustration).

1.3.3.2 Defect Prevention for Future Projects

Most defect tracking and removal methods discussed so far address detecting and removing defects as soon as possible after they are injected into the current project during requirements, design, etc. Defect prevention, according to Tian [Tian, 2005], is to reduce the chances of defects being injected in the first place. Soni states that "prevention is better than cure" is a saying that applies to defects in the software development cycle as well as illness in medical science (see Defect Prevention: Reducing Costs and Enhancing Quality). Version 1.2 of the CMMI for Development (CMMI-DEV) Process Area of Causal Analysis and Resolution has the purpose of identifying the causes of defects and other problems and take action to prevent them from happening in the future [Konrad, 2003]. It would appear that defect prevention can have even more positive impact than defect appraisal activities like walkthroughs and inspections. According to Schulmeyer and McManus, the defect prevention process has a 15-to-1 payback and avoids 50% of all defects normally injected during the first year and first project in which it is used, and 70% avoidance within 2 or 3 years [Mcmanus, 1999].

The defect tracking and removal activities discussed throughout this document are prerequisite for defect prevention. For example, the Causal Analysis and Resolution Process Area for the CMMI-DEV has the specific goals (SGs) and specific practices (SPs) that are shown in Table 25. The table shows the SGs and SPs associated with this CMMI Process Areaa, and the subpractices associated with each practice:

Table 25: Causal Analysis and Resolution Process Area [Konrad, 2003]

Specific Goal (SG)	Specific Practice (SP)	Subpractices
SG1 Determine Causes of Defects	SP 1.1 Select Data for Analysis	Gather relevant defect or problem data Determine which defects or problems will be analyzed further
	SP 1.2 Analyze Causes	Conduct causal analysis with people responsible for a task Analyze selected defects and other problems to determine their root causes Group the selected defects and other problems based on their root causes Propose and document actions to be taken to prevent future occurrence of these defects and problems
SG2 Address causes of defects	SP 2.1 Implement the action proposals	Analyze action proposals and determine their priorities Select the action proposals to be implemented Create action items for implementing action proposals Identify and remove similar defects that may exist in other processes and work products Identify and document improvement proposals for the organization's set of standard processes
	SP 2.2 Evaluate the effects of changes	Measure the changes in performance of the project's defined process as appropriate Measure the capability of the project's defined process as appropriate
	SP2.3 Record Data	(No subpractices)

In Table 25, the first subpractice listed, "Gather relevant defect or problem data", implies that an organization is already performing the defect tracking activities discussed previously. Also, SP 2.2, "Evaluate the effects of changes", implies that an organization will track defects in future projects to see if the defect prevention measures proposed are indeed effective.

In the CMMI-DEV, Causal Analysis and Resolution is a "Maturity Level 5" Process Area. Level 5 is the highest maturity level; it implies an organization is not only actively collecting quantitative data (such as defect data), but is using this data to continuously improve their processes. Defect prevention, then, requires not only a high level of capability in other CMMI Process Areas, but also requires a high degree of commitment from an organization to perform defect prevention activities. Table 25 shows that defect prevention can be very involved; it requires a lot of time and effort. However, the rewards can be worth it.

Version 1.2 of the CMMI for Acquisition (CMMI-ACQ) also has Causal Analysis and Resolution as a Maturity Level 5 Process Area for acquisition agencies. According to the CMMI-ACQ, acquirers use Causal Analysis and Resolution practices to guide identification of root causes of selected defects and other problems and taking action to prevent their reoccurrence [CMMI PRODUCT TEAM, 2007]. The same SGs and SPs apply, although the acquirer will focus on defects reported by customers, end users, and suppliers. In either case, causal analysis is best performed by people who have the best understanding of the selected defect or problem being studied [Hofmann, 2007].

If an organization desires to perform defect prevention activities, they must be ready. In his "Quality Questionnaire", El Eman asks the following questions for defect prevention [El Emam, 2005]:

Are defect activities planned?
Does the project conduct causal analysis meetings to identify common causes of defects?
Once identified, are common causes of defects prioritized and systematically eliminated?

Question 3 shows that defect prevention goes far beyond defect removal. Defect removal involves removing current defects, while defect prevention removes the very causes of these defects. In Figure 14, Soni illustrates the activities associated with defect prevention and compares them with activities associated with defect tracking and removal. The area in gray represents those areas associated with "the current philosophy" of handling defects: detection, tracking/documenting and analysis of defects to arrive with quick, short-term solutions" (from Defect Prevention: Reducing Costs and Enhancing Quality).

Figure 14: A Defect Prevention Cycle (from Defect Prevention: Reducing Costs and Enhancing Quality)

When an organization determines it is ready for a defect prevention program, the organization needs to plan and document its program. There are several resources that explain how to set up a program and that provide examples. For instance, Figure 14 illustrates some steps in implementing a defect prevention program. Kan also lists three steps for a defect prevention program: Analyzing existing defects or errors to trace root causes; suggesting preventive actions to eliminate the defect root causes; and implementing the preventive actions [Kan, 1995]. This process consists of four elements:

Conduct causal analysis meetings, which are 2-hour brainstorming sessions at the end of each development phase (in the current project). Suggestions are recorded in an action database for subsequent reporting and tracking.
Form an action team to screen, prioritize, and implement suggested actions from causal analysis meetings. The action team is the "engine" of the process.
Stage kickoff meetings at the beginning of each development phase to determine an effective process, and tools and methods, to prevent defects during that phase. The meetings serve both as a primary feedback mechanism for the defect prevention process, and as a preventative measure.
Track actions and collect data to prevent suggestions from being lost over time, aid action implementation, and enhance communication among groups [Kan, 1995].

The Software Productivity Consortium has issued a Defect Prevention Guidebook which addresses a recommended defect prevention process that can be implemented on a single project or across the organization, the organizational and project roles and responsibilities, and a set of defect reduction strategies. The Guidebook also provides information for process improvement change agents and project managers for classifying defects from inspections to provide causal analysis, performing causal analysis, identifying defect prevention elements with the highest ROI for defect prevention investment, and planning and implementing preventive actions [Brown, 1998].

The Defect Prevention Guidebook explains all process elements in detail, devoting an entire chapter to each key area. An example is a chapter on performing causal analysis, where the steps are: forming a causal analysis team; holding causal analysis meetings (like Kan's first step); and implementing defect prevention strategies. Holding meetings is further divided into six steps: selecting problems; classifying problems; identifying systematic errors; determining principal causes; developing corrective action proposals; and documenting meeting results. An example of a tool that can be used in identifying systematic errors (the same defect being repeated on different occasions) is a Pareto Chart for coding defects, which is shown in Figure 15. The Pareto chart shows the number of defects by cause in descending order, with the leftmost bars showing the most common causes, which subsequently should probably receive the most attention. The Guide contains an appendix that describes several useful tools, including more information about Pareto charts.

Figure 15: Pareto Chart Used In Causal Analysis [Brown, 1998]

While the Guidebook and other resources give useful information about setting up a defect prevention program, the project or organizational manager or team needs to determine the best processes to use in setting up a program for a specific project or organization. Some resources do provide guidance in this area; for example, Capers Jones provides some useful guidance on which software development methods are most useful for which categories of defects, or errors. One or more of these development methods can then be used in a defect prevention program. According to Capers Jones, "defect prevention" means "the overall set of technologies that simplify complexity and minimize a natural human tendency to make errors while performing complex tasks" [Jones, 2008]. Table 26 shows some development tools and methods, such as Joint Application Design (JAD) and Quality Function Deployment (QFD), with their effectiveness for different categories of defects.

Table 26: Defect Prevention Methods [Jones, 2008]

Suggestions for Defect Prevention

Defect prevention can greatly enhance quality and ROI for projects beyond what defect appraisal methods, including tracking and removal methods, alone can do. However, defect prevention can only be undertaken for projects and organizations that are already effective at defect appraisal activities, including defect tracking and removal activities discussed thus far. The CMMI-DEV Process Area of Causal Analysis and Resolution summarizes the goals, practices, and methods of associated with defect prevention. There are several resources, including the Defect Prevention Guidebook [Brown, 1998], that can help an organization or project manager establish a defect prevention program, and other resources, such as Capers Jones' book, "Applied Software Measurement" [Jones, 2008] can help determine specific strategies or ideas for a particular project or organization.

2.0 Relationships to Other Practices

A Best Practice is rarely successful as a stand-alone process. The DACS Gold Practice concept extends beyond typical Best Practice implementation by emphasizing the integration of other implicit or explicit practice/process concepts that may influence, or be influenced by, the Gold Practice.

Figure 16 is a high-level process architecture for the practice of Defect Tracking Against Quality Targets, depicting relationships to this practice and the nature of the influences on the practice. Defect Tracking Against Quality Targets uses as inputs information from Gold Practices associated with setting quality gates and tracking defects, while the outputs of Defect Tracking Against Quality Targets are information for other Gold Practices which are associated with analyzing data and identifying defect causes and corrective action, implementing and verifying corrective action, and assessing and reporting quality improvements. Thus, Figure 16 shows feedback. These relationships are based on definitions of specific �best practices� found in the literature and the notion that the successful implementation of practices may influence (or be influenced by) the ability to successfully implement other practices. Section 2.1 discusses the inputs to this Gold Practice, while Section 2.2 discusses outputs.

Figure 16: Gold Practice "Defect Tracking Against Quality Targets" Relationship to other Gold Practices

2.1 Inputs

This section describes the Gold Practices that are inputs to the Gold Practice, "Defect Tracking Against Quality Targets". The two primary inputs are the identification of quality gates and methods for detect detection. While information in both areas is included in this Gold Practice, the other Gold Practices shown as "inputs" in Figure 16 may be consulted for further insight into these input areas. Table 27 describes the inputs from other Gold Practices, and how those Gold Practices contribute to these inputs.

Table 27: Inputs to Gold Practice, "Defect Tracking against Quality Targets"

Inputs

Gold Practices Involved

Identify Quality Gates (or Targets)

The Gold Practice of "Binary Quality Gates at the Inch-Pebble Level" involves defining completion criteria for each task in the lowest-level activity network as gates that assess either the quality of the products produced, or the adequacy and completeness of the finished process, and then tracking completion at the binary level. It helps in setting detailed quality targets, or goals, for tracking and measuring.

The Gold Practice "Requirements Trade-Offs/Negotiations" has the purpose of reaching stakeholder agreement on what the real requirements are for a designated phase or release of a software product given the reality of cost, schedule, and perhaps technology constraints. It helps identify clear requirements for quality. Requirements defects can cause problems if not addressed early in a program, and well-defined requirements are less defect-prone.

The Gold Practice "Goal-Question-Metric Approach" describes a paradigm for developing and maintaining a meaningful metrics program that supports metric alignment with organization business and technical goals, software process improvement, risk management, and product quality improvement. It can help in determining quality targets by identifying realistic quality goals for the program. It is also useful in determining what measures to use for tracking defects.

The Gold Practice "Establish Clear Goals and Decision Points" involves establishing clear goals and decision points for a program up front. Decision points are then used to review progress and achievement and to determine the future direction of a program. This can be used in conjunction with the Goal-Question-Metric Approach Gold Practice in setting realistic quality goals. It can also be useful in establishing milestones, or decision points, for quality, such as when to release a software product.

Detect Defects

The Gold Practice "Compile and Smoke Test Frequently" involves performing frequent (e.g., twice a week) software compile-builds(as soon as code is available) and performing regression testing as a means for finding software integration problems early. Results are distributed to the entire project team. This Gold Practice helps in detecting defects as early as possible during software testing.

The Gold Practice "Independent Expert Reviews" involves having an independent team assess if programs are adequately addressing issues of cost, schedule, technology, risk and process. These expert reviews can be useful in assessing whether quality goals are being met and in finding defects.

The Gold Practice "Formal Inspections" describes in depth the practice of conducting formal inspections on requirements, architecture, designs at all levels (particularly detailed design), on code prior to unit test, and on test plans, to assess the quality of all baselined artifacts prior to release for project use. As shown in the current Gold Practice, inspections are a very effective way to find defects.

The Gold Practice "Model Based Testing" describes the Model Based Testing process, which begins with requirements. A model for user behavior is built from requirements for the system. Those building the model need to develop an understanding of the system under test and of the characteristics of the users, the inputs and output of each user, the conditions under which an input can be applied, etc. The model is used to generate test cases, typically in an automated fashion. The specification of test cases should include expected outputs. The model can generate some information on outputs, such as the expected state of the system. Other information on expected outputs may come from somewhere else, such as a test oracle. The system is run against the generated tests and the outputs are compared with the expected outputs. Here, too, automation is extremely useful. The failures are used to identify bugs in the system. The test data is also used to make decisions, for example, on whether testing should be terminated and the system released.

2.2 Outputs

This section describes the Gold Practices that use outputs from the Gold Practice, "Defect Tracking Against Quality Targets". The three primary outputs are analyzing data and identifying defect causes and corrective action, implementing and verifying corrective action, and assessing and reporting quality improvements. While information in these three areas is included in this Gold Practice, the other Gold Practices shown as "outputs" in Figure 16 may be consulted for further insight into these output areas. Table 28 describes the outputs to other Gold Practices, and how those Gold Practices use these outputs.

Table 28: Outputs From Gold Practice "Defect tracking Against Quality Targets"

Outputs	Gold Practices Affected
Analyze data and identify defect causes and corrective action	The Gold Practice "Statistical Process Control" involves using this process for software development. Defect tracking data from one or several programs can be used in setting statistical process control upper and lower limits for future programs.
Implement and verify corrective action	The Gold Practice "Manage Requirements" involves the process of eliciting, documenting, organizing, and tracking changing requirements and communicating this information across the project team to ensure that iterative and unanticipated changes are maintained throughout the project lifecycle. Corrective actions for defects may involve changing requirements on current or future projects. The Gold Practice "Formal Risk Management" involves using a proactive approach for minimizing the uncertainty and potential loss associated with a project. A risk is an event or condition that, if it occurs, has a positive or negative effect on a project"s objectives. Corrective actions will probably reduce risk for a project or program. The Gold Practice "People-Aware Management Accountability" involves a management practice of committing to staffing qualified people and fostering an environment conducive to high morale and low voluntary staff turnover. Corrective action results in better products, which tend to instill pride in an organization. Also, the commitment of an organization to "do things right" will have a positive effect on the people. The Gold Practice "Configuration Management" involves using an integrated process for identifying, documenting, monitoring, evaluating, controlling, and approving all changes made during the life-cycle of the program for information that is shared by more than one individual or organization. Corrective actions may affect the configuration of a software program or project, and may require formal approval.
Assess and report quality improvements	The Gold Practice "Quantitative Progress Management" involves establishing and using quantitative (rather than qualitative) measurement to assess progress on software projects. Defect tracking does result in quantitative data. The Gold Practice "Metrics-Based Scheduling and Management" is about establishing realistic software development or maintenance schedules based on accurate estimates of software size and effort. The practice necessitates use of a minimum set of four metrics: Software size, effort, time/schedule and quality. Quality metrics, or measures, can be obtained from defect tracking and quality improvements. The Gold Practice "Track Earned Value" discusses earned value management, an integrated system of project management and control that enables a contractor and its customer to monitor the progress of a project in terms of integrated cost, schedule, and technical performance measures. Quality measures and improvements can be used to assess earned value, or the value of work accomplished. The Gold Practice "Program-Wide Visibility of Progress vs. Plan" involves making the core indicators of project health or dysfunction readily available to all project participants and encouraging anonymous channel feedback to enable unfavorable news to move freely up and down the project hierarchy. Reporting quality information can help in this visibility.

3.0 Definitions

Defect

Both Capers Jones and the SPMN define a defect as �a problem or 'bug' that, if not removed, could cause a program to either produce erroneous results or otherwise fail� [Jones, 2008] [SPMN, 2000]. According to Pressman [Pressman, 2005], the terms defect, bug, error, and fault are usually synonymous; however, Pressman himself uses the term �error� to describe quality problems found before a product is released and �defect� to describe problems after release. Schulmeyer and McManus also differentiate between errors and defects in that errors are deviations from requirements that arise during a process while defects are specific kinds of unwanted conditions delivered to the customer [Mcmanus, 1999].

In IEEE Std 1044-1993(R2002), [IEEE STANDARDS BOARD, 2002], the term anomaly is preferred to the term defect because an anomaly may not represent a problem, but a condition which can be enhanced. (For the purposes of this Gold Practice, the SPMN / Jones definition will be used.).

According to the IEEE Standard Glossary of Software Engineering Terminology [IEEE STANDARDS BOARD, 1990], a fault is an incorrect step, process, or data definition in a computer program. Although this term is often used synonymously with errors or bugs, it is the term used in the fault tolerance disciplines. Also, IEEE gives other definitions for the term �error�. Fault tolerance is the ability of a system or component to continue normal operation despite the presence of faults. Another view of fault tolerance is that it prevents a fault, or error, from resulting in a failure, which IEEE defines as �the inability of a system or component to perform its required functions within specified performance requirements� [IEEE STANDARDS BOARD, 1990]. However, as discussed in Chapter 11 of the SWEBOK, different cultures and different standards may use several different meanings to define these terms [Abran, 2004].

Defect Tracking

One definition of defect tracking by Tian is monitoring and recording what happens to each defect after its initial discovery until its final resolution [Tian, 2005]. Here, defect tracking is listed as one activity for defect handling; the others are defect resolution and defect logging.

Quality Targets

Version 1.2 of the CMMI for Development, as explained in [Chrissis, 2007], defines quality and process-performance objectives as �Objectives and requirements for product quality, service quality, and process performance. Process-performance objectives include quality; however, to emphasize the importance of quality in the CMMI Product Suite, the phrase quality and process-performance objectives is used rather than just process-performance objectives�. Tian [Tian, 2005] describes the process of setting quality goals as part of a quality planning process. Part of a quality assessment and improvement process is determining whether the goals are satisfied.

4.0 Sources

The following paragraphs summarize the chronology/history of defect tracking against quality targets and highlight a few key elements.

Any organization that staffs a call center to which users can report problems will track problems, so defects have been tracked to some extent for decades. Processes that use tracked defects to quantitatively assess the quality of software processes and products are of comparatively more recent origin.

Developments in processes for tracking software defects parallel developments in hardware processes, often with lags. Hardware engineers have developed a variety of reliability models, methods for allocating system reliability to components, procedures for accelerated life testing, and techniques to conduct formal statistical tests of reliability [Mann, 1974]. Researchers in software engineering modified this technology, developing families of software reliability growth models and techniques for using those models to specify and measure reliability [Lyu, 1996].

Hardware systems are often developed and supported with a Failure Reporting Analysis and Corrective Action System (FRACAS). An effective FRACAS system captures:

Failure reporting information supporting an historical trend database
The steps in a failure analysis, and the results obtained, for determining the root cause of a failure
The documented corrective action that, once implemented and verified, minimizes the reoccurrence of the failure [Nicholls, 2001]

Software organizations also track defects. The IEEE has developed a standard [IEEE STANDARDS BOARD, 2002] and a guide [IEEE STANDARDS BOARD, 1996] for classifying anomalies. The guide states that immature organizations typically only �record the facts and impressions about an anomaly when it is encountered so that it can be understood�at a later date�. The information described in the standard �will facilitate the discovery of common errors made, most effective activities to find anomalies, the place where anomalies were first introduced and other telling facts about your development process.� Analysis of the data provided through use of the standard �can be used for project management, process improvement, and product assessment� [IEEE STANDARDS BOARD, 1996].

Other techniques have been developed for using software defect data for more than qualitative causal analysis. Orthogonal Defect Classification (ODC) is one such technique. ODC is a systematic framework enabling in-process feedback to software developers, the evaluation of software processes, and other measurement and analysis techniques [Bhandari, 1992].

4.1 Recommending Sources

This section identifies segments from specific DoD, industry, or academic sources that recommend, specify or otherwise support adoption of the subject practice either directly, or as part of a broader best practice initiative, or a practice with a different name.

1. Capability Maturity Model Integration for Development (CMMI-DEV)

Elements of defect tracking against quality targets are spread across several Key Process Areas in the CMMI-DEV [Paulk, 1994].

Table 29: Defect Tracking in the CMMI-DEV

Process Maturity Level	Key Process Area	Specific Goals (SGs) and Specific Practices (SPs)
1. Managed	Process and Product Quality Assurance	SG 1 Objectively evaluate process and work products SP 1.1 Objectively evaluate process SP 1.2 Objectively evaluate work products and services SG 2 Provide objective insight SP 2.1 Communicate and ensure resolution of noncompliance issues SP 2.2 Establish records
	Configuration Management	SG 2 Track and control changes SP 2.1 Track change requests SP 2.2 Control configuration items
	Measurement and Analysis	SG 1 Align measurement and analysis activities SP 1.1 Establish measurement objectives SP 1.2 Specify measures SP 1.3 Specify data collection and storage procedures SP 1.4 Specify analysis procedures SG 2 Provide measurement results SP 2.1 Collect measurement data SP 2.2 Analyze measurement data SP 2.3 Store data and results SP 2.4 Communicate results
2. Defined	Verification	SG 2 Perform peer reviews SP 2.1 Prepare for peer reviews SP 2.2 Conduct peer reviews SP 2.3 Analyze peer review data
	Integrated Project Management + IPPD	SG 1 Use the project's defined process SP 1.1 Establish the project's defined process SP 1.2 Use organizational process assets for planning project activities SG 2 Coordinate and collaborate with relevant stakeholders SP 2.2 Manage dependencies SP 2.3 Resolve coordination issues
	Requirements Development	SG 3 Analyze and validate requirements SP 3.1 Establish Operational Concepts and Scenarios SP 3.2 Establish a Definition of Required Functionality SP 3.3 Analyze Requirements SP 3.4 Analyze Requirements to Achieve Balance SP 3.5 Validate Requirements
3. Quantitatively Managed	Quantitative Project Management	SG1 Quantitatively manage the project SP 1.1 Establish project objectives SP 1.2 Compose the defined process SP 1.3 Select the subprocesses that will be statistically managed SP 1.4 Manage project performance SG2 Statistically manage subprocess performance SP 2.1 Select measures and analytic techniques SP 2.2 Apply statistical methods to understand variation SP 2.3 Monitor performance of the selected subprocesses SP 2.4 Record statistical management data
3. Quantitatively Managed	Organizational Process Performance	SG1 Establish performance baselines and models SP 1.1 Select processes SP 1.2 Establish process-performance measures SP 1.3 Establish quality and process-performance objectives SP 1.4 Establish process-performance baselines SP 1.5 Establish process-performance models
4. Optimizing	Causal Analysis and Resolution	SG 1 Determine causes of defects SP 1.1 Select defect data for analysis SP 1.2 Analyze causes SG 2 Analyze causes of defects SP 2.1 Implement the action proposals SP 2.2 Evaluate the effect of changes SP 2.3 Record data
4. Optimizing	Organizational Innovation and Development	SG 1 Select improvements SP 1.1 Collect and analyze improvement proposals SP 1.2 Identify and analyze innovations SP 1.3 Pilot improvements SP 1.4 Select improvements for deployment SG 2 Deploy improvements SP 2.1 Plan the deployment SP 2.2 Manage the deployment SP 2.3 Measure improvement effects

2. Turner, R., �Implementation of Best Practices in US Department of Defense Software-Intensive Systems�, Dissertation, George Washington University, January, 2002 [Turner, 2002]

Dr. Richard Turner, under the sponsorship of the Software-Intensive Systems Office of the Deputy Under Secretary of Defense (at the time, Science and Technology), surveyed 14 military software centers. Seven responded, with data on 150 software acquisition programs (53% Air Force, 26% Army, 21% Navy). Turner found that 68% of these programs had partially implemented defect tracking against quality targets, while 31% had full implementations. The table shows the perceived effectiveness of defect tracking against quality targets, according to these survey respondents.

Table 30: Turner Survey Results

Perceived Effectiveness of Defect Tracking Against Quality Targets
Highly Effective	23%
Very Effective	38%
Moderately Effective	38%
Somewhat Effective	0%
Not Effective	0%

Turner conducted a second survey of experts from academia, industry, and government. The 35 respondents to this survey ranked defect tracking against quality targets thirteenth of 32 in terms of perceived effectiveness.

3. Cleanroom Software Engineering

Cleanroom processes have common measurement elements. These measurement elements include defect tracking:

-Measure the Process

Measure the effectiveness of a review in terms of the percentage of all defects originating prior to the review that are found in the review. These percentages are determined, of course, after execution testing.

-Measure the Product

Measure the quality of work products that define the software in terms of the percentages of execution failures that are traced to defects in the work products. These percentages are determined, of course, after execution testing� [Linger, 1999].

4. Nine Best Practices: The Software Management Framework, 3 June 1999

The Navy�s Software Program Manager�s Network (SPMN) convened a group of experts in 1995 at the Airlie Center outside Warrenton, Virginia (the Airlie Council). The Airlie Council originally identified nine best practices, of which defect tracking against quality targets was one. The council provided a rationale for defect tracking against quality:

"In many cases there is no defect tracking mechanism at the project level. Keeping bug reports is not sufficient, the metrics of who, what, where, when, how, and why are as important as the existence of a simple bug list. By tracking these metrics, basic indicators relating to defects, schedule, cost, requirements, documentation and staff can be made visible."

Appendices

Testing 123

Nobody

Case Studies

Many papers report analyses of software defects from one project or another. Many of these papers, however, do not give quantitative data on the benefits derived from adopting defect tracking against quality targets. Some (for example, [Bassin, 2001], [Mohapatra, 2001], and [Paul, 2000]) report case studies that applied this practice, but provide qualitative results or have another direction. The table summarizes two papers which do contain quantitative statements.

Name	Description
Bellcore	Description: Four projects, for which between 30 and 150 defects were analyzed. Benefits: "The recommended prevention measures, if implemented would reduce the field defects by 50% to 60%. The recommended detection measures would allow 40% to 80% of the field defects to be found earlier. If all the processes changes (prevention and detection measures) are implemented, the number of field defects would decrease by 75% to 90%." [Casey, 1991]
IBM	Description: Orthogonal Defect Classification (ODC) "has been deployed at a dozen IBM locations, worldwide, in over 50 projects" Benefits: "When applied to reduce the cost of classical root-cause analysis, [ODC] achieves cost reductions by a factor of 10." Ram Chillarege's chapter in [Lyu, 1996]

Tools

A plethora of automated tools, both proprietary and open source, are available for tracking problem reports. Many Configuration Management (CM) tools have integrated problem reporting systems. Some CM tools, such as CVS, do not have bug tracking capabilities but can be integrated with some existing problem reporting tools. The European Computer Manufacturers Association (ECMA) reference model [Earl, 1990] provides one way of thinking about tool capabilities.

Desirable Capabilities for Problem Tracking Tools

Tracks problems reported by customers over telephone lines, e-mail, Web forms, etc.
Receives input from geographically dispersed locations
Allows assignment of report to responsible person
Monitors and reports status of problem reports
Provides development process management and change management authorization
Implements defined and configurable workflow
Includes statistics and metrics gathering and reporting
Archives historical records
Integrates with configuration management and test tools
Provides a Graphical User Interface front end, such as forms-based Web pages

Tool Name	Description and URL
Bugzilla	Current project leader: Dave Miller Inputs: Problem Reports Outputs: Problem Report Database, Reports on bugs, Integration with CVS http://www.bugzilla.org/
COQUALMO	COnstructive QUALity MOdel(COQUALMO), formerly called CODEFMO, is an estimation model that can be used for predicting the number of residual defects/KSLOC (Thousands of Source Lines of Code) or defects/FP (Function Point) in a software product. http://csse.usc.edu/csse/research/COQUALMO/
GNATS	Current GNATS maintainer: Andrew Gray Inputs: Problem Reports Outputs: Problem Report Database, Reports on bugs, Integration with CVS http://www.gnu.org/software/gnats/
JitterBug	Current maintainer: Andrew Tridgell Inputs: Problem Reports Outputs: Reports on bugs, Tracking of replies, Audit trail http://samba.anu.edu.au/cgi-bin/jitterbug/
Mantis	In Beta. Author: Kenzaburo Ito Inputs: Problem Reports Outputs: Reports on bugs http://mantisbt.sourceforge.net/
Merant Tracker	Merant Inputs: Change Requests, Bug Reports Outputs: Problem Report Database, Reports on bugs http://www.merant.com/Products/ECM/tracker/home.asp
PhpBugTracker	Developer: Benjamin Curtis Inputs: Problem Reports Outputs: Problem Report Database, Reports on bugs, Integration with CVS http://phpbt.sourceforge.net/
The NASA Goddard ARM Tool	The Automated Requirement Measurement (ARM) Tool was developed by the Software Assurance Technology Center (SATC) at the NASA Goddard Space Flight Center as an early life cycle tool for assessing requirements that are specified in natural language. The objective of the ARM tool is to provide measures that can be used by project managers to assess the quality of a requirements specification document. The ARM tool scans a requirements specification document for key words and phrases and generates a report file summarizing the specific quality indicators. The tool is not intended to evaluate the correctness of the specified requirements. It is an aid to "writing the requirements right," not "writing the right requirements." http://sw-assurance.gsfc.nasa.gov/disciplines/quality/index.php

Websites

Call Center, Bug Tracking and Project Management Tools for Linux
Linas Vepstas' page about call center, bug tracking, and project management tools for Linux.
http://linas.org/linux/pm.html

Goddard Space Flight Center - Software Quality Website
Contains software quality information in the areas of procedures and guidelines, work instructions, checklists, and tools
http://sw-assurance.gsfc.nasa.gov/disciplines/quality/index.php

Problem Management Tools Summary
Part 3 of Dave Eaton's "Frequently Asked Questions (FAQ)" for comp.software.config-mgmt.
http://www.daveeaton.com/scm/PMTools.html

Software Quality Research Library
The Software Quality Research Library is a powerful resource available to IT and business professionals who have a role in software quality. Search and browse through a vast database of vendor and analyst white papers, case studies, webcasts, and product information to help you expand your knowledge on a variety of software quality-related topics
http://searchsoftwarequality.bitpipe.com/?asrc=SS_NAV_WP

SPMN 16 Critical Software Practices
The entire guide, 16 Critical Software Practices for Performance-Based Management, Version 5.2. This paper outlines the 16 Critical Software Practices that serve as the basis for implementing effective performance-based management of software-intensive projects. They are intended to be used by programs desiring to implement effective high-leverage practices to improve their bottom-line measures -- time to fielding, quality, cost, predictability, and customer satisfaction -- and are for CIOs, PMs, sponsoring agencies, software project managers, and others involved in software engineering.
http://www.spmn.com/16cspv5-2.doc

SPMN Airlie Council Checklists
A copy of the SPMN checklists from the Little Yellow Book of Software Management Questions (July, 1995) is available on R.S. Pressman's website. This is the result of work conducted by the Airlie Council and the SPMN.
http://www.rspa.com/checklists/airlie.html

USC Center for Software Engineering
Thirty-six Affiliates help keep USC CSE in touch with technology, applications, and economic trends through participation in their annual workshops and collaborative projects, covering such areas as value-based spiral model extensions, COCOMO cost-schedule-quality estimation model extensions, UML-xADL architecture specification extensions, COTS-based system development, and agile methods.
http://sunset.usc.edu/cse/

Expert(s)

Name	Description
Basili, Victor	A professor at the University of Maryland, Dr. Basili is a noted advocate and practitioner of quantitative methods in software engineering. He was a co-founder of NASA's Software Engineering Laboratory and has won numerous awards. His Web page provides pointers to the journals and organizations with which he is associated, as well as papers and other descriptions of his research interests. http://www.cs.umd.edu/users/basili/
Fenton, Norman	Norman Fenton is Professor of Computing at Queen Mary and Westfield College (London University) and is the Head of RADAR (Risk Assessment and Decision Analysis) Group. He is also Managing Director of Agena, a company that specializes in risk management for critical systems. He has been project manager and principal researcher in many major collaborative projects in the areas of: software metrics; formal methods; empirical software engineering; software standards, and safety critical systems. His recent research projects, however, have focused on the use of Bayesian Belief nets (BBNs) and Multi-Criteria Decision Aid for risk assessment. http://www.dcs.qmul.ac.uk/~norman/
Jones, Capers	Capers Jones is Chief Scientist Emeritus of Software Productivity Research (SPR). Capers Jones founded the company. He has almost 40 years of experience in software cost estimating. Jones designed IBM's first automated estimation tool in 1975, and is also one of the designers of three commercial software estimation tools: SPQR/20, Checkpoint, and KnowledgePlan. These software estimation tools pioneered the use of function point metrics for sizing and estimating. They also pioneered sizing of paper documents, and the estimation of quality and defect levels. To build these tools, SPR has collected quantified data from more than 600 companies. http://www.spr.com/
Kan, Stephen H.	Stephen H. Kan is Senior Technical Staff Member (STSM) and a technical manager in programming for IBM in Rochester, Minnesota. As process manager of the quality management process in product development for IBM's eServer iSeries software development, his responsibilities include quality goal setting, supplier quality requirements, quality plans, in-process metrics, field quality status, and quality and project assessments. Dr. Kan has been a faculty member of the Master of Science in Software Engineering program at the University of Minnesota since 1998. He is certified by the American Society for Quality as a Quality Engineer, a Reliability Engineer, and a Quality Manager, and by the Project Management Institute as a Project Management Professional. http://www.msse.umn.edu/directory/Stephen-Kan-PhD
Pressman, Roger S.	Roger S. Pressman is an internationally recognized consultant and author in software engineering. For over 30 years, he has worked as a software engineer, a manager, a professor, an author, and a consultant, focusing on software engineering issues. Dr. Pressman specializes in helping companies establish effective software engineering practices. He is the developer of Process Advisor, the industry's first self-directed software process improvement product and Essential Software Engineering, the industry's most comprehensive video curriculum in software engineering. His new Internet-based software engineering training curriculum, eSchool, developed in partnership with QAI, provides a unique on-line learning experience. http://www.rspa.com/
Wiegers, Karl	Karl E. Wiegers is Principal Consultant with Process Impact in Portland, Oregon. He has provided training and consulting services worldwide on many aspects of software development, management and process improvement. Prior to starting Process Impact in 1997, Karl spent 18 years at Eastman Kodak Company. His responsibilities there included experience as a photographic research scientist, software applications developer, software manager, and software process and quality improvement leader. Karl has led process improvement activities in small application development groups, Kodak's Internet development group, and a division of 500 software engineers developing embedded and host-based digital imaging software products. Karl is the author of the following books: Practical Project Initiation: A Handbook with Tools (Microsoft Press, 2007) More About Software Requirements: Thorny Issues and Practical Advice (Microsoft Press, 2006) Software Requirements, 2nd Edition (Microsoft Press, 2003) Peer Reviews in Software: A Practical Guide (Addison-Wesley, 2002) Creating a Software Engineering Culture (Dorset House Publishing, 1996) Karl has also written more than 170 articles on many aspects of software, chemistry, and military history. He has served on the Editorial Board of IEEE Software and as a Contributing Editor for Software Development magazine. http://www.processimpact.com/bio.shtml
Boehm, Barry	Currently the TRW Professor of Software Engineering, Computer Science Department Director, USC Center for Software Engineering. His current research interests include software process modeling, software requirements engineering, software architectures, software metrics and cost models, software engineering environments, and knowledge-based software engineering. His contributions to the field include the Constructive Cost Model (COCOMO), the Spiral Model of the software process, the Theory W (win-win) approach to software management and requirements determination. http://sunset.usc.edu/Research_Group/barry.html

Training Opportunities

The fact that a course is listed (or not listed) here does not constitute an evaluation or endorsement of the course by the DACS. These are simply links to courses currently available relative to the subject practice.

Offeror	Course Description and URL
American Society for Quality	From the web site: "ASQ's training courses put you and your organization on the fast track to success. Our instructors are industry experts � they average 23 years of experience." http://www.asq.org/training/
EdistaLearning	EdistaLearning, QAI's initiative, is the elearning suite of Software Engineering and Quality Management courses. "The QAI & Dr. Pressman Series" in the Edista Learning portfolio has courses that draw upon Dr. Pressman's highly successful and the industry's most comprehensive video curriculum in software engineering�Essential Software Engineering. Course categories include, but are not limited to: Software Engineering Process Approach Software Project Management Software Quality Management Software Configuration Management Software Testing Software Estimation CMMI IT Services Management Edista also offers/supports certification exams in most of these areas. http://www.edistalearning.com
Fagan Defect-Free Process Course Course	This course is offered by Michael Fagan Associates. From the web site: "The FAGAN DEFECT-FREE PROCESS will be described. It is composed of three interdependent components: Formal Process Definition, Fagan Inspection Process, and Continuous Process Improvement (Removal of Systemic defects from the work process)." Case studies are included to help the student understand the process. http://www.mfagan.com/outline_frame.html
IEEE Reliability Society Tutorial Videos	In the past decade the IEEE Reliability Society has produced numerous tutorials in a number of areas of interest to RS members. These tutorials are available in VHS video (PAL/NTSC) and/or CD ROM and come with a printed workbook or documentation integrated with the CD. The videos are co-produced by the IEEE Educational Activities Board and the IEEE Reliability Society. The videos are professionally done, utilizing experts in their fields. For a fraction of the cost of sending a single person to training, you can own a video or CD ROM which will provide state-of-the-art information to your entire workforce. http://www.ieee.org/portal/site/relsoc/menuitem.e3d19081e6eb2578fb2275875bac26c8/index.jsp?&pName=relsoc_level1&path=relsoc/Training&file=index.xml&xsl=generic.xsl
QAI Course: Using Measurement for Improving Software Quality	From the Quality Assurance Institute (QAI) Web Site: "Economic pressures, competition, and a better informed consumer are forcing companies to develop and deploy better quality products more quickly and inexpensively. A measurement program is an effective method to identify areas of improvement, to increase the productivity of your employees, and to improve the overall quality of goods and services. Measurement is necessary to prove the effectiveness of an organization and to increase quality and productivity. Measurement can be used as a thermometer to monitor activities, as a thermostat to change activities, and as radar to serve as an early warning indicator of problems. Measurement is also a management tool to ensure that positive progress is made toward achieving goals and objectives. This two-day seminar will equip you with a customizable process for measurement designed to drive management's objectives. In this seminar you will learn how to strategically align a measurement program through the use of strategic and tactical dashboards." http://www.qaiworldwide.org/qaicampus/schedule.html
Quality America	Founded in 1983, Quality America offers Statistical Process Control (SPC) and Document Control software, as well as ASQ Certification materials, SPC Training, and Six Sigma software, training, and consulting. http://www.qualityamerica.com/
ReliaSoft Corporation	Reliability Software, Training, Consulting & Related Services from ReliaSoft Corporation - ReliaSoft Corporation provides reliability software, reliability training, consulting and services for reliability engineering and related fields. http://www.ReliaSoft.com
RIAC Training Courses	From the RIAC Web Site: "The Reliability Information and Analysis Center (RIAC) provides training in virtually every aspect of reliability, maintainability, quality, supportability and interoperability. RIAC instructors have years of diverse practical reliability problem solving experience. Scheduling is flexible and, most importantly, you can typically train up to 30 people for the cost of sending four people to one of our standard open courses." http://www.theriac.org/training/courses.swn

Glossary
Anomaly	Any condition that deviates from expectations based on requirements specifications, design documents, standards, etc. or from someone�s perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis, compilation, or use of software products or applicable documentation. [IEEE STANDARDS BOARD, 2002]
CMMI	Capability Maturity Model Integration
CVS	Concurrent Versions System
Defect	(1) Any unintended imperfection that impairs the utility or worth of an item, (2) Any kind of shortcoming, imperfection or deficiency, (3) Any flaw or imperfection in a software work product or software process. Examples include such things as mistakes, omissions and imperfections in software artifacts, or faults contained in software sufficiently mature for test or operation. [Abran, 2004]
Error	(1) differences between computed, observed, or measured values and the true, specified, or theoretically correct value or conditions, (2) an incorrect step process or data definition, (3) an incorrect result, (4) a human action that produces an incorrect result. Distinguished by using "error" for (1), "fault" for (2), "failure" for (3), and "mistake" for (4). [IEEE STANDARDS BOARD, 1990]
Failure	The inability of a system or component to perform its required functions within specified performance requirements. Note: The fault tolerance discipline distinguishes between a human action (a mistake), a manifestation (a hardware or software fault), the result of a fault (a failure), and the amount by which the result is incorrect (the error). [IEEE STANDARDS BOARD, 1990]
Fault	An incorrect step, process, or data definition in a computer program. In common usage, the terms "error" and "bug" are used to express this meaning. [IEEE STANDARDS BOARD, 1990]
FRACAS	Failure Reporting Analysis and Corrective Action System.
Mistake	A human action that produces an incorrect result. [IEEE STANDARDS BOARD, 1990]
ODC	Orthogonal Defect Classification
Operational Profile	The set of operations that the software can execute along with the probability with which they occur. [Lyu, 1996]

Bibliography

[Abran, 2004]	Abran, A., J. Moore, P. Bourque, and R. Dupuis, Guide to the Software Engineering Body of Knowledge (SWEBOK): 2004 Version, IEEE Computer Society (2004)
[Abts, 2000]	Abts, C., B. Boehm, A. Brown, S. Chulani, B. Clark, and E. Horowitz, Software Cost Estimation With COCOMO II, (2000)
[Bassin, 2001]	Bassin, K., and P. Santhanam, "Managing the Maintenance of Ported, Outsourced, and Legacy Software via Orthogonal Defect Classification", IEEE International Conference on Software Maintenance (November 2001): 726-734
[Bhandari, 1992]	Bhandari, I., J. Chaar, R. Chillarege, M. Halliday, D. Moebus, B. Ray, and M. Wong, "ORTHOGONAL DEFECT CLASSIFICATION - A CONCEPT FOR IN-PROCESS MEASUREMENTS", IEEE Transactions on Software Engineering V. 18 N. 11 (November 1992): pp. 943-956
[Black, 2000]	Black, R., "Investing in Software Testing: The Cost of Software Quality", ezine (January 2000): p1
[Brown, 1996]	Brown, N., "INDUSTRIAL-STRENGTH MANAGEMENT STRATEGIES, PP. 94-103.", IEEE Software V. 13 N. 4 (July 1996): 94-103
[Brown, 1998]	Brown, C., S. Cassell, J. Seppy, and Software Productivity Consortium, Inc., Defect Prevention Guidebook, (1998)
[CMMI PRODUCT TEAM, 2007]	CMMI PRODUCT TEAM, CMMI for Acquisition, Version 1.2, Software Engineering Institute, CMU/SEI-2007-TR-017 (2007)
[Campo, 2007]	Campo, M., and A. Frost, "Advancing Defect Containment to Quantitative Defect Management", CrossTalk V. 20 N. 12 (December 2007): 24-28
[Casey, 1991]	Casey, B., E. Kaldon, J. Sun, and J. Watters, "Application of defect analysis techniques to achieve continuous quality and productivity improvements", IEEE International Conference on Communications, 1991 (June 1991): 1450-1454
[Chrissis, 2007]	Chrissis, M., M. Konrad, and S. Shrum, CMMI Second Edition: Guidelines for Process Integration and Product Improvement (Version 1.2), Addison-Wesley (2007)
[Demirors, 2000]	Demirors, O., and O. Yildiz, "Using Cost of Software Quality as a Process Improvement Initiative", Proceedings of the 26th EuroMicro Conference (January 2000): p1
[Earl, 1990]	Earl, A., A Reference Model for Computer Assisted Software Engineering Environment Frameworks, (1990)
[El Emam, 2005]	El Emam, K., The ROI from Software Quality, (2005)
[Evans, 2006]	Evans, M., and D. Galorath, Software Sizing, Estimation, and Risk Management, (2006)
[Fagan, 1986]	Fagan, M., "Advances in Software Inspections", IEEE Transactions on Software Engineering V. 12 N. 7 (July 1986): pp. 744-751
[Fagan, 1976]	Fagan, M., "Design and code inspections to reduce errors in program development", IBM Systems Journal V. 15 N. 3 (January 1976): 181-211
[Fenton, 1997]	Fenton, N., and PFLEEGER. SHARI L., Software Metrics: A Rigorous & Practical Approach, (1997)
[Ferens, 2009]	Ferens, D., Quality Assurance and Engineering, DACS Data & Analysis Center for Software (2009)
[Ferens, 2010]	Ferens, D., Measurement and Analysis, DACS Data & Analysis Center for Software (2010)
[Ferens, 2007]	Ferens, D., T. Mcgibbon, and R. Vienneau, A Business Case for Software Process Improvement (2007 Update): Measuring Return on investment from Software Engineering, DACS Data & Analysis Center for Software (2007)
[Freimut, 2005]	Freimut, B., L. Briand, and F. Vollei, "Determining Inspection Cost-Effectiveness by Combining Project Data and Expert Opinion", IEEE Transactions on Software Engineering V. 31 N. 12 (December 2005): pp. 1074-1092
[Gack, 2009]	Gack, G., Software Defect Containment, Process-Fusion.net (2009)
[Gokhale, 2002]	Gokhale, S., and K. Trivedi, "Reliability prediction and sensitivity analysis based on software architecture", Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE 02) (November 2002): p64-75
[Gokhale, 1998]	Gokhale, S., S. Ramani, and K. Trivedi, "SREPT: Software Reliability Estimation and Prediction Tool", Computer Performance Evaluation (December 1997): p1
[Hart, 1982]	Hart, J., "THE EFFECTIVENESS OF DESIGN AND CODE WALKTHROUGHS", COMPSAC 1982, Proceedings (November 1982): p1
[Hofmann, 2007]	Hofmann, H., D. Yedlin, J. Mishler, and S. Kushner, CMMI For Outsourcing: Guidelines for Software, Systems, and IT Acquisition, Pearson Education Inc. (2007)
[IEEE STANDARDS BOARD, 1990]	IEEE STANDARDS BOARD, IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990, IEEE (1990)
[IEEE STANDARDS BOARD, 2002]	IEEE STANDARDS BOARD, IEEE Standard Classification for Software Anomalies, IEEE Std 1044-1993(R2002), IEEE (2002)
[IEEE STANDARDS BOARD, 1996]	IEEE STANDARDS BOARD, IEEE Guide to Classification for Software Anomalies, IEEE (1996)
[Jones, 2008]	Jones, C., Applied Software Measurement - Third Edition, McGraw-Hill Book Company (2008)
[Jones, 2007]	Jones, C., "Geriatric Issues of Aging Software", CrossTalk V. 20 N. 12 (December 2007): 4-8
[Kan, 1995]	Kan, S., Metrics and Models in Software Quality Engineering, (1995)
[Konrad, 2003]	Konrad, M., S. Shrum, and M. Chrissis, CMMI Guidelines for Integration and Product Improvement, Addison-Wesley Publishing Company (2003)
[Konrad, 1997]	Konrad, M., S. Shrum, and M. Chrissis, Software Engineering 1997 Edition, The Institute of Electrical and Electronics Engineers Inc. (1997)
[Lindvall, 2003]	Lindvall, M., D. Cohen, and P. Costa, Agile Software Development, (2003)
[Linger, 1999]	Linger, R., J. Poore, S. Prowell, and C. Trammell, Cleanroom Software Engineering: Technology and Process, (1999)
[Lyu, 1996]	Lyu, M., Handbook of Software Reliability Engineering, (1996)
[Malaiya, 2000]	Malaiya, Y., and J. Denton, "Module size distribution and defect density", (October 2000): 62 - 71
[Mann, 1974]	Mann, N., R. Schafer, and N. Singpurwalla, Methods for Statistical Analysis of Reliability and Life Data, John Wiley & Sons Inc. (1974)
[Marciniak, 1994]	Marciniak, J., Encyclopedia of Software Engineering, (1994)
[Mcgarry, 2002]	Mcgarry, J., B. Layman, E. Clark, J. Dean, F. Hall, D. Card, and C. Jones, Practical Software Measurement: Objective Information for Decision Makers, Addison Wesley (2002)
[Mcmanus, 1999]	Mcmanus, J., and G. Schulmeyer, Handbook of Software Quality Assurance - 3rd Edition, (1999)
[Mohapatra, 2001]	Mohapatra, S., and B. Mohanty, "Defect prevention through defect prediction", IEEE International Conference on Software Maintenance (November 2001): 260-272
[Musa, 1999]	Musa, J., Software Reliability Engineering: More Reliable Software; Faster Development and Testing, (1999)
[Myers, 2003]	Myers, W., and L. Putnam, Five Core Metrics: The Intelligence Behind Successful Software Management, (2003)
[Nicholls, 2001]	Nicholls, D., E. Fedchak, and R. Vienneau, The DACS Software Reliability Sourcebook, (2001)
[Paul, 2000]	Paul, R., F. Bastani, I. Yen, and V. Challagulla, "Defect-based reliability analysis for mission-critical software", Compsac 2000 (October 2000): p1
[Paulk, 1994]	Paulk, M., C. Weber, B. Curtis, and M. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley Professional (1994)
[Polydys, 2009]	Polydys, M., and S. Wisseman, "Software Assurance in Acquisition: Mitigating Risks to the Enterprise", (February 2009): p1
[Pressman, 2005]	Pressman, R., Software Engineering: A Practitioner's Approach - sixth edition, (2005)
[Project Management Institute, 2008]	Project Management Institute, A Guide to the Project Management Body of Knowledge (PMBOK GUIDE): Fourth Edition, Project Management Institute, Inc. (2008)
[SPMN, 2000]	SPMN, Sixteen Critical Software Practices for Performance-Based Measurement, Version 5.2, (2000)
[Schwalbe, 2006]	Schwalbe, K., Information Technology Project Management: Fourth Edition, Thomson Course Technology (2006)
[Tian, 2005]	Tian, J., Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, (2005)
[Turner, 2002]	Turner, R., "Implementation of Best Practices in US Department of Defense Software-Intensive Systems Acquisitions", George Washington University (January 2002): p1
[Van Solingen, 1999]	Van Solingen, R., and E. Berghout, The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development, (1999)
[Walker, 2005]	Walker, E., GOAL-QUESTION-METRIC (GQM) APPROACH, (2005)
[Webb, 2008]	Webb, G., and L. Patton, "Quality and Cost - It's Not Either/Or: Making the Case With Cost of Quality", CrossTalk V. 21 N. 11 (November 2008): 23-28

Copyright © 2010, ITT Corporation Privacy Policy	webmaster@thedacs.com 775 Daedalian Drive Rome, NY 13441 (800) 214-7921 Fax: 315-838-7130
This site is best viewed in Firefox 1.0+ or IE 6.0+