FORMAL INSPECTIONS
FOCUS AREA: QUALITY – Risk Management
Definition and Summary: The practice of conducting formal Inspections on requirements, architecture, designs at all levels (particularly detailed design), on code prior to unit test, and on test plans, to assess the quality of all baselined artifacts prior to release for project use.
Specific process steps of the formal inspections practice are (1) Planning, (2) Overview, (3) Preparation, (4) Inspection meeting, (5) Rework, and (5) Follow-up.
The inspection process includes the collection of data with which one can analyze the effectiveness of the process. There are many articles reporting the results of such analyses. Some researchers have summarized these reports. Table 1 shows the effectiveness of inspections, in percent of defects caught, based on data from the literature. Inspection effectiveness can be seen as a benefit of inspections. Table 2 shows the average effort to detect defects in hours per defect.
Table 1: Distribution of Defect Detection Effectiveness of Inspections [Briand, et. al,. 1998]
Defect Detection
Technique |
Minimum Value |
Most Likely Value |
Maximum Value |
Design Inspections |
25% |
57% |
84% |
Code Inspections |
19% |
57% |
70% |
Table 2: Distribution of Average Effort To Detect Defects [Briand, et. al., 1998]
Defect Detection
Technique |
Minimum Value |
Most Likely Value |
Maximum Value |
Design Inspections |
0.58 |
1.58 |
2.9 |
Code Inspections |
0.67 |
1.46 |
2.7 |
Testing |
4.5 |
6 |
17 |
Software inspections allow software development teams to find defects earlier and cheaper, thus reducing rework costs. In addition, there are other benefits more difficult to quantify. Software inspections aid in project management; and they provide more definite and more dependable milestones than less formal review processes. Software inspections can also promote closer teamwork, provide on-the-job training, and support the transfer of skills from more experienced team members to others.
There are some costs to starting up inspections, specifically in training management, moderators, and inspectors. The conduct of trained moderators and the attitude of management are key to the acceptance of inspections by engineers. Inspection results should not be used for personnel performance appraisals.
SUMMARY DESCRIPTION
Formal inspections are a type of review distinguished from other types of reviews by being:
· In process
· Directed towards finding defects
· Conducted following a certain rigorous process
These reviews are conducted during the development cycle. They are not a post-mortem meant to identify lessons learned from a completed project. Some reviews, such as Preliminary Design Reviews and Critical Design Reviews, are intended to show that a design is complete to a certain level of elaboration. Unlike formal inspections, these reviews are focused more on explaining a design than identifying defects. A defined process, with definite steps, participant roles, entry and exit criteria, and data to be collected, distinguishes formal inspections from more informal walkthroughs. Formal inspections differ from management reviews, informal walkthroughs, audits, and technical reviews not following the Fagan process.
The table below defines the inspection process.
Inspection Process Steps
Step |
Description |
Planning |
Confirms material to be inspected meets entry criteria. Arranges the availability of appropriate participants. Schedules a meeting place and time. |
Overview |
Educates group of participants in what is to be inspected. Assigns inspection roles to participants. |
Preparation |
Participants separately learn the material and find potential defects |
Inspection Meeting |
Identified defects are agreed on by the group and classified |
Rework |
The author corrects all defects |
Follow-up |
The moderator or the entire team verifies that all fixes are effective and that no additional defects have been introduced |
DETAILED DESCRIPTION
Formal inspections are a type of review distinguished from other types of reviews by being:
· In process
· Directed towards finding defects
· Conducted following a certain rigorous process
These reviews are conducted during the development cycle. They are not a post-mortem meant to identify lessons learned from a completed project. Some reviews, such as Preliminary Design Reviews and Critical Design Reviews, are intended to show that a design is complete to a certain level of elaboration. Unlike formal inspections, these reviews are focused more on explaining a design than identifying defects. A defined process, with definite steps, participant roles, entry and exit criteria, and data to be collected, distinguishes formal inspections from more informal walkthroughs. Formal inspections differ from management reviews, informal walkthroughs, audits, and technical reviews not following the Fagan process [IEEE 1997].
A number of documents (e.g., [Ebanau and Strauss 1994], [Fagan 1976], [Fagan 1986], [IEEE 1997], [Gilb and Graham 1993], [Laitenberger 2002], and [NASA 1993]) characterize formal inspections. This section presents a consolidate overview of these characterizations.
An inspection begins with an artifact to be inspected. The artifact must conform to standards applicable to the project (e.g., coding standards). The artifact must be accompanied by a document of its design or specification. Code is inspected against a low-level design document; a low-level design document is inspected against a high-level design document; and so on. For re-inspection of an artifact, all documented issues are resolved.
Other inputs to an inspection might include a documented inspection procedure, checklists, and data collection forms.
Table 1 defines the inspection process. In the overview, the author introduces the other members of the inspection team to the artifact to be inspected.
Each team member examines the inspected artifact during the preparation phase. The main goal of the preparation phase is to thoroughly understand the artifact’s “intent and logic”, not to identify defects [Porter and Johnson 1997]. Issues raised during this phase, however, are recorded. The inspectors use a reading technique defined for the relevant variant of the inspection process.
The goal of the inspection meeting is to identify defects, not to propose or analyze solutions. A moderator leads the meeting. A reader, who is typically not the author of the inspected artifact, steps the inspectors through the artifact in a systematic and orderly fashion. Defects found during this phase are recorded and categorized. These defects include issues raised during preparation and agreed upon by the group.
The author fixes defects during the rework phase. The correctness of these repairs is verified during the follow-up phase.
Table 1: Inspection Process Steps
Step |
Description |
Planning |
Confirms material to be inspected meets entry criteria. Arranges the availability of appropriate participants. Schedules a meeting place and time. |
Overview |
Educates group of participants in what is to be inspected. Assigns inspection roles to participants. |
Preparation |
Participants separately learn the material and find potential defects |
Inspection Meeting |
Identified defects are agreed on by the group and classified |
Rework |
The author corrects all defects |
Follow-up |
The moderator or the entire team verifies that all fixes are effective and that no additional defects have been introduced |
Table 2 summarizes the roles of members of the inspection teams. Typically, more than one inspector will conduct an inspection. Inspectors may be assigned to represent different viewpoints (e.g., user, requirements, tester) or to focus on different aspects of the artifact under inspection.
Table 2: Inspection Roles
Role |
Description |
Moderator |
Responsible for administrative tasks, disseminating inspection materials, scheduling meetings, moderating the inspection meeting, collecting data, overseeing follow-up, and issuing the inspection report |
Author |
Responsible for artifact to be inspected satisfying inspection entry criteria, for providing the overview of the product, for providing clarifications at the meeting, and for rework needed to satisfy exit criteria. |
Reader |
During the meeting, leads the inspection team through the artifact being inspected. Interprets sections of the artifact by paraphrase. Highlights important parts. |
Inspector |
Familiarizes himself/herself with the artifact to be inspected and identifies issues with it |
Recorder |
Documents issues, decisions, and recommendations. Records inspection data for process analysis. |
Specifications of the software inspection process usually include definitions of data to be collected. Such data includes defects detected (categorized by severity, etc.), the size of the inspected material, the effort expended by each inspector in preparing for the inspection, and the effort spent in the inspection meeting. This data is typically analyzed to determine the defect detection effectiveness of inspections, the average effort to detect defects, and the costs saved from avoiding rework later in the software development process.
The exit criteria for an artifact that has been inspected is that entry criteria continue to be met and that all rework resulting from the inspection has been included in the artifact and verified.
SUMMARY CHARACTERISTICS
NO DATA CURRENTLY AVAILABLE
The inspection process includes the collection of data with which one can analyze the effectiveness of the process. There are many articles reporting the results of such analyses. Some researchers have summarized these reports. [Briand, et. al., 1998] describe an efficiency benchmark. They define Defect Detection Effectiveness as the percentage of defects in an artifact discovered by a detection technique. The first table below shows the effectiveness of inspections, in percent of defects caught, based on data from the literature. Inspection effectiveness can be seen as a benefit of inspections. The next table shows the average effort to detect defects in hours per defect. Briand, et. al., use a Monte Carlo technique to combine this data to yield a measure of the efficiency of inspections.
Distribution of Defect Detection Effectiveness of Inspections [Briand, et. al.,1998]
Defect Detection Technique |
Minimum Value |
Most Likely Value |
Maximum Value |
Design Inspections |
25% |
57% |
84% |
Code Inspections |
19% |
57% |
70% |
Distribution of Average Effort To Detect Defects [Briand, et. al.,1998]
Defect Detection Technique |
Minimum Value |
Most Likely Value |
Maximum Value |
Design Inspections |
0.58 |
1.58 |
2.9 |
Code Inspections |
0.67 |
1.46 |
2.7 |
Testing |
4.5 |
6 |
17 |
Financial analyses can present this sort of data in various ways. The figure presented next, based on the most likely value from the table above and the model in [Vienneau 1995], shows that the net present values of code inspections is greater than the net present value of testing for finding defects for practical ranges of schedules and interest rates. The figure does not imply testing should not be performed. Finding defects is not the only goal of testing. For example, testing is still needed to assess reliability.
Present Values Of Two Bug-Detection Methods
Grady and Van Slack (1994) summarize the benefits of testing in another fashion, as seen in the next table. An increasing implementation of inspections has moved rework to earlier in the lifecycle. The last column shows the dollar benefits of avoiding later rework.
Hewlett Packard Experience [Grady and Van Slack 1994]
Work Product |
Estimated Starting Point (Extent of Adoption) |
Estimated 1993 (Extent of Adoption) |
Percentage Total Cost Saved |
Rework Percentage |
Estimated $ Savings Per Year |
Specification |
1% |
29.5% |
28.5% |
17% |
$10,175,000 |
Design |
1% |
34.8% |
33.8% |
11% |
$7,808,000 |
Code |
5% |
42.3% |
37.3% |
4% |
$3,133,000 |
Test Plan |
1% |
17.1% |
16.1% |
1% |
$338,000 |
TOTAL |
33% |
$21,454,000 |
Software inspections allow software development teams to find defects earlier and cheaper, thus reducing rework costs. In addition, there are other benefits more difficult to quantify. Software inspections aid in project management; they provide “more definite and more dependable milestones than less formal review processes” [Ackerman, et. al.,1989]. Software inspections promote esprit de corps, provide on-the-job training, and support the transfer of skills from more experienced team members to others [Doolan 1992].
There are some costs to starting up inspections, specifically in training management, moderators, and inspectors. Fagan [1976] states that the conduct of trained moderators and the attitude of management are key to the acceptance of inspections by engineers. Inspection results should not be used for personnel performance appraisals.
DETAILED CHARACTERISTICS
Researchers and practitioners have applied a number of variants to Fagan’s original formal inspection techniques. These variants diverge on roles and process steps, reading techniques, and checklists.
Formal Inspection Variants
Active Design Reviews (ADRs) |
Reviewers have different focuses and participate in a series of specialized reviews. Questionnaires are specialized for each review. [Parnas and Weiss 1985] |
Formal Technical Asynchronous review method (FTArm) |
A type of asynchronous inspection in which the inspectors never have to simultaneously meet. [Johnson 1994] |
Gilb Inspection |
Preparation phase (called logging) emphasizes finding bugs. Meeting may include brainstorming session in which bug fixes are suggested. [Macdonald and Miller 1995b] |
Humphrey’s Inspection Process |
described by Watt’s Humphrey. Preparation phase emphasizes finding and logging bugs, unlike Fagan inspections. Includes analysis phase between preparation and meeting. In analysis phase, individual logs are analyzed and combined into single list. [Macdonald and Miller 1995b] |
N-Fold inspections |
For requirements. N independent teams conduct inspections of the same User Requirements Document, supervised by one moderator. [Martin and Tsai 1990] |
Phased Inspection |
Each phase ensures artifact has a single specific property or a small set of related properties. Not solely for finding errors. [Knight and Myers 1993] |
Structured Walkthrough |
Described by Yourdon. less formal and rigorous than formal inspections. Roles are coordinator, scribe, presenter, reviewers, maintenance oracle, standards bearer, user representative. Process steps are Organization, Preparation, Walkthrough, and Rework. [Macdonald and Miller 1995b] Lacks data collection requirements of formal inspections. |
Two-Person Inspection |
Implemented experimentally with undergraduate students as subjects. [Bisant and Lyle 1989] |
Verification-Based Inspection |
Organized around a method developed by Harlan Mills for formally proving programs. [Britcher 1988] |
Roles differ among the variants described above. In some the differences are merely in the name of a role. For example, in some cases the recorder is called a scribe. The moderator role can be partitioned into two roles – one role for administrating the process and another for moderating the inspection meeting. In ADRs and Structured Walkthroughs, for example, inspectors are chosen to represent specific concerns. Two inspectors, one of whom is the author of the artifact under inspection, participate in Two-Person Inspections.
The steps in formal inspections can be divided into three phases:
· Organization: Planning and Overview
- A single Organization step in Structured Walkthroughs and Phased Inspections
- Entry, Planning and Kickoff steps in Gilb Inspections
- Setup and Orientation steps in FTArm
- A single Overview step in ADR
· Detection: individual Preparation and the Inspection Meeting
- Preparation and Walkthrough steps Structured Walkthroughs
- Preparation, Analysis, and Inspection steps in Humphrey Inspections
- Checking, Logging, and Brainstorming steps in Gilb Inspections
- Private Review, Public Review, Consolidation, and Group Review steps in FTArm
- A single Review step in ADR
- Examination, Inspection, and Reconciliation steps in Phased Inspections
- Inspection steps, conducted in parallel, followed by a single Collation step in N-Fold Inspections
· Completion: Rework and Follow-Up
- A single Rework step in Structured Walkthroughs
- Edit, Follow-up, and Exit steps in Gilb Inspections
- A single Conclusion step in FTArm
- Discussion and Rework steps in ADR
- A single Completion step in Phased Inspections
The content of these steps differs, as well as their names. More recent variants have transformed the goal of the Preparation or Private Review step from having the inspectors learn the material to finding bugs. A meeting can be costly and difficult to schedule. Some of the variants for software inspections have been designed to make meetings more effective or replace a physical meeting with distributed and asynchronous virtual meetings mediated through computer tools. ADRs, Phased Inspections, and Verification-Based Inspections adopt more focused meetings than Fagan’s original approach. FTArm is an example of a technique with virtual meetings.
Reading techniques, some of which are closely tied to particular variants of formal inspections, are another method of increasing the effectiveness of formal inspections. Ad-hoc and checklists reading are industrial practice. Perspective-based reading has experienced initial industrial use. Reading by stepwise-abstraction is applied in Cleanroom projects. An initial case study applied ADRs. [Laitenberger 2002]
Reading Techniques
Reading Technique |
Description |
Active Design Review (ADR) |
An inspection method developed by Parnas and Weiss, in which an inspector is guided in his reading by a questionnaire provided by the author of the inspected artifact. The questions are open-ended, not yes/no questions, and are typically organized around assertions about the design. |
Ad-hoc |
No systematic guidance provided to inspectors |
Checklist-based reading |
General, untailored guidance provided by checklists |
Defect-based reading |
For requirements documents. Scenarios are organized around defect classes. |
Function Point-based reading |
Function Points provide a software size metric involving a weighted sum of inputs, files, inquiries, and outputs. Scenarios are organized around these items. |
Perspective-based reading |
Scenarios are organized around the perspectives of different stakeholders in a software program. For example, a tester’s perspective emphasizes the design of test cases. |
Scenario-based reading |
A general term including Defect-based reading, Function Point-based reading, and Perspective-based reading. A scenario, encapsulated in a set of questions or a detailed description, guides an inspector’s reading. |
Reading by stepwise abstraction |
Associated with Cleanroom software development. The reader develops a description of the function of individual code segments, typically expressed in a set-theoretic notation. This is repeated until the reader expresses the function of the entire artifact. |
Formal inspections and certain variants (e.g., ADR, FTArm, Gilb Inspections, Phased Inspections, and Verification-Based inspections) use checklists or questionnaires. These checklists vary with the technique. The first table below shows a generic checklist that might be used with a Fagan inspection. The author and inspectors collaborate to develop questionnaires in Verification-based Inspections. The next table provides an example. Questionnaires in ADR are tailored to the artifact under inspection. The third table shows an example of part of an ADR questionnaire. The variation in the style of the questions in these checklists and questionnaires illustrates software inspection variants.
A Generic Checklist [Ackerman, et. al., 1989]
Completeness |
1. Are all sources of input identified? |
2. What is the total input space? |
3. Are there any “don’t care” sets in the input space? |
4. Is there a need for robustness across the entire input space? |
5. Are there any timing constraints on the inputs? |
6. Are all types of outputs identified? |
7. What is the total output space? |
8. Are there any timing constraints on the output space? |
9. Are there sets that are in both the input space and the output space? Is there any state information? |
10. What are all the types of runs? |
11. What is the input space and the output space of each type of run? |
12. For each type of run, is an output value specified for each input value? |
13. Is the invocation mechanism for each run type defined? Are initial states defined? |
14. Are all environmental constraints described? |
15. Are sample input/output examples provided where appropriate? |
16. Are all necessary performance requirements described? |
17. Should reliability requirements be specified? Is an operational profile specified? |
Ambiguity |
1. Are all special terms clearly defined? |
2. Does each sentence have a single interpretation in the problem domain? |
3. Is the input-to-output mapping clearly defined for each type of run? |
Consistency |
1. Do any of the designated requirements conflict with the descriptive material? |
2. Are there any input states that are mapped to more than one output state? |
3. Is a consistent set of quantitative units used? Are all numeric quantities consistent? |
A Checklist for Verification-Based Inspections [Britcher 1988]
Topology |
Does the program (procedure) represent a function derived from a state machine? |
Is the state machine defined precisely? |
How many variables comprise the state data? Are they related? Does the state represent a coherent set of values? |
How large is the state space? Is the range of values represented by the variables too large? |
Does the program act on an inordinately high percentage of the variables in the state space? |
Does the program modify the state or just refer to it? |
Does the program modify or refer to state data in another state machine? |
How many local variables were invented in the procedure and what is their rationale? |
How complex is the procedure? How many predicates does it contain and how deeply are they nested? |
Does the program consist of proper programs that are easy to identify and replace with more abstract proper programs? |
Algebra |
Is the state machine specified in an abstract and precise way, or does it just repeat the information of an earlier (higher-level) specification? |
Is the program specified precisely, as in the form o = f(I)? |
Is the program’s input domain and output range a subset of the inputs and outputs defined by the state machine specification? |
Does the exhibit define the initial state? |
Does it precisely define the range of possible values in the state? |
Does it precisely specify the proper programs that compose the program, and does the specification appear in the program? |
Is each proper program functionally equivalent to its specification? |
Robustness |
How rich is the input domain? How many variables are imported and how coherent are they? |
How rich is the output range? Is enough information conveyed to the program user? |
Is the range of values explicit in the input domain and the output range? |
Does the program guard the input domain? |
If the program invokes other functions that are defined in other state machines, are the functions, results, and possible error returns defined explicitly? |
Is the domain of each predicate explicitly defined so that each can be evaluated as true or false? |
Does the program define all conditions that could result in an error? |
Is the meaning of each error clear? |
Does this program return error values to its users and are their meanings clear? |
Active Design Review for Consistency Between Assumptions and Functions [Parnas and Weiss 1985]
For each of the access functions the reviewer should answer the following questions: |
C1. Which assumptions tell you that this function can be implemented as described? |
C2. Under what conditions may this function be applied? Which assumptions describe those conditions? |
C3. Is the behavior of this function, i.e., its effects on other functions, described in the assumptions? |
The Figure below represents a high-level process architecture for the subject practice, depicting relationships among this practice and the nature of the influences on the practice (describing how other practices might relate to this practice). These relationship statements are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may “influence” (or are influenced by) the ability to successfully implement other practices. A brief description of these influences is included in the table below.
Process Architecture for the "Formal Inspections" Gold Practice
Summary of Relationship Factors
INPUTS TO THE PRACTICE |
Identify what to inspect
|
Several practices influence the Formal Inspection (FI) practice by providing artifacts for inspection. Leveraging COTS/NDI typically includes an analysis process. Inspections can be performed on analysis results (or artifacts of the analysis process) in preparation for technical decision making regarding COTS selection or implementation. Applying FI to Reuse Components can identify issues and problems relating to reuse of the components in the new environment thus mitigating some of the risks. Agreement on Interfaces addresses identifying all the interfaces and understanding their functions. The artifacts used to represent the interfaces should be inspected for accuracy and completeness as part of the agreement process. The Architecture-First Approach is an iterative approach that produces technical artifacts (architectural diagrams from a variety of perspectives, or demos), which are reviewed by key stakeholders with each iteration resulting in refinement until the architecture becomes stable. Embedding a formal inspection practice into the review cycle will likely find flaws in the architecture. Feedback from the results of inspections can then be used to improve the architecture. The Manage Requirements practice influences formal inspections in two ways: (1) Requirements documents are subjected to formal inspections to improve the quality and completeness of the requirements and (2) specific requirements are often part of the inspection criteria that drives the inspection process. This presumes an iterative requirements process where inspection results are actionable items that result in appropriate requirements modifications. The Goal-Question-Metric Approach aids in establishing the relative importance of issues and requirements thus helping to determine what needs the rigor of the formal inspection process vs. some other verification process. |
Establish inspection criteria
|
Establishing Clear Goals and Decision Points up front provides a focus and milestone schedule for assessing progress and deciding on the future path a project may take. Thus it may address when Formal Inspections should occur since they may be used as key decision points. In addition to requirements documents being the target of a formal inspection, detailed requirements often are part of the inspection criteria – especially those that relate to quality and performance. When a program adopts the Open Systems Approach to development Formal Inspections assess compliance with the appropriate selected Commercial Specifications and Standards. Binary Quality Gates are completion criteria for tasks and are defined at a high degree of granularity so that determining completion is a trivial matter of checking “done” or “not done” rather than making a subjective judgment. Each gate may be comprised of one to many criteria statements. The value of the practice lies in the quality and completeness of the quality gate definitions. Formal Inspections can be applied to improve the quality of those definitions, and then the definitions are used as inspection criteria by the Formal Inspections practice in tracking progress on tasks. |
Manage the inspection process |
Formal Inspections follow a defined process with definite steps, participant roles, entry and exit criteria, and data to be collected (actionable output). They do not fit well with “ad hoc” development. In order for inspections to work project leaders need to provide the appropriate resources, specifically time within the project schedule, and training for participant roles. This is part of the People Aware Management Accountability practice. Configuration Management is needed to ensure the integrity of the artifacts that are the subject of Formal Inspection, and to ensure that inspection results are actually transferred to corrective actions and tracked through completion. This implies that the inspection process would feed into a defect tracking system as part of configuration management. A Structured Development Methodology is the natural environment for implementing formal inspections since results of inspections provide input to successive iterations of the product under development. Formal Inspections provide objective data to feed Statistical Process Control (SPC), which in turn is used to effect software development process improvement. SPC may be applied to the inspection process itself as part of an overall management strategy. When SPC finds that the FI process is “out of control” then the FI process would need to be modified. Applying a formal inspection to a project or product artifact may be part of the criteria for a Binary Quality Gate. Thus the two practices are managed in concert because of their complementary relationship. |
OUTPUTS FROM THE PRACTICE |
Use inspection results for improvement
|
Results of FI provide objective data for successive iterations of a product under development focusing on refinement or improvement of the product. If the artifact is an interface design, or an Architecture representation, an iterative cycle, typically associated with a Structured Development Method, enables optimum use of formal inspections results contributing to the refinement of the interface definitions or the architecture. The FI process generates defect data which is used in Defect Tracking Against Quality Targets (such as defects per function point) to monitor both process and product improvement. Formal Inspections are valuable tools for assessing compliance to specifications and standards and are thus part of the strategy for Ensuring Interoperability. |
Determine the effectiveness of inspections
|
Formal Inspections differ from other types of assessment in the degree to which they employ a rigorous process to the subject artifact, and in the objectivity of data that results. Objective data is necessary for and thus affects the quality of any Statistical Process Control implementation. Formal Inspections can impact the value of Demonstration-based Reviews by applying the rigorous process and defined participant roles to the review process relative to what is being demonstrated, contributing to a more focused and effective review. The goal of Defect Tracking Against Quality Targets is to identify a desired quality target and then seek to achieve it. Defects must be tracked in order to assess progress. Formal Inspections contribute to defect detection but also to defect prevention when applied to artifacts early in the life cycle. Thus they can be a major influence in reducing defects and achieving desired quality goals. Independent Expert Reviews (such as Software Capability Evaluations (SCEs)), are conducted to assess the degree to which a project uses “repeatable” defined processes that include process improvement cycles. Implementing Formal Inspections throughout a project is one way of achieving that process focus. The training and experience of participants during formal inspections tends to “spill over” into other aspects of the development process contributing to the overall acceptance and cultural migration to a process-oriented approach to software development. |
RESOURCES:
§ Websites
§ The documents and links presented at this site are to assist the practice of software peer reviews. You are free to use them as you like, to modify them to best suit your needs, and to share them with your colleagues. They are provided without warranty of any kind. http://www.processimpact.com/pr_goodies.shtml
§ Quality Assurance Links http://www.qalinks.com/
§ DACS catalog of sites for software inspections http://www.dacs.dtic.mil/databases/url/key.hts?keycode=165
§ Software Engineering Institute (SEI) from their Software Technology Review http://www.sei.cmu.edu/str/descriptions/inspections_body.html
§ NASA page, including links to the NASA Formal Inspection Guidebook and NASA-STD-2202-93 http://satc.gsfc.nasa.gov/fi/fipage.html
§ Bill Brykczynski’s bibliography http://gd.tuwien.ac.at/softeng/lambs-archive/archive/inspect
§ Don O’Neill’s Return On Investment (ROI) on inspections tool http://members.aol.com/ONeillDon/nsqe-roi.html
§ Tools and Methods
Software tools can assist software inspections, especially distributed and asynchronous inspections. Tool capabilities can be classified into four categories (see table below). There is also a need for software inspection tools to be integrated with other tools, such as Configuration Management Systems and issue tracking systems.
Tool Capabilities for Software Inspections (Based on [Macdonald, et. al., 1995a] and [Harjumaa, et. al., 2001])
Document Handling |
Document support; electronic distribution of inspection material; supports diagrams and non-textual material; includes hyperlinks; annotation support |
Individual Preparation |
Automatic detection of simple defects; enforcement of style guidelines; checklists; enforcement of minimum preparation time standards |
Meeting Support |
Distributed meeting support; support of asynchronous meetings; groupware capabilities; polling capabilities |
Data Collection |
Metrics collection; defect classification (e.g., by severity and category) facilities; decision support |
Software Inspection Tools
Tool Name |
Description |
Asynchronous Inspector of Software Artifacts (AISA) |
Web-based support for discussion threads, annotations, and coordination of inspections (same group produced CAIS.) |
AnnoSpec |
An annotation tool allowing users to broaden or narrow annotation visibility [Stein, et. al.,1999]. |
Asynchronous/Synchronous Software Inspection Support Tool (ASSIST) |
The Inspection Process Definition Language (IPDL) allows users to define their own inspection process. Includes browsers for ASCII documents, source code, and lists (e.g., of comments and defects). Supports annotations and checklists. Client/server architecture. (http://www.cs.strath.ac.uk/~efocs/home/assist.html) |
Collaborative Asynchronous Inspection of Software (CAIS) |
Early tool for asynchronous inspections. (http://www.cs.umn.edu/research/cais/development/html/overview.htm) |
CodeSurfer |
A static analysis tool for examining the system-dependence graph representation of programs in answering checklist questions. (http://www.grammatech.com/products/codesurfer/) |
Collaborative Software Inspection (CSI) |
Provides distributed, structured environment for inspections (same group later produced CAIS) |
Collaborative Software Review System (CSRS) |
Implements Formal, Technical, Asynchronous review method (FTArm). Supports document handling, annotations, issue tracking, and polling in a hypertext environment [Johnson 1994]. |
Internet-Based Inspection System (IBIS) |
Web-based groupware using email for event notification. Supports all phases of inspection process. Includes defect logging, checklists, asynchronous discussion, and defect tracking [Lanubile and Mallardo 2002]. |
Intelligent Code Inspection in a C Language (ICICLE) |
Finds common defects by performing static analysis (like lint). Allows recording of individual comments in preparation phase and in the inspection meeting. Comments classified, for example, as deferred, ignored, or transferred, and by category. Provides cross-referencing information on source code. During meetings, can coordinate user windows to be the reader’s view. Generates defect reports. |
InspectA |
E-mail based support for discussion threads, annotations, and coordination of inspections [Murphy and Miller 1997] |
Inspecting software in phases to ensure Quality (InspeQ) |
Provides display of work products, checklists, standards, highlighting, annotations. Supports tracking of personnel to phases, of files, and of phase completion [Knight and Myers 1993]. |
Lightweight Empirical Automated Portable (LEAP) tool |
Supports software review and personal process improvement through empirical data collection and analysis. (http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html) |
ReviewPro |
Web-based groupware which supports all phases of the inspection process, provides an on-line discussion forum, supports checklists, supports issue lists, and automates collection of certain metrics. (http://www.sdtcorp.com/reviewpro.htm) |
Site Annotation Tool for Inspection (SATI) |
For reviewing and commenting on any HTML-based document [Harjumaa, et. al., 2001]. |
Scrutiny |
Distributed system supporting geographically separated users. Supports inspection process from participant invitation to defect tracking. Provides private and shared annotation capabilities. Reader synchronizes inspectors’ screens during meeting. Supports defect collection and categorization. Provides polling and discussion capabilities [Gintell, et. al., 1995]. |
|
|
|
§ Experts/Contact Points
o Bordin Sapsomboon’s PhD Thesis: http://www.sis.pitt.edu/~cascade/bordin/
o Tom Gilb’s company: http://Result-Planning.com/
o Philip Johnson’s page: http://csdl.ics.hawaii.edu/~johnson/
§ Training Opportunities:
o Informal list of pointers to trainers and consultants http://www2.ics.hawaii.edu/~johnson/FTR/training
o SWQuality, Incorporated http://www.swquality.com/users/pustaver/seminars/. 1 Day Seminar on Moderating Software Inspections and 1 Day Seminar on Software Inspections: What Managers Need To Know
Bibliography:
[Ackerman, et. al.,1989] |
Ackerman, A. F., Buchwald, L. S., Lewski, F. H., “Software Inspections: An Effective Verification Process”, IEEE Software, May 1989, pp. 31-36. |
[Anderson, et. al.,2003] |
Anderson, P., Reps, T., Teitelbaum, T., and Zarins, M., “Tool Support for Fine-Grained Software Inspection”, IEEE Software, July/August 2003, pp. 42-50. |
[Barnard and Price 1994] |
Barnard, J. and Price, A. “Managing Code Inspection Information”, IEEE Software, V. 11, N. 2, March 1994, pp. 59-69. |
[Bisant and Lyle 1989] |
Bisant, D. B. and Lyle, J. R., “A Two-Person Inspection Method to Improve Programming Productivity”, IEEE Transactions on Software Engineering, V. 15, N. 10, October 1989, pp. 1294-1304. |
[Bourgeois 1996] |
Bourgeois, K. V., “Process Insights from a Large-Scale Software Inspections Data Analysis,” Crosstalk: The Journal of Defense Software Engineering, October 1996. |
[Briand, et. al.,1998]/p> |
Briand, L., El Emam, K., Laitenberger, O., and Fussbroich, T., “Using Simulation to Build Inspection Efficiency Benchmarks for Development Projects”, Proceedings of the 20th International Conference on Software Engineering, 1998, pp. 340-349. |
[Britcher 1998] |
Britchner, R. N., “Using Inspection to Investigate Program Correctness”, IEEE Computer, November 1998, pp. 38-44. |
[Collofello and Woodfield 1989] |
Collofello, J. S. and Woodfield, S. N., “Evaluating the Effectiveness of Reliability Assurance Techniques”, Journal of Systems and Software, V. 9, N. 3, March 1989, pp. 191-195. |
[Conradi, et. al.,1999] |
Conradi, R., Marjara, A. S., and Skatevik, B., “Empirical Studies of Inspection and Test Data”, Proceedings of the First Conference on Product-Focused Process Improvement, Oulo, Finland, 1999, pp. 263-284. |
[Doolan 1992] |
Doolan, E. P., “Experience with Fagan’s Inspection Method”, Software – Practice and Experience, V. 22, N. 2, February 1992, pp. 173-182. |
[Ebanau and Strauss 1994] |
Ebanau, R. G. and Strauss, S. H., “Software Inspection Process”, McGraw Hill, 1994. |
[Fagan, 1976] |
Fagan, M. E., "Design and Code Inspections to Reduce Errors in Program Development", IBM System Journal, V. 15, N. 3, 1976, pp. 182-211. |
[Fagan 1986] |
Fagan, M. E., “Advances in Software Inspections”, IEEE Transactions on Software Engineering, V. SE-12, N. 7, July 1986, pp. 744-751. |
[Franz and Shih, 1994] |
Franz, L. A. and Shih, J. C., “Estimating the Value of Inspections and Early Testing for Software Projects”, Hewlett-Packard Journal, December 1994, pp. 60-67. |
[Gately 1999] |
Gately, A. A., “Design and Code Inspection Metrics”, International Conference on Software Management and Applications of Software Measurement, San Jose, CA, 1999. |
[Gintell, et. al.,1995] |
Gintell, J. W., Houde, M. B., and McKenney, R. F., “Lessons Learned by Building and Using Scrutiny a Collaborative Software Inspection System”, Proceedings of the Seventh International Workshop on Computer-Aided Software Engineering, 10-14 July 1995, pp. 350-357. |
[Grady and Van Slack 1994] |
Grady, R. B. and Van Slack, T., “Key Lessons in Achieving Widespread Inspection Use”, IEEE Software, V. 11, N. 4, Month, 1994, pp. 46-57 |
[Gilb and Graham, 1993] |
Gilb, T. and Graham, D., “Software Inspection”, Addison-Wesley, 1993 |
[Harjumaa, et. al. 2001] |
Harjumaa, L., Hedberg, H. and Tervonen, I., “A Path to Virtual Software Inspection”, Proceedings of the Second Asia-Pacific Conference on Quality Software, 10-11 December 2001, pp. ???? |
[IEEE 1997] |
ANSI/IEEE Std. 1028-1997, “Standard for Software Reviews” |
[Jeffrey and Scott 2002] |
Jeffrey, R. and Scott, L., “Has Twenty-five Years of Empirical Software Engineering Made a Difference?", Proceedings of the Ninth Asia-Pacific Software Engineering Conference (APSEC’02), 2002, pp. ???? |
[Johnson 1994] |
Johnson, P. M., “An Instrumented Approach to Improving Software Quality Through Formal Technical Review”, Proceedings of the 16th International Conference on Software Engineering (ICSE-16), May 1994, pp. ???? |
[Kan 1995] |
Kan, S. H., “Metrics and Models in Software Quality Engineering”, Addison-Wesley, 1995 |
[Kaner 1998] |
Kaner, C., “The Performance of the N-Fold Requirement Inspection Method,” Requirements Engineering Journal, V. 2, N. 2, Month, 1998, pp. 114-116 |
[Kelly, et. al. 1992] |
Kelly, J. C., Sherif, J. S., and Hops, J., “An Analysis of Defect Densities Found During Software Inspections,” Journal of Systems and Software, V. 17, No. ??, Month, 1992, pp. 111-117 |
[Kitchenham, et. al. 1986] |
Kitchenham, B., Kitchenham, A., and Fellows, J., “The Effects of Inspections on Software Quality and Productivity”, ICL Technical Journal, Month, 1986, pp. ???? |
[Knight and Myers 1993] |
Knight, J. C., and Myers, E. A., “An Improved Inspection Technique”, Communications of the ACM, V. 36, N. 11, November, 1993, pp. 51-61 |
[Laitenberger 2002] |
Laitenberger, O., “A Survey of Software Inspection Technologies”. Handbook on Software Engineering and Knowledge Engineering, V. 2, World Scientific Publishing, Month, 2002, pp. 517-555 |
[Lanubile and Mallardo 2002] |
Lanubile, F. and Mallardo, T., “Tool Support for Distributed Inspection”, Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC’02), 2002, pp. ???? |
[Macdonald, et. al. 1995a] |
Macdonald, F., Miller, J., Brooks, A., Roper, M., and Wood, M., “A Review of Tool Support for Software Inspection”, Proceedings of the Seventh International Workshop on Computer-Aided Software Engineering, 10-14 July 1995, pp. ???? |
[Macdonald, et. al. 1995b] |
Macdonald, F. and Miller, J., “Modeling Software Inspection Methods for the Application of Tool Support”, December, 1995 |
[Martin and Tsai 1990] |
Martin, J. and Tsai, W. T., “N-Fold Inspection: A Requirements Analysis Technique”, Communications of the ACM, V. 33, N. 2, February, 1990, pp. 225-232 |
[McGibbon 1996] |
McGibbon, T., “A Business Case for Software Process Improvement”, Data & Analysis Center for Software, 1996 |
[Myers 1978] |
Myers, G. T., “A Controlled Experiment in Program Testing and Code Walkthroughs/Inspections”, Communications of ACM. V. 21, No. 9, Month, 1978, pp. 760-768 |
[Murphy and Miller 1997] |
Murphy, P. and Miller, J., “A Process for Asynchronous Software Inspection”, Proceedings of the Eighth IEEE International Workshop on Computer Aided Software Engineering, July, 1997, pp. ???? |
[NASA 1993] |
Software Formal Inspections Guidebook, Office of Safety and Mission Assurance, NASA-GB-A302, 1993 |
[Parnas and Weiss, 1985] |
Parnas, D. L. and Weiss, D. M., “Active Design Reviews: Principles and Practices,”Proceedings of the Eighth International Conference on Software Engineering, August, 1985, pp. ???? |
[Porter and Johnson, 1997] |
Porter, A. A. and Johnson, P. M., Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies, IEEE Transactions on Software Engineering, V. 23, N. 3, Month, 1997, pp. 129-145 |
[Porter and Votta, 1997] |
Porter, A. and Votta, L., “What Makes Inspections Work?”, IEEE Software, November/December, 1997, pp. ???? |
[Prowell, et. al. 1999] |
Prowell, S. J., Trammell, C. J., Linger, R. C., and Poore, J. H., “Cleanroom Software Engineering: Technology and Process”, Addison-Wesley, 1999 |
[Remus 1984] |
Remus, H., “Integrated Software Validation in the View of Inspections/Reviews”, Software Validation, Month, 1984, pp. 57-65 |
[Sauer, et. al. 2002] |
Sauer, C., Jeffrey, D. R., Land, L., and Yetton, P., “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research”, IEEE Transactions on Software Engineering, V. 26, N. 1, January, 2002, pp. ???? |
[Shirey 1992] |
Shirey, G. C., “How Inspections Fail,” Proceedings of the 9th International Conference on Testing Computer Software”, Month, 1992, pp. ???? |
[Stein, et.al. 1999] |
Stein, M. V., Heimdahl, M. P. E., and Riedl, J. T., “Enhancing Annotation Visibility for Software Inspection,” 14th IEEE International Conference on Automated Software Engineering, October, 1999, pp. ???? |
[Vienneau 1995] |
Vienneau, R. L., “The Present Value of Software Maintenance”, Journal of Parametrics, April, 1995, pp. ???? |
[Weller 1992] |
Weller, E. F., “Experiences with Inspections at Bull HN Information System,” Proceedings of the 4th Annual Software Quality Workshop, 1992, pp. ???? |
[Wheeler, et. al. 1996] |
Wheeler, D. A., Brykczynski, B., and Meeson, R. N. (editors), “Software Inspection: An Industry Best Practice”, IEEE Computer Society, Month, 1996, pp. ???? |
APPENDICES
“A visual examination of a software product to detect and identify software anomalies, including errors and deviations from standards and specifications. Inspections are peer examinations led by impartial facilitators who are trained in inspection techniques. Determination of remedial or investigative action for an anomaly is a mandatory element of a software inspection...” (ANSI/IEEE Std. 1028-1997) [IEEE 1997]
-
“A well-defined and disciplined process in which qualified personnel analyze a software product using a reading technique for the purpose of detecting defects.” [Laitenberger 2002]
Michael Fagan first described formal inspections in 1976. Since then, many have applied software inspections in diverse environments, refined the definition of formal inspections, or otherwise experimented with this process. A meeting in which inspectors agree on what bugs they have detected is an important aspect of the software inspection process. Over the last decade, researchers and practitioners have experimented with converting this meeting into a distributed and asynchronous (that is, virtual) meeting. A number of tools have been developed to support formal inspections, especially with virtual meetings.
Fagan originally defined inspections for source code. Nowadays, inspections are applied to artifacts developed throughout the software lifecycle. The development of variants and new techniques accompanied this extension of formal inspections. For example, Harlan Mills and his colleagues defined a reading technique based on stepwise abstraction to support the incorporation of formal inspections into Cleanroom software engineering. A number of researchers have developed other code-reading techniques for formal inspections.
This work has resulted in a number of definitions of formal inspections (e.g., [Ebanau and Strauss 1994], [Gilb and Graham 1993], [IEEE 1997], and [NASA 1993]). There have also been a number of surveys of formal inspections (e.g., [Ackerman, et. al.,1989], [Fagan 1986], [Laitenberger 2002], [Macdonald, et. al.,1995b], and [Wheeler, et. al.,1996]).
Researchers have been conducting controlled experiments with software inspections almost from their inception [Myers 1978]. Most of this empirical work has been devoted to quantifying the costs and benefits of different inspection techniques in different environments, not in validating theories about what drives inspections:
The majority of work in …academic empirical research into software inspections… has been laboratory based, with much of it concerned with testing new techniques rather than aimed at understanding or developing theory to explain phenomena. [Jeffrey and Scott 2002]
(Sauer, et. al.,[2002] is an exception.) This empirical work shows that formal inspections are a cost-effective method for removing bugs.
§ Cleanroom Software Engineering.
Two types of reviews are defined for Cleanroom – development and verification reviews.
A verification review “focus[es] on correctness and completeness of work products through a formal verification process. These verifications are usually carried out through verbal assertions, with the designer articulating the reasoning required to show that function-based correctness conditions are met.”
“Cleanroom correctness verification permits additional technical rigor and precision in inspections and reviews. Beyond local checklists that may be used, a Cleanroom review of design and code artifacts employs reasoning based on function theory: A program (the code) implements a function (the specification). The purpose of a Cleanroom review is to verify that the correctness of the function specification has been preserved in the implementation. Code is never reviewed in a vacuum; it is always reviewed against the function specification it implements.
Cleanroom specification and design produce artifacts with built-in traceability. Peer review is employed at each step in box structure specification and design. Every work product is reviewed, and every team member is responsible for the correctness of every work product. Ultimate successes are regarded as team practices, and team accountability for correctness results in an extremely effective approach to defect prevention.”
§ Nine Best Practices: The Software Management Framework. 3 June 1999.
The Navy’s Software Program Manager’s Network (SPMN) convened a group of experts in 1995 at the Airlie Center outside Warrenton, Virginia (the Airlie Council). The Airlie Council originally identified nine best practices, of which formal inspections was one. The council justified software inspections based on the following observations:
“Rework done to fix defects not found when the defect was introduced accounts for 40% to 50% of the software development budget on large projects. Since the cost of fixing these defects increases dramatically as the software matures, finding and repairing the defects before they are absorbed by the next phase is critical to the success of the project. Formal inspections are one solution to finding defects before they move into the system.”
§ Capability Maturity Model Integration (CMMI), Version 1.1, Software Engineering Institute, CMU/SEI-2002-TR-011, TR-012, March 2002.
The Integrated Capability Maturity Model (CMMI) consists of five maturity levels: (1) Initial, (2) Managed, (3) Defined, (4) Quantitatively Managed, and (5) Optimizing. In the staged representation, verification is a Process Area needed for Level 3 and higher. “Perform peer reviews” is the second specific goal for the Verification Process Area.
“The term ‘peer review’ is used in the CMMI Product Suite instead of the term ‘work product inspection’. Essentially, these terms mean the same thing. A peer review is the review of work products performed by peers during development of the work products to identify defects for removal.”
Specific practices to accomplish peer reviews consist of:
· SP 2.1: Prepare for peer reviews
· SP 2.2: Conduct peer reviews:
1. Perform the assigned roles in the peer review
2. Identify and document defects and other issues in the work product
3. Record the results of the peer review, including the action items
4. Collect peer review data
5. Identify action items and communicate the issues to relevant stakeholders
6. Conduct an additional peer review if the defined criteria indicate the need
7. Ensure that the exit criteria for the peer review are satisfied
· SP 2.3: Analyze peer review data
§ Turner, R., “Implementation of Best Practices in US Department of Defense Software-Intensive Systems”, Dissertation, George Washington University, January, 2002
Dr. Richard Turner, under the sponsorship of the Software-Intensive Systems Office of the Deputy Under Secretary of Defense (at the time, Science and Technology), surveyed 14 military software centers. Seven responded, with data on 150 software acquisition programs (53% Air Force, 26% Army, 21% Navy). Turner found that 65% of these programs had partially implemented formal inspections, while 20% had full implementations. The table shows the perceived effectiveness of formal inspections, according to these survey respondents.
Table 1: Turner Survey Results
Perceived Effectiveness of Formal Inspections |
Highly Effective |
21% |
Very Effective |
36% |
Moderately Effective |
14% |
Somewhat Effective |
21% |
Not Effective |
7% |
GLOSSARY
ADR |
|
Airlie Council |
A group of experts convened by the Navy’s Software Program Manager’s Network (SPMN) in 1995 that established/identified nine best practices. These practices have been augmented with other practices since 1995, and in current literature are referenced as the original Airlie best practices.
|
Best Practice |
A documented practice aimed at lowering an identified risk in a system acquisition and is required or recommended by a bona fide DoD, industry, or academic source.
- [Turner, 2002]
Methodologies and tools that consistently yield productivity and quality results when implemented in a minimum of 10 organizations and 50 software projects, and is asserted by those who use it to have been beneficial in all or most of the projects.
- Jones, Software Assessments, Benchmarks, and Best Practices, 2000
|
CMM |
|
CMMI |
|
FTArm |
|
Gold Practice |
|
SPMN |
|
There is a large body of literature reporting experiences with software inspections, including quantitative results. Laitenberger [2002] provides a survey of the literature. The next two tables summarize his analysis of a sample of 24 articles from the literature.
Defect Detection Effectiveness of Inspections [Laitenberger 2002]
Reference |
Environment |
Result |
[Fagan 1976], [Fagan 1986] |
Aetna Life Casualty |
38 defects from 46 detected |
IBM Respond, United Kingdom |
93% of all defects detected by inspections |
Standard Bank of South Africa |
Over 50% of all defects detected by inspections |
[Weller 1992] |
Bull HN Information Systems |
70% of all defects detected by inspection |
[Grady and van Slack 1994] |
Hewlett-Packard |
60%-70% of all defects detected by inspection |
[Shirey 1992] |
|
60%-70% of all defects detected by inspections |
[Barnard and Price 1994] |
AT&T Bell Laboratories |
30%-75% of all defects detected by inspections |
[McGibbon 1996] |
Cardiac Pacemakers, Inc. |
70% to 90% of all defects detected by inspections |
[Collofello and Woodfield 1989] |
Large real time software project |
Defect detection effectiveness is 54% for design inspection, 64% for code inspection, and 38% for testing. |
[Kitchenham, et. al.,1986] |
ICL |
57.7% of all defects found by code inspection |
[Franz and Shih 1994] |
Hewlett-Packard |
19% of all defects found by inspection |
[Gately 1999] |
Raytheon Systems Company |
Average number of defects found by inspection is 18.2 |
[Conradi, et. al.,1999] |
Ericsson |
Average number of defects found by inspections is 3.41 |
Average Effort To Detect Defects [Laitenberger 2002]
Reference |
Environment |
Result |
[Kaner 1998] |
Jet Propulsion Laboratory |
Ratio of the cost of fixing defects during inspection to fixing them during formal testing ranges from 1:10 to 1:34 |
[Remus 1984] |
IBM Santa Teresa Lab |
Ratio of the cost of fixing defects during inspection to fixing them during formal testing is 1:20 |
[Kan 1995] |
IBM Rochester Lab |
Ratio of the cost of fixing defects during inspection to fixing them during formal testing is 1:13 |
[Ackerman, et. al.,1989] |
Different projects |
0.58 – 5 hours per defect found (lower than in testing) |
[Weller 1992] |
Bull HN Information Systems |
1.43 hours per defect in inspection and 6 hours in testing |
[Collofello and Woodfield 1989] |
Large real time software project |
7.5 hours per defect for design inspection, 6.3 hours per defect for code inspection, and 11.6 hours per defect for testing |
[Franz and Shih 1994] |
Hewlett-Packard |
1 hour per defect for inspection and 6 hours per defect for testing |
[Kelly, et. al.,1992] |
Jet Propulsion Laboratory |
1.75 hours per defect of design inspection, 1.46 hours per defect for code inspection, 17 hours per defect for testing |
[Kitchenham, et. al.,1986] |
ICL |
1.58 hours per defect in design inspection |
[Gilb and Graham 1993] |
Applicon |
0.9 hours to find and fix a major defect |
[Bourgeois 1996] |
Lockheed Martin Western Development Labs |
1.3 hours per defect found and 1.4 – 1.8 hours per defect found and fixed |
There are many controlled experiments with software inspections. Porter and Johnson (1997) provide a good example. They compare and contrast two influential controlled experiments with software inspections in the next table. The goal of each experiment was to investigate the effect of the inspection meeting on the formal inspection process. They found that meeting-based software inspections are neither more nor less effective than non-meeting based software reviews in finding defects. Non-meeting based reviews, however, raise more false positive issues and more duplicate issues.
Overview of Two Controlled Experiments
Experiment Location |
Participants |
Design |
Defect Detection Effectiveness |
University of Hawaii |
72 undergraduate students (24 groups of 3) |
Each group performed a real and nominal review (no group meeting occurred in the nominal review) |
45% |
University of Maryland |
21 Graduate students, 27 professional software developers (16 groups of 3) |
Partial factorial design. Each group performed 2 of 3 types of review. |
30% |
|
DACS Gold Practice Initiative
ROI Dashboard
