SOFTWARE ACQUISITION GOLD PRACTICE^TM

GOAL-QUESTION-METRIC (GQM) APPROACH

FOCUS AREA: process - Measurement

Definition and Summary:  A goal-driven method for developing and maintaining a meaningful metrics program that is based on three levels, Goals, Questions and Metrics. The approach uses metrics to improve the software development process (and its resulting software products) while maintaining alignment with organization business and technical goals.

The consensus of a recent GAO Report on Defense Acquisitions (GAO-04-393), published in 2004, was that stronger management practices are needed to improve DOD’s software-intensive weapons programs.  The report goes on to state that the success (quality software product, on time and within budget) of leading software companies is attributable to creating a manageable evolutionary development environment, using disciplined processes, and collecting and analyzing meaningful metrics to measure progress. This GQM Gold Practice^TM provides practical guidance for implementing the GQM approach to measurement in order to establish and use meaningful metrics, (which are well aligned with the goals of the organization), to help mitigate risk, track progress and support improvement initiatives.

.

GQM is a top-down approach to establish a goal-driven measurement system for software development, in that the team starts with organizational goals, defines measurement goals, poses questions to address the goals, and identifies metrics that provide answers to the questions.  GQM defines a measurement model on three levels as illustrated in the figure below:

GQM can be applied to all life-cycle products, processes, and resources, and is well aligned with the organizational environment.  The approach was developed by Dr. Victor Basili and colleagues during the 1980s in conjunction with their work at the NASA Software Engineering Laboratory (SEL), was refined during the 1990s, and now, serves as the foundation framework for many measurement initiatives.  It is an appropriate means to achieve reliable empirical data and knowledge about the organization’s software practices to drive systematic process improvement.  In that context, it is particularly useful for the following purposes:

§         Understanding and baselining an organization’s software practices

§         Guiding and monitoring software processes

§         Assessing new software engineering technologies

§         Evaluating and certifying improvement activities

Although the primary focus of the practice is goal-driven metrics definition, the approach also  addresses data collection, analysis and interpretation and packaging experiences for use in future initiatives. These activities are equally as important as defining the metrics because they guide how the data is actually used.

Some effective practices for implementing GQM, discussed in detail in the document, are:

§         Get the right people(all levels of developers) involved in the GQM process

§         Set explicit measurement goals and state them explicitly

§         Thoroughly plan the measurement program and document it (explicit and operational definitions)

§         Don’t create false measurement goals

§         Acquire implicit quality models from the team 

§         Consider context

§         Derive appropriate metrics

§         Stay focused on goals when analyzing data

§         Let the data be interpreted by the people involved

§         Integrate the measurement activities with regular project activities

§         Do not use measurements for other purposes

§         Secure management commitment to support measurement results

§         Establish an infrastructure to support the measurement program

§         Ensure  that measurement is viewed  as a tool, not the end goal

§         Get training in GQM before going forward

ANTICIPATED BENEFITS OF IMPLEMENTATION:

Some benefits of this practice that were identified in the PERFECT project [PERFECT,1997] include:

§        Achievement of improvement goals

§        Financial gains

§        Increased capability to perform improvement initiatives

§        Improved group synergy

§        Increased quality awareness and Quality Assurance (QA) involvement

Description of the practice:

Summary DESCRIPTION

The Goal-Question-Metric (GQM) Approach is a paradigm for developing and maintaining a meaningful metrics program that supports:

·         Metric alignment with organization business and technical goals

·         Software process improvement

·         Risk management

·         Product quality improvement

This Gold Practice document describes the basic concepts of the GQM paradigm and addresses key aspects of its implementation.  The practice focuses on measurement of goals and supports interpretation of measurement results relative to those goals.  Although GQM originated as a measurement methodology for software development, the basic concepts of GQM can be used anywhere that effective metrics are needed to assess satisfaction of goals.

GQM can be applied to all life-cycle products, processes, and resources, and is well aligned with the organizational environment.  The approach was developed by Dr. Victor Basili and colleagues during the 1980s in conjunction with their work at the NASA Software Engineering Laboratory (SEL).  It was refined during the 1990s, and now, serves as a foundation framework for many measurement initiatives.

Benefits of GQM

Although the direct benefit of GQM is establishing meaningful metrics, when GQM is applied within the context of systematic process improvement, it is particularly useful for the following purposes:

§         Understanding and baselining an organization’s software practices

§         Guiding and monitoring software processes

§         Assessing new software engineering technologies

§         Evaluating and certifying improvement activities

In Europe, between 1993 and 1996, several industries formed a consortium with the goal of initiating measurement-based software process improvement.  That initiative, called project “PERFECT” (Measurement Based Improvement of Software Processes), noted the following benefits from using the GQM approach [PERFECT 1997]:

§         Achievement of improvement goals

§         Increased quality awareness and Quality Assurance (QA) involvement

§         Increased capability to perform improvement initiatives

§         Improved group synergy

§         Financial gains

GQM Basics

The open literature typically describes GQM in terms of a six-step process where the first three steps are about using business goals to drive the identification of the right metrics and the last three steps are about gathering the measurement data and making effective use of the measurement results to drive decision making and improvements.  In a recent seminar, [Basili 2005] Basili described his six-step GQM process as follows:

1.       Develop a set of corporate, division and project business goals and associated measurement goals for productivity and quality

2.       Generate questions (based on models) that define those goals as completely as possible in a quantifiable way

3.       Specify the measures needed to be collected to answer those questions and track process and product conformance to the goals

4.       Develop mechanisms for data collection

5.       Collect, validate and analyze the data in real time to provide feedback to projects for corrective action

6.       Analyze the data in a postmortem fashion to assess conformance to the goals and to make recommendations for future improvements

As illustrated in the figure below, GQM begins by identifying measurement goals (conceptual level) that support (are aligned with) business goals.  The team (project managers, development team, customers, or other stakeholders) then poses questions (operational level) to further clarify and refine the goals as well as capture the variation of understanding of the goals that exists among the stakeholders with respect to their notions of quality and the environment that will impact goal attainment.  The team then identifies metrics that will provide answers to the questions (Quantitative level).  What distinguishes GQM from other measurement paradigms is the hierarchical tree structure used to maintain the relationships among goals, questions and metrics.

Once appropriate metrics are identified, the last three steps of the GQM process address how to implement the metrics program in a way that ensures the focus will remain on goal attainment.  Basili and other GQM experts stress the importance of planning data collection mechanisms and planning how the resulting measurement data should to be organized and presented in order to maximize its value to the stakeholders who will interpret the results in relation to the goals.  The literature notes that when measurement programs fail, the primary cause of failure is often a lack of attention to how the measurement results will be used.

Implementing GQM

GQM can be applied at the strategic level of an organization, or the project level, or at both levels concurrently.  When applied at the strategic level, the measurement data consists of results from targeted pilot projects providing the feedback to strategic level planners for decision making relative to product and process strategies.

Organizations often use a phased approach for implementing GQM that integrates GQM-related activities (derived from Basili’s GQM process) with project planning and management activities.  The phases are GQM Planning, Definition, Data Collection and Interpretation.  A more extensive explanation of the phased approach, and how it relates to Basili’s six step process, is provided in the body of this document.

Some effective practices for implementing GQM, described in detail in the body of this document, are:

§         Get the right people(at all levels of developers) involved in the GQM process

§         Set and state explicit measurement goals and state them explicitly

§         Thoroughly plan the measurement program and document it (explicit and operational definitions)

§         Don’t create false measurement goals

§         Acquire implicit quality models from the team

§         Consider context

§         Derive appropriate metrics

§         Stay focused on goals when analyzing data

§         Let the data be interpreted by the people involved

§         Integrate the measurement activities with regular project activities

§         Do not use measurement for other purposes

§         Secure management commitment to support measurement results

§         Establish an infrastructure to support the measurement program

§         Ensure  that measurement is viewed  as a tool, not the end goal

§         Get training in GQM before going forward

DetailED description

The Goal-Question-Metric (GQM) practice focuses on following the GQM paradigm for establishing a metrics program to support software development and maintenance.  Organizations typically implement GQM as part of an overall software process improvement initiative, but it is not limited to that role.  The basic concepts of GQM can be used anywhere that effective metrics are needed to assess satisfaction of goals.  It can even be used by individual members of a project team to focus their work and assess their progress toward achieving their specific goals.

GQM can be applied to all life-cycle products, processes, and resources, and is well aligned with the organizational environment.  The approach was developed by Dr. Victor Basili and colleagues during the 1980s in conjunction with their work at the NASA Software Engineering Laboratory (SEL).  It was refined during the 1990s, and now, serves as a foundation framework for many measurement initiatives.

GQM Basics

In one of his most recent professional development seminars [Basili 2005], Basili defines the GQM process as follows:

1.       Develop a set of corporate, division and project business goals and associated measurement goals for productivity and quality

2.       Generate questions (based on models) that define those goals as completely as possible in a quantifiable way

3.       Specify the measures needed to be collected to answer those questions and track process and product conformance to the goals

4.       Develop mechanisms for data collection

5.       Collect, validate and analyze the data in real time to provide feedback to projects for corrective action

6.       Analyze the data in a postmortem fashion to assess conformance to the goals and to make recommendations for future improvements

The first three steps are about establishing a goal-driven measurement program where the identification of goals triggers the identification of appropriate metrics.  The remaining steps are about collecting and using the measurement results for better decision making.  Later in this document, we will discuss how the process is implemented as a phased approach that is integrated with project planning and project management, and show how the process can be implemented at various levels within the organization.

Figure 1 presents a high-level visual synopsis of the definition phase of the GQM paradigm, illustrating the outputs of first three steps of Basili’s GQM process, the hierarchy of goals, questions, and, ultimately, meaningful metrics.

Figure 1: Basic GQM Hierarchy

This part (the first three steps) of Basili’s GQM process, often called the definition phase of GQM, provides the process structure for migrating from concepts to meaningful metrics that, when implemented, quantify the goals and provide meaningful data for decision-making.  Goals identify what we want to accomplish; questions, when answered, tell us whether we are meeting the goals or help us understand how to interpret them; and the metrics identify the measurements that are needed to answer the questions and quantify the goal.

As illustrated in Figure 1, the mapping among goals, questions and metrics is not one-to-one.  A single measurement goal may apply to multiple business goals and vice versa; for each goal, there can be several questions and the same question can be linked to multiple goals as appropriate.  For each question, there can be multiple metrics, and some metrics may be applicable to more than one question.  Adherence to, and preservation of, this hierarchical structure helps ensure that the measurement program focuses on the right metrics and that we avoid extra work associated with collecting metrics that are not really needed.

The remaining steps of Basili’s GQM process (relating to data collection and analysis for decision-making and for future recommendations) are actionable only when appropriate definition of metrics has occurred.  Goal-driven metric definition, using the goal-question-metric process, is what separates GQM from other measurement methodologies.  Thus, this document emphasizes the initial steps of Basili’s GQM process because they provide the foundation for the remaining steps of the process.

A primary tenet of GQM, not usually evident in illustrations of the GQM paradigm, is that stakeholders need to be involved throughout the process in order for it to be successful.  Basili, and others [Basili 2005, PERFECT 1997], advocate for planning the implementation of GQM to ensure that those with a stake in any part of the process actually participate to ensure their knowledge is considered, that they understand their contribution (role) in the process, and to promote their buy in and acceptance of the measurement program.  Those who implement GQM may use a variety of approaches to ensure the appropriate level of participation.  The key message is that the measurement program should be planned and implemented from within the organization or project, rather than be outsourced.  However, the experts agree that it is helpful to have a consultant (GQM expert) work with the team or organization in the initial stages to ensure that the principles of GQM are implemented, and to transition those principles to key people within the organization.

GQM Process Details

This section provides details about each of the six steps of Basili’s GQM process.

Step 1 - Establishing Goals

In his recent DACS course [Basili 2005] Basili lumps all goal setting into the first step of his GQM process description because he is teaching organizational leaders what the organization must do to build core competencies in software development (applying GQM at a strategic level).  Others [PERFECT 1997] suggest that the GQM process begins with establishing measurement goals using previously defined business goals as a driver.  The essence of the step is:

·         There are two types of goals --- business goals and measurement goals

·         Business goals drive the identification of measurement goals

Sometimes, it is difficult to distinguish between a business goal and a measurement goal; they may not always be mutually exclusive.  What is important is that the driving goals originate from the group or organization which is responsible for the broader scope of a software initiative, the business environment in which the initiative occurs, rather than from within a particular project.  It is not important whether the business goals are developed under the umbrella of GQM, or as a function of strategic planning.  Business goals must exist; they must be identified and be the focus for establishing the measurement goals.  Without them, the measurement program has no focus.  Without this alignment, it is unlikely that implementing the rest of GQM will have a significant impact.  When business goals exist, then multiple projects or sub-groups within an organization have a basis for identifying the measurement goals relating to their role or scope of influence within the organization.

The goals at the top of the GQM tree (see Figure 1) are the measurement goals that are the outcome of step 1 of the GQM process.  They are conceptual, not quantitative.  They are quantified by their linkage to questions and metrics as noted in the mapping.  Some examples are provided later in this document to illustrate this point (see Figure 3).

Basili and his followers express GQM goals (measurement goals) using five facets of information to define what the measurement should accomplish in precise terms.  Each GQM goal statement explicitly contains these facets:

·         Object:  The product or process under study; e.g., testing phase or a subsystem of the end product

·         Purpose:  Motivation behind the goal (why); e.g., better understanding, better guidance, control, prediction, improvement

·         Focus:  The quality attribute of the object under study (what); e.g., reliability, effort, error slippage

·         Viewpoint:  Perspective of the goal (who’s viewpoint); e.g., project manager, developer, customer, project team

·         Environment:  Context or scope of the measurement program; e.g., project X or division B

Figure 2 illustrates the refinement of a measurement concept into a GQM goal statement containing these facets of information.  Similar examples can be found in [van Latum 1998], [PERFECT 1997], [van Solingen 1999b] and other referenced articles.

Figure 2: The five Facets of a GQM Goal Statement

Some implementations of GQM use a table-formatted template for goal definition (see Table 1).  The left column identifies the five facets of information and remains constant on the form; the right column changes for each goal defined [van Solingen 1999b].

Table 1:  GQM Goal Definition Template and Example

Step 2 – Generating Questions

The purpose of Basili’s Step 2 is to clarify and refine the measurement goals, moving from a conceptual level to an operational level by posing questions.  By answering the questions, one should be able to conclude whether a goal is reached.  Questions help identify interpretations of the goal that may exist among the stakeholders as well as constraints imposed by the environment.  Typically, at the project level (or perhaps for a group of related projects), conceptual measurement goals are identified relating to product quality, process, resources, or the environment.  The project team then identifies questions that the team (individually or collectively) feels should be asked to capture various perspectives of the goal and address whether the goal is being met.  These questions would typically get at all of the nuances and perceptions relating to the goal, addressing both perceptions of quality and the context or environment in which the object will evolve.  This is essentially a process of stakeholders converging on a common understanding and interpretation of the goal at the appropriate level of abstraction.  In other words, the individual project managers and software engineers provide their perspective of what the goal means in the given environment.  They do this by posing questions and responding to them with their metrics.  Figure 3 provides a simplified example illustrating some of the questions that might emerge for the specified goals.

On the surface this step may appear to be trivial, and for some goals that may be the case, but GQM experts [Basili 2005, van Solingen 1999b, PERFECT, 1997] and implementers have found that getting the right level of abstraction for GQM questions can be difficult.  If questions are too abstract, the relationship between the metrics and the question may be muddied.  If they are too detailed, it becomes more difficult to get a clear interpretation of the goal.  In many instances, particularly where the purpose of the goal is to understand or characterize the process or product, questions may need to be broken out into many sub-questions to drive appropriate identification of metrics.  The implementing organization may tailor this step of the process as needed to ensure that the level of questioning is sufficient to drive the identification of the right metrics.  Some excellent examples of ensuring the appropriate level of questioning are provided in the Case Studies section of van Solingen’s book [van Solingen 1999b].

In some implementations [van Solingen 1999b], a GQM team interviews the stakeholders (members of the project team) individually to capture their perspectives of the goal (their questions) and to have them formulate expected answers as hypotheses.  These hypotheses make explicit the current knowledge that is in the minds of the team members thus, forming a baseline for later analysis of metrics.  Comparing actual results with these hypotheses during the interpretation phase of GQM increases the learning effect from measurement.

Step 3 – Specifying the Measures

Step 3 is about examining how the questions could be answered, moving from the qualitative (or operational level) to a quantitative level.  Once goals are refined into a list of questions (GQM process step 2), metrics need to be defined that provide all the quantitative information to answer the questions in a satisfactory way.  Stakeholders, those directly involved with the object of the goal, must be directly involved in the metric identification step as well as the Question step.  Direct involvement of these stakeholders minimizes ambiguities and false assumptions and contributes to the overall consistency and completeness of the metrics identification.

In this context, the term metric is loosely defined; it can mean a base measure, a derived measure, a composite or aggregate of measures, or, what some would call, an indicator.  The level of definition depends on the scope of the goal and the environment of the GQM implementation.  Further details are provided later in the section describing GQM implementation.  Figure 3 provides a simple example of how GQM would derive metrics for two goals.  It shows some of the essential questions that would emerge, and some of the metrics that might be identified. 

The diagram is intentionally oversimplified in order to convey the notion of how one gets from a conceptual level goal to the right quantitative data that renders the goal measurable. It also conveys the multiple mapping of metrics to questions and questions to goals. 

Figure 3: Sample of GQM Definition Phase

A refined GQM goal statement for Goal 1 in Figure 3 might read as follows:

Analyze:                         Change request processing

For the purpose of:         Improvement

With respect to:               CR processing cycle time

From the viewpoint of:    The project team

In the context of:             The current project timeframe

Note a distinction between the metrics that are defined and the data elements that support them.  The metric is at a more abstract level than the actual data items and measurements that need to be collected to provide the correct data for preparing the metric.

Step 4 – Preparing for Data Collection

Once the metrics are identified, one can determine what data items are needed to support those metrics, and how those items will be collected.  The metric provides insight regarding how the data needs to be organized in order to be meaningful to the viewer/recipient of the information.  A significant amount of planning is necessary to provide the detailed procedures for data collection that support the identified metrics.  Most projects accomplish this detailed planning by preparing a Measurement Plan that includes at least the following [van Solingen 1999b]:

·         Formal definitions of direct measurements

·         Textual descriptions of direct measurements

·         All possible outcomes (values) of the direct measurements

·         The person (role) that collects each direct measurement

·         The moment in time (or frequency) when the direct measurement should be collected

·         The medium (tool or form) that should be used for collecting the measurement

The plan also defines and describes all types of data collection forms and automated data collection tools that should be used.  It addresses the question of how the data can be collected most efficiently and effectively and to whom it should be delivered.

Referring to the example in Figure 3 of Change Request processing time, one cannot assume that all stakeholders have the same understanding of what constitutes CR processing time.  Does it begin when the CR is first documented, or after it is diagnosed and categorized for action? When is processing considered done?  Which CRs are included for averaging --- only closed out CRs, or also those on hold?  The Measurement Plan anticipates and addresses such questions.

Once a plan is developed, the measurement procedures need to be tested and validated before implementing the program.  Exercising the forms and procedures during a trial period will reveal flaws that can subsequently be corrected before full-scale implementation of the measurement program begins, or before adding the new procedures and data to an existing program [van Solingen 1999b].

In addition, it is important to train individuals involved in data collection to ensure that they understand why the data is needed, how it is going to be used and how their action contributes to the overall validity of the data collection process.

Step 5 – Collecting, Validating and Analyzing the Data for Decision Making

This step presumes that data collection follows the procedures pre-defined in the Measurement Plan.  It is often a continuous (or periodic) process rather than a one-time activity.  However, it is only a means to an end.  Data collection is a worthless process if one does not do anything with the data.  The focus needs to be on preparing the data for optimal usage.  Regardless of the collection medium, data needs to be validated before it is used for analysis.

Recently a Project Manager told the DACS a story about a group of developers responsible for maintaining an application that was particularly volatile.  Each time they fixed a problem they entered a “1” in the Problem Report (PR) form field that asked for actual time spent on the fix.  From their perspective, this was an irrelevant field because nothing was ever done with the data.  They were very busy with a backlog of PRs and it took time to accurately estimate the time to fix each problem.  Their Manager, who knew that many PRs required extensive time to fix, showed them a request for additional staff that had been denied based on a report of staff utilization that showed a graph of average time per PR, clearly demonstrating that the current staff was underutilized.  Once the staff understood the significance of that data item, what decisions it affected, and that it was not measuring their personal performance, they started entering their estimates of time appropriately and the manager began to get valid data for his management purposes.

Automation can assist, but it cannot replace all forms of data collection and validation.  The key is to minimize the overhead imposed on the people who are required to provide data while ensuring that they understand the significance of their data collection effort.  Validation consists of checking the data collected for correctness, completeness and consistency.  Completeness is the most significant data collection problem.  What does one do with a form that is only half completed?  How does one ensure that all instances of data are actually captured?  These, and similar questions, need to be addressed in planning the measurement program so that there is a proper course of action available to those people tasked with validating.

As the story about the maintenance staff showed, leaders need to reinforce the purpose and value of data collection to promote better data collection quality.  A significant part of the validation process is checking validity as close to the data source as possible and taking immediate corrective action for invalid data.

Once validated, it is important to store the measurement data in such a way that it can be accessed for varying analysis and reporting purposes.  Because of the sheer volume of measurement data needed for all but the smallest of projects, it is useful to develop and use a measurement support system that contains a database for the storage of the metric data and tools for analysis (spreadsheets) and presentation.  Flexibility and accessibility are the most important features of such a system.

Analysis is about organizing the data and preparing the metrics for presentation to the stakeholders to address the questions pertaining to the measurement goal.  Typically, a GQM team, together with the project team, develops an analysis plan as soon as they know what metrics are needed.  The plan details how the data should be organized, what presentation formats are needed, who will review the data and when.  Developing the analysis plan often helps with decisions about data collection.  Basili [Basili 2005] uses the term analysis to mean both analysis and interpretation, but some implementers of GQM make a distinction between the two terms, primarily, to assert that the analysis can be done by a GQM team (measurement expert), but the interpretation must be done by the project team who are the owners of the measurement goals.

Some form of feedback is required to communicate measurement results to the appropriate stakeholders.  These sessions are focused on the measurement goal and reviewing the measurement results to answer the questions posed in step 2 of the GQM process.  The project team can then decide on corrective action when progress toward goals is not deemed adequate.

Analysis and interpretation is an iterative step typically integrated with the progress reporting cycle of a project.

Step 6 – Analyzing the Data for Goal Attainment and Learning

The last step in Basili’s GQM process is about looking at the measurement results in a post-mortem fashion to assess goal attainment and also to determine lessons learned and what might be valuable to pass on to future projects.

When GQM is implemented to support an organization-wide improvement process, the experiences and lessons learned from each implementation are packaged in the form of policies, procedures and best practices, to support future projects and improvement initiatives and to help the organization achieve greater leverage from its measurement program.

Implementing GQM

Describing GQM in terms of a six-step process (see previous section titled GQM Basics) tends to convey it as a strictly sequential process, but that is not the case when GQM is actually implemented.  Van Solingen [van Solingen 1999b] and others assert that implementation of GQM should be viewed in terms of phases of activities that are integrated with project planning and management and have dependency relationships with each other.  Although they embody Basili’s GQM process, the focus of the phases is on the planning and implementation details needed to make GQM a reality within an organization.  Solingen’s phases are named as follows and their relationship to each other is illustrated in Figure 5:

·         GQM Planning:  deals with the logistics of implementing GQM and key plans that need to be documented.  This phase, therefore, touches steps 1 -5 of Basili’s process.

·         Definition:  focuses on using the GQM method to derive meaningful metrics.  It encompasses the first three steps of Basili’s six-step process.

·         Data Collection: addresses planning for and executing data collection activities to obtain the data needed for the defined metrics.  This phase addresses steps 4 and 5 of Basili’s process.

·         Interpretation:  Addresses preparing the measurement data in ways that facilitate analysis and interpretation of results relative to the pre-defined goals and actually doing the analysis and interpretation.  This phase implements steps 5 and 6 of Basili’s process.

The following paragraphs provide further details about van Solingen’s GQM implementation phases.

GQM Planning and Definition Phases

It takes a significant amount of planning to ensure that data collection and analysis are in fact coordinated with and actually address the defined measurement goals.  Therefore, GQM needs to be integrated with, rather than separated from, project planning.  The overall Project Plan specifies, that the GQM method will be utilized, and that GQM planning activities will occur.  The GQM Planning phase oversees the implementation of GQM within the context of the project.  It does not precede all other phases, but rather, is intermingled with them.  Initially, the GQM planning phase establishes how the Definition phase will be implemented and who will be involved; then Definition occurs. Then GQM planning uses the output of the definition phase as a basis for the planning of data collection mechanisms and for analysis and interpretation.  GQM planning thus provides documentation that serves as a guide for other phases of GQM, and provides the necessary integration with project planning.  That is why the Planning phase block in Figure 5 stretches across the remaining phases.

Artifacts of the GQM Planning phase include, but are not limited to, the following documents:

(1)     GQM Plan:  As illustrated in Figure 5, the Definition phase consists of identifying measurement goals, posing questions, and identifying appropriate metrics.  The GQM plan is a document that contains each measurement goal and its corresponding breakdown into questions and metrics, thus preserving the relationships of goals to questions to metrics.  This document provides the foundation for progressing through the other phases of GQM, just as a requirements document provides the foundation for the activities of a development effort.  The GQM plan contains the output from the first three steps of Basili’s GQM process [Basili 2005] described earlier.

(2)     Measurement Plan:  The GQM planning phase, additionally, includes developing a specific Measurement Plan that defines the actual base measures needed to generate the metrics defined in the GQM plan and establishes the detailed procedures for collecting the measurement data and generating the identified metrics.  It should address what data is collected, how it is to be collected, by whom, and when.  The Measurement Plan includes a description of the media needed to report, collect, and validate data.  It serves to guide the activity of the Data Collection phase.  Developing this plan is part of step 3 of Basili’s GQM process.

(3)     Analysis Plan:  The GQM Planning phase also encompasses developing an Analysis Plan that focuses on how to analyze, aggregate, and present the collected measurement data in ways that are meaningful to the stakeholders.  What the project manager needs to see is different from what the Vice President needs to see and from what the Quality Assurance manager needs to see.  The plan sets the stage for the Interpretation phase of GQM by providing guidance on how the information needs to be organized to facilitate its use and ensure that the focus remains on the goals.  Basili’s process description does not specifically mention preparing this plan, but he implies (or perhaps assumes) that one makes a plan before one does analysis.  It is likely to be considered part of Basili’s steps 4 or 5 that address developing data collection mechanisms and analyzing the data.  The important point is that the planning for analysis and interpretation should be done prior to data collection so that it is clearly tied to the goals and does not fluctuate depending on the actual data values collected.  The Analysis plan defines the appropriate level of abstraction for the presentation of the data.

Data Collection Phase

The Data Collection phase includes collecting the measurement data according to the Measurement Plan and preparing it for analysis in accordance with the Analysis Plan.  This phase is covered by Basili’s GQM process step 5.  During collection, data needs to be validated by checking for such things as data completeness, timeliness, and accuracy.  Once validated, the data can be presented to the team and other appropriate recipients for interpretation.  A key activity of this phase is to store the data in such a way that it facilitates access by the project team and others both during and after a project.  Most GQM implementations develop a metric database for this purpose.

Interpretation Phase

After the data is collected, validated and prepared for review, in accordance with the Analysis Plan, the Interpretation phase involves review of the measurement results by the stakeholders (typically the project team).  The team interprets the data in light of the questions asked and the ultimate goals defined.  Do the measurement results provide answers to the questions addressed?  To what extent has the goal been attained?  Typically, during the Interpretation phase the team determines not only the extent to which the actual measurements address their corresponding questions and the extent to which the measurement goals were attained, but also what improvements can be made to the measurement program itself.  Success in this phase is premised on the principle that the measurement program must address the interests of those providing the data and must be based on the project team’s knowledge.  These feedback sessions typically lead to updates to the Measurement and Analysis Plans, and sometimes, to the alteration or addition of measurement goals.  Keep in mind that measurement goals may be related to the product as well as the process or resources.  The visibility of information provided by the presentation format of the measurement results makes it easier for the project team to take corrective action during the project, since they are monitoring progress on a periodic basis.  They use the quantitative information for fact-based decision-making.  Basili includes this analysis and interpretation activity as part of his step 5 of the GQM process.

This phase also addresses making decisions about the preservation of measurement results for use in the future and lessons learned.  The GQM team (or persons with overall responsibility for the measurement program) determines how such data should be packaged to provide optimum accessibility and use for future efforts.  This activity is consistent with the last step of Basili’s GQM process.

Applying GQM at Varying Levels Within an Organization

The GQM methodology is quite generic and thus its scope of implementation can range from somewhat narrow (an individual using GQM to achieve desired goals) to very broad, such as when it is used in strategic planning.  The open literature delineates the GQM approach in terms of a six-step process, but the descriptions and boundaries (entry and exit points) of each step vary depending on the context of implementation, and the publication period within the body of literature.  Basili himself now defines the steps somewhat differently than he did in his original seminal article in 1984 [Basili 1984].  A review of several variations of the six-step process reveals that, irrespective of the step number assigned, or the different wording, each of the descriptions, collectively, contain the same essential concepts and assign the same relationships to activities of the overall process.

In the context of organizational process improvement, GQM has been applied concurrently both at a project level and the strategic level [PERFECT 1997].  This interplay of strategic versus project level GQM is illustrated in Figure 6.  Although the steps are not described in exactly the same manner, these steps are consistent with the steps outlined by Basili as noted earlier in this document.  The major distinction between strategic level GQM and project level GQM relates to the data collection phase (step 4 of both levels).  The strategic level implementation identifies pilots and projects as the source of data for strategic level metrics.  Therefore, strategic level data collection is dependent on the designated projects actually implementing GQM at their project level and packaging results to feed back to the strategic level implementation.

Figure 6:  GQM Process - Strategic vs. Project Level

When applied at a strategic level, GQM is still a six-step process, with the first three steps focusing on goal and metric identification and planning the measurement implementation, step 4 addressing data collection, and steps 5 and 6 addressing analysis and interpretation.  The objects of focus differ slightly at the strategic level from those at the project level.  Data collection, (strategic level, step 4) is about gathering the information, experiences and lessons learned from the multiple targeted projects, as the basis for the strategic level analysis phase.

For example, a software development organization may desire to implement formal inspections as a practice in all of its software development projects.  Before making it a policy, they would identify some target projects in which to test the formal inspection process.  The strategic level measurement goal may be characterizing the ROI from implementing formal inspections, and setting a specific threshold that would indicate that it would be beneficial to include the process as a fundamental practice for all projects.  They might identify a variety of projects for pilots to include both large and small projects and projects of short and long duration.  Each pilot project would then apply GQM to measure the impact of formal inspections in their particular project environment.  The organization would follow its plan for collecting and analyzing the measurement results from the projects and use those results to plan the roll out of formal inspections across all projects within the organization.

This is easier to see when we look at how these steps are aligned with the GQM phases defined earlier.  At both levels (strategic and project), the first three steps of the process entail goal alignment, goal identification, metric identification and development of the measurement plan.  These activities are consistent with what occurs in the Planning and Definition phases of a generic GQM implementation.  Step 4, at both levels, is implementing the data collection phase of GQM; however, the type of data collected and reported differs depending on the level.  Steps 5 and 6 at both levels address the Analysis and Interpretation phase of GQM.  At each level, the analysis is tailored to address the needs of the stakeholders at that level.  At both levels, steps 5 and 6 are addressing actual use of the data to assess the degree to which goals are being met, and determining how best to package and/or preserve the results for future or broader use.

The notion of packaging experiences is essentially documenting and sharing lessons learned, making the information available to future projects or for the next refinement of the improvement cycle.  This is analogous to Basili’s, process step 6 --- doing post-mortem data analysis.

Strategic level GQM analyzes the experiences from the individual projects in order to identify what improvements yield the biggest gain and under what circumstances.  This information is then packaged for use in future projects in the form of guidelines, policies and procedures and, perhaps, support tools provided to the projects.  This is difficult because future needs for measurement information are generally unknown.  The more precisely you can describe your actual experiences and their specific context, the greater your chances for adaptive reuse.

The notion of packaging experiences for adaptive reuse is captured in what Basili has coined an “Experience Factory” (see Figure 7).  It is an organizational concept established for institutionalizing the collective learning of the organization, the basis for their continual improvement.

Figure 7:  Concept of an Experience Factory

The concept was developed by Victor Basili and first implemented at the NASA Software Engineering Laboratory (SEL) in the late 1970s.  A separate group is formed (which might start out as the GQM team) that supports reuse of experience by developing, updating, and delivering experience packages to the project groups responsible for developing and maintaining software.  The Experience Factory team focuses on organizing and preserving the information and data to optimize its reuse, and on presenting solid information back to the project groups to guide them in future projects.

GQM and the Quality Improvement Paradigm

GQM is effective when implemented as part of a broader quality improvement initiative since one of the main purposes of measurement is improvement.

Figure 8 illustrates Basili’s six step GQM process overlaid on the six step process (1 - Characterize, 2 - Set Goals, 3 - Choose Process, 4 - Execute, 5 - Analyze, 6 - Package) of the Quality Improvement Paradigm (QIP), as described in the Perfect Handbook [PERFECT, 1997] and other sources [van Latum, 1998].  What this means is that, although the terminology may be slightly different depending on which perspective or focus one has, what is happening is essentially the same cycle of activities.  This is because it does not make sense to talk about improvement without metrics, and it does not make sense to implement a metrics program without a greater purpose, be it process improvement, risk mitigation, or product quality, all of which can be viewed as forms of improvement.  If one implements all phases of the GQM paradigm, one is seeking improvement.  Note, however, that this does not preclude implementing the definition phase of GQM outside of a formal improvement framework simply to assist with the identification of meaningful metrics.

In Figure 8, the boxes containing the GQM steps have intentionally been positioned across the QIP boundary lines to show an overlapping of Basili’s GQM steps with adjacent steps in the QIP process.

GQM plays an important role in software process improvement in that it helps in achieving reliable data about software projects and software development practices of the organization and provides the basic information necessary for organizational learning and guiding of software projects.

Figure 8:  GQM Integrated with the Quality Improvement Paradigm

Key Practices of the GQM Approach:

Based on the work of Basili and others [Basili 1984,1994, 1999, 2002, 2005; van Latum 1998; van Solingen 1999; Lavazza 2000], the practices listed herein are a combination of the main tenets of GQM, success factors identified by developers in GQM feedback sessions and generic practices that are applicable to the implementation of any measurement methodology.

1 - Get the right people involved in the GQM process:  Although GQM is often described as a top down approach, it does not mean that upper management defines the metrics and hands their plan over to the project team for execution.  Upper management provides guidance and direction by making available clearly defined project and organizational goals.  The developers (usually a GQM team working with the project team) define the measurement goals and ultimately, the metrics.  A GQM team may need to coordinate this task among several projects to ensure consistency of some metrics across projects.  This activity would be driven by a higher organizational need for consistent metrics to improve activities such as cost estimating or benchmarking.  The right people must be involved in every phase of GQM measurement.

The key roles involved in GQM are:

·         GQM Goal Owner – the person, or persons, whose viewpoint is stated in the measurement goal and  who is responsible for the whole analysis task; a project manager, test manager, quality assurance (could also be a customer)

·         Measurement Manager – the person responsible for running the measurement program (operational perspective) and executing the measurement plan.  This role, which could be within or outside of the project team, is explicitly identified in both the project plan and the measurement plan.

·         Data Provider – any person who provides data during data collection.  They need to be involved in order to understand exactly how the data they collect will be used and that it is being used as planned.

·         GQM Expert - person who performs the technical tasks of the measurement program, such as interviewing, and deriving the measurement plan.  Initially this might be a consultant but, after a training period, this role typically transitions to key project staff.

·         GQM Team – depending on the size of your organization and the scope of the proposed GQM implementation, it may be valuable to create a GQM team that specializes in the GQM process.  Such a team is often integrated with the group that provides Quality Assurance to the projects, but its focus is on implementing GQM.  The team might initially include an external GQM consultant.  The team does a lot of coordination with project managers as well as much of the initial planning and analysis.  However, the GQM team does not interpret the data.  It merely facilitates data interpretation by focusing on various ways to present measurement results to key people.

2 – Set explicit measurement goals and state them explicitly:  Measurement goals are not organizational goals or project goals.  They are goals that describe how to assess progress toward project goals and organizational goals.  There are subtle differences.  It is important for all members of the project team to understand and distinguish among these three types of goals.  Measurement goals direct the alignment of measurement activity with the business goals (project level and organization level) of the organization and, as such, guide all subsequent activity of the GQM process, so it is important to focus on them.

Are they the right goals?  Have we identified all of the key goals?  Is their meaning crystal clear?  Developing explicit goal statements, using the five facets of information identified in Table 1, helps provide that clarity and remove the ambiguity of implied assumptions among the team.

3 – Don’t create false measurement goals:   Do not create goals simply to match the metrics you may already have.  You may be collecting the metrics that are easy to collect, without proper analysis to determine if or how they fit into your measurement program.  It is easy to get over zealous about measurement and identify metrics that are easy to collect and look at.  The perspective of “We can get this data, so what can we do with it?” or “ Let’s look at the data that we already have (so as not to create extra work) in different ways and see what we can discover” is a form of creating false goals.  This perspective is sometimes evident in projects that do not follow a disciplined development process.  It is comfortable; it is sometimes a good way to get people who are “anti-metrics” to adjust their attitude, but this mindset is, in fact, the reverse of GQM.  It bypasses goal definition and refinement (questioning) and goes directly to identifying metrics first.  Since the focus is on the metrics, this practice results in implicit assumptions about the goals or purpose of the metrics.  This practice may be better than not measuring at all, but it may be difficult to migrate from this to a GQM mindset.  The key point is, if you are implementing GQM you must let goals drive the selection of metrics. You must focus on goals first, independent of current metrics or data you might have.  The goals should drive the identification of the right metrics, or you are not doing GQM.

4 – Acquire implicit quality models from the people involved:  Note that, while it is important to define measurement goals explicitly, GQM tries to acquire the implicit quality models, i.e., the notions of quality that the development team members or the customers have in their heads.  Developers, at all levels, need to be involved in identifying their notions of quality, project constraints and specific elements of the environment that can affect quality.  Their perception of quality is important because they are closest to the development effort.  Their input, in the form of questions that address their perception of quality, is essential if meaningful metrics are to be identified.  Without their understanding and acceptance of both the goals and the metrics that quantify them, GQM will probably not be an effective course of action.  Here are some questions that tend to reveal the variation of perceptions that might exist within the development team:

·         What is product quality? Process quality?

·         When is it ok to release product?

·         Is our current product quality good or bad? How do we know? 

·         Are certain quality attributes more important than others to us? To our customers?

·         Does our notion of good quality match that of the customer? 

·         Does the project manager have a different notion of quality than the designer, or the test engineer? 

·         Does everyone assume that team members all have the same notion of quality, so, it has never actually been discussed?

Somewhere within the project, or the organization, there may be some definitions of various quality attributes, but that does not mean that a common awareness of those concepts exists among the team members.  Through questions, the team members put their implicit notions (their quality models) of the object of measure on the table for discussion, thus providing a more detailed definition of the goal’s quality focus [van Latum, 1998].

Although a GQM team (or specialist) typically leads the goal definition initiative, all project team members should be consulted for two reasons, (1) to solidify their understanding of the goals and (2) to optimize their motivation to reach the goals.

The success of this practice depends on many people issues, and therefore, techniques, such as facilitated discussion and personal interviews, should be applied as necessary.  Some generic techniques that may be useful for getting through the GQM definition phase are:

·         Brainstorming - a workshop technique used to generate ideas that could be useful in solving some problem.  It enables a group of people to come up with various suggestions for a possible solution to a specific issue.  It is run by a facilitator who should merely record suggestions without comment.  It is important that no suggestions be in any way belittled as sometimes an off-the-wall suggestion leads to an innovative solution.  Once the brainstorming session is completed, the ideas are reviewed rationally to see if any of them contribute to the solution of the problem in question.

·         Facilitation - facilitation is a technique of encouraging discussion among a group of individuals who are engaged in some problem solving activity.  Many groups contain dominant personalities whose influence can overpower the discussion.  The facilitator should be capable of encouraging the widest possible input from everybody taking part.  The facilitator is also required to suggest lines of debate, if the discussion dries up or comes to a natural halt.  They should ensure that all aspects of the subject under debate are discussed.

·         Mentoring -a mentor is an advisor to a less experienced person in some field of endeavor. The mentor is (or should be) seen as a trusted advisor who can provide support and guidance which is encouraging and non-judgmental.  In all too many cases, mentors are not properly trained and are often chosen because of their technical skills rather than their personal ones.  Effective mentoring is very reliant on personal skills; a badly chosen mentor may do irreparable harm to a less experienced person.  Conversely, well conducted mentoring can be a very powerful way of transferring knowledge, skills and responsibilities.  The very best mentors often claim that they have learned more from the experience than they taught.

·         Using a collaborative workflow tool – web-based software that allows for idea sharing, planning and decision making in a controlled environment, usually involving a moderator or facilitator.  It is deemed useful for requirements negotiation, and GQM is about establishing and implementing requirements for measurement.

Team members need to feel comfortable and confident to express themselves.  The process can easily break down into a situation where a single individual dominates and only their thoughts surface.  Conflicting notions may emerge and need to be resolved among the team members before proceeding to the metrics identification stage.  Fifty questions might surface, but the team may end up with five or six that are deemed sufficient to fully address the goal.  Facilitation is advisable for this process.  Techniques such as affinity grouping may also be valuable.  Some of this can be done via interviews with results captured in abstraction sheets.  Figure 9 shows the structure of a typical abstraction sheet.  The content represents an interview with the project manager of the specified project.  In large projects, where having face-to-face meetings is difficult due to the size of the staff, personal interviews, conducted by the GQM team, take the place of open discussion.  Abstraction sheets are used to capture the perspective of individuals, and then analyzed by the GQM team.  Differences in the quality focus or the variation factors emerge among the interviewees.  Such differences need to be addressed through further iterations of GQM in order to ensure that the process will result in convergence of the team regarding the quality focus of each of the goals.

Figure 9: Sample of a Completed Abstraction Sheet

5 - Consider context:   What are the variation factors of the quality focus due to the context of our project?  What constraints or limitations do we have?  What factors do we expect will most influence our perceived quality model?  Team members pose questions (Q stage) to address how the particular context of the project environment could impact the quality focus.  One example might be related to the level of code reviews done on the project.  Q = How will code reviews affect the quality?  If more code reviews are done, we are likely to find fewer faults during testing.  For example, if the quality focus refers to achieving a 10% reduction in defects for the next release, but we have not adequately tracked defects in the past, then how will we know whether the goal is attained?  This would probably trigger a revision of the goal, and perhaps some new goals relating to establishing a baseline for defect tracking.  In essence, in this part of the Q stage, we are customizing the quality focus to our particular environment to make it realistic and, therefore, attainable.

6 - Derive appropriate metrics:   For a given goal and question, there are many relevant metrics.  The key is to identify those metrics that clearly satisfy the question.  More metrics are not necessarily better.  Metric definition, data collection, and analysis and interpretation are extra work, and costly to produce.  Do not create more metrics than are really needed.  In some cases, the same metric may address more than one question.  The mapping of questions to metrics need not be one-to-one.

Recognize that it is possible to select the wrong metrics.

§         Metrics chosen should be specific to what you are trying to measure, not generic metrics that you collect because they can be collected

§         Choosing the wrong metrics causes extra effort collecting and reporting on the wrong data and often results in having to collect more relevant metrics after-the-fact

The task is complicated by the fact that the software community has not converged on a standard definition of  “metric”, often using it to describe many different measurement concepts, or interchanging the terms “metric”, “measure (or measurement)”, and “indicator”.  Gopal [Gopal, 2002] defines a metric as:

“A method of quantitatively determining the extent to which software process, product, or project possesses a certain attribute”

McGarry [McGarry, 2002] defines a measure as:

“A variable to which a value is assigned to represent one or more attributes”

He defines a base measure as:

“A measure of a single attribute defined by a specified measurement method, functionally independent of all other measures and capturing information about a single attribute”

Within the context of this document, the terms should be viewed as a hierarchy with “indicator” at the top and “measure” at the bottom.  A metric is an expression composed of one or more measures (base or derived) and/or other metrics that can include conditional selection statements.  For example, one metric might be the number of defects found during testing.  The measures used, as components for the metric, are things like defect count, and phase defect detected, and so on.  A reduction in the number of defects found in testing might be considered an indicator of an improved development process. In planning, developers might hypothesize that a reduction in the defects found in testing would mean, perhaps, that their implementation of formal inspections was working.  Indicators convey the significance of the metrics in the specified environment.  The example in Figure 10 illustrates the hierarchical nature of these terms.

Figure 10:  Hierarchy of Measurement Terms

7 – Stay focused on goals when analyzing data:  GQM is about seeing if the measurement results answer the questions posed and address the goal.  It is not about mining the data to see what else can be discovered.  In the analysis plan, the analysis process should be clearly specified, including how the data needs to be organized and presented for interpretation.  The GQM team (or project counterpart) has the responsibility for determining how to aggregate and present data so that it addresses the goal, but they do it with input and review from the project team.  Their initial plan may not be perfect, but the project team will be able to identify issues, as they are the ones involved in interpreting the data.  This feedback is then used to improve the analysis support.

8 – Let the data be interpreted by the people involved:  Just as those involved in development need to be part of the definition team, they also need to be involved in interpreting the measurement results.  They are the people who can say,

 “Yes, but --- we didn’t consider this when we were defining the metrics …”, or, 

  “This is only meaningful in the following context … “, or, 

   “Although this is technically correct, if it was organized this way it would be more useful.”

All too often measurement is seen as extraneous to real project work and is assigned to someone outside of the project who performs various analyses and interpretations that may or may not be communicated to the project team.  Often, measurement-related reports are sent up the line – beyond the development team.  Interpretation should be done by the people who represent the viewpoint of the goal, i.e., the goal owners.  If the project manager (PM) owns the goal, then the PM should interpret the results.  Note that when goals are explicitly stated and the owner identified up front, then this task is easy.  This does not mean that others are excluded from the interpretation process; the goal owner, however, leads the interpretation process.  Without such direct involvement, the measurement process tends to break down, as people tend to view it as extraneous to their real work.

9 – Integrate the measurement activities with regular project activities:  Recognize that a metrics program is, in fact, extra work.  Implementing the metrics program is itself a project that should be interwoven within software project or process activities.  The PM should incorporate the planning aspects of GQM into the project plan, budget for measurement, track it, and be able to demonstrate the cost effectiveness (or lack of) of the implementation.  Responsibilities regarding measurement need to be clearly delineated and assigned within the project team.  If it is feasible to use a GQM team across several projects, then each project needs to understand how the GQM team and the project team will interact.  Both the GQM and project teams review the GQM plan (which contains the Gs,Qs, and Ms) thoroughly to arrive at a set of standard terms for the process and product models, and to eliminate obscurities and ambiguities.  The measurement plan describes the metrics, procedures, and media needed to collect, report and validate data for each goal, embedding data collection into the development process.

The analysis plan describes how to analyze and aggregate measurement data into presentation formats, which others can easily interpret and use to answer the questions in the GQM plan.  It is important to describe how to compare actual data with the hypotheses established during the interviews used to formulate the GQM plan. Together, the GQM and analysis plans help during the interpretation stage to focus on what the project team deems essential.

If this is the first implementation of a measurement program, then it is important to establish project goals that establish a baseline for measurement activity.  Measurement goals in that scenario would focus on understanding the current state of the development process in order to establish a baseline for quantifying improvement.

Another important integration element is participation of the project team in feedback sessions, interpreting the data that is prepared and presented by the GQM team in accordance with the plan.  Such sessions should occur on a regular basis (frequency to be determined by the project team), allowing sufficient time for new data collection, yet short enough to hold the interest of the team.  The sessions lead to significant updates in the analysis, measurement and GQM plans and set the stage for evolving improvement.

10 – Do not use measurements for other purposes:   The fastest way to derail the measurement program is to use the measurement results for other purposes not related to the goal.  This is especially true when such measures are misused to assess the productivity of individual team members or as a basis for individual awards.  This can occur when someone other than the goal owner interprets the measurements, perhaps with their own hidden agenda.  It causes a breakdown in the trust relationship among team members and between the project team and management, often resulting in actual data collection being skewed to influence the outcome and degraded data integrity.

11 – Secure management commitment to support measurement results:   This is, perhaps, the most critical of all GQM success factors, but it is not unique to GQM; it applies for any measurement implementation.  If management ignores recommendations supported by sound measurement results, then measurement itself is perceived as an exercise in futility, and in that scenario, most measurement programs do not survive.  One way to gain support from upper management in the early stages of the initiative is to communicate to them how the measurement goals will support their business activity.  This prepares management for anticipated results, removing the element of surprise.  Since business or organizational goals indirectly drive the identification of measurement goals, this is not difficult to accomplish.  However, it is a “people” issue in that there must be the appropriate level of communication between the technical staff and the business managers, and any political issues need to be addressed early, and perhaps identified as risks.  Without real commitment, it does not make sense to proceed with any formal measurement program.

Take the time to develop a measurement project plan that identifies the resources, the period, and the coordination needed to implement measurement.  This plan will help to align management’s expectations. [Dekkers, 2003]

12 – Establish an infrastructure to support the measurement program:   Since measurement is extra work, it is important to prevent that work from infringing on the product development effort.  Lavazza [Lavazza, 2000] notes that GQM plans can easily grow into quite complex and large graphs.  A single goal can generate more than 100 metrics.  Without specific automated support, keeping this amount of metadata under control is a challenging task.  Wherever possible, automate data collection and provide tools for storage and analysis of metric data.  A common database should be established that can be used by all projects for storage and analysis, and also serves as an organizational resource for estimating future efforts.  In addition to the actual measurement data, the following metric information should be preserved:

·         Name and definition of each unique metric

·         Classification for each metric

·         Association point in product development that identifies when and how the data is to be collected

·         Definitions of the data collection forms or other mechanisms

·         Procedures for data reporting, collection, and validation

·         References to the metrics in the GQM plan (link to goals)

There are many useful tools for data collection and analysis.  The key is the degree to which the tools interoperate or the resulting data can be integrated.  Making good tool choices is necessary in order to automate as much of the data collection as possible.  In many cases, such tools may already exist within the organization but have not been addressed from the perspective of interoperability or integration.  Here are some types of tools that are important components of the measurement infrastructure:

·         Defect tracking/event tracking

·         Software configuration management

·         Applications/packages that generate web statistics

·         Relational database management systems

·         Spread sheets

·         Drawing/charting tools

·         Requirements management software

·         Collaboration tools

13 – Ensure that measurement is viewed as a tool, not the end goal:  Measurement should aid, rather than hinder, the development process.  Do not get so focused on metrics and measurement that you lose site of the project itself.  Leaders must work diligently to keep the focus on the project, product, or process-focused improvement goal rather than the stockpile of measures collected.

14 – Get training in GQM before going forward:  GQM sounds simple, and there may be the temptation to do it yourself, but beneath the surface is a sophisticated process that is likely to be different from the current thinking and modeling of most practitioners.  Some initial training is important to understand and master the salient points of transforming the team thoughts and concerns into the GQM hierarchy.  Team coaching through the definition and interpretation phases by a skilled trainer or someone who has had a successful experience with GQM, can be very important for the first implementation.  The role of a GQM expert can be transitioned to internal personnel after initial pilot training has been successfully implemented.

Characteristics of implementation

Summary characteristics

NOT AVAILABLE

ANTICIPATED BENEFITS OF IMPLEMENTATION:

The most direct benefit of implementing GQM is the identification of meaningful metrics.  However, some additional benefits of this practice that were identified in the PERFECT project [PERFECT 1997] include:

§        Achievement of improvement goals

§        Financial gains

§        Increased capability to perform improvement initiatives

§        Improved group synergy

§        Increased quality awareness and Quality Assurance (QA) involvement

Harris Corporation [Natwick 2003] used GQM to develop an integrated metrics set for quantitative management of performance, progress, cost, schedule, and resources across systems, software and hardware engineering disciplines.  They assert that institutionalizing this approach resulted in their achieving CMM for Software Level 4.

Detailed characteristics

Key Characteristics of the “Goal-Question-Metric Approach” Gold Practice

Relationships to other practices:

The figure below (and the table of inputs and outputs that follows) represents the high-level process architecture for the subject practice, depicting the nature of the influences on the practice (describing how other practices might relate to this practice).  These statements are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may “influence” (or be influenced by) the ability to successfully implement other practices.  A brief description of these influences is included in the table below.  The arrangement of the statements in the figure and corresponding table does not imply any particular order or sequence in which they should be addressed.

Process Architecture for the "Goal-Question-Metric Approach" Gold Practice

Note: In the table below, specific Gold Practices with strong relationships to Goal-Question-Metric are identified with italics.

Summary of Relationship Factors

RESOURCES:

§         Websites

0            NASA Software Engineering Laboratory - created to investigate the effectiveness of software engineering technologies when applied to the development of applications software. It maintains an Experience Factory (experiences, packaged into useable formats such as downloadable handbooks etc.), which includes documents describing GQM.

http://sel.gsfc.nasa.gov/website/exp-factory.htm

0            DACS Return-On-Investment (ROI) Dashboard  - presents benefits data from software technical and management improvements in a dynamic graphic display format. Additional data is being added to the dashboard on a continuing basis.  Benefits from measurement programs is one of the optional selections.

http://www.thedacs.com/databases/roi/      

0            The Goal/Question/Metric Method website – These internet pages have been written and will be maintained to support the ideas published in the textbook ‘The Goal/Question/Metric Method’, written by Rini Van Solingen and Egon Berghout (McGraw-Hill, ISBN 007-709553-7, 1999).   

http://www.gqm.nl/

0            PERFECT Project website – The objective of PERFECT is to assist European industry in measurement-based improvement of software processes.  A set of techniques, methods and tools supporting the improvement activities have been developed and are available in downloadable format from this site.

               http://www.iese.fraunhofer.de/perfect/

0            History of Software Measurement by Horst Zeus – provides a comprehensive history of measurement up to 1995, with links to the major publications      

       http://irb.cs.tu-berlin.de/~zuse/metrics/History_00.html

0            DACS Topic Area on Measurement – updated on a continuing basis, this site has links to literature, people, programs, organizations, tools and other resources relating to measurement in general, and related subtopics such as GQM.

http://www.thedacs.com/databases/url/key.hts?keycode=3

§         Tools and Methods

A multitude of methods and tools may be useful in implementing and improving your measurement program.  GQM is itself, a method for establishing a metrics program.  As stated in the description of this practice, the success of a GQM implementation is dependent on establishing the appropriate infrastructure to support metric definition, data collection and analysis, and to capture for reuse the various artifacts of the metrics program.  Having a database management system that is sufficiently flexible to allow ad hoc query, as well as dynamic reporting capabilities, is essential.  Readers are encouraged to share information about tools they are using by participating in the Gold Practice discussion forum for this practice.  Note: you must be a Gold Practice member to contribute to the discussion forum.  [Click here] to register for free membership if you are not already a member.  To date, there are a few references found in the literature to commercial tools designed specifically to support the GQM process.  Most organizations report developing their own GQM tool by integrating aspects of generic commercial products.

[The fact that a tool or tool vendor is listed (or not listed) here does not constitute an evaluation or endorsement of the product or vendor by the DACS.  These are simply links to tools currently available relative to the subject practice.]

Tools That Specifically Address GQM

§         Experts and Contact Points

Basili, Victor  A professor at the University of Maryland, Dr. Basili is the developer of GQM and is a noted advocate and practitioner of quantitative methods in software engineering.  He was a co-founder of NASA's Software Engineering Laboratory and has won numerous awards.  His Web page provides pointers to the journals and organizations with which he is associated, as well as papers and other descriptions of his research interests.

http://www.cs.umd.edu/users/basili/

basili@cs.umd.edu

Prof. Rini van Solingen, Logica CMG, The Netherlands – Professor van Solingen has been involved with many implementations of GQM in conjunction with the  PERFECT consortium, a European software improvement initiative active in the late 1990s.  He can provide real world experiences.  He is also the Industry Chair for the PROFES (Product Focused Software Process Improvement) 2005 conference.
rini.van.solingen@logicacmg.com

Luigi Lavazza has written about providing automated support for the GQM approach and has real world experience in making GQM successful through automation of data collection.

lavazza@cefriel.it

§         Training Opportunities

The fact that a course is listed (or not listed) here does not constitute an evaluation or endorsement of the course by the DACS.  These are simply links to courses currently available relative to the subject practice.  In the interest of keeping the document manageable, the following types of training have not been included:

§         Pure consulting services

§         Courses focused on broader areas, with GQM identified only as a component

§         Tool–specific training courses

Some vendors offer a virtual classroom alternative to the traditional classroom training, and may offer on-site training opportunities.  Click on the links provided to get this type of information. 

Courses on GQM

§         Bibliography

 

APPENDICES

 

DEFINITIONS:

 

GQM Paradigm

Goal Question Metric (GQM):  A method used to define measurement on a software project, process and product.  GQM defines a measurement model on three levels: (1) Conceptual level (goal):  A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view, and relative to a particular environment.  (2) Operational level (question): A set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.  ( 3) Quantitative level (metric):  A set of metrics, based on the models, is associated with every question in order to answer it in a measurable way.

A collective term for the set of basic principles concerning goal-oriented measurement, the templates and guidelines that assists in defining goals, and methods of applying the basic principles.

 [Differding, 1996]

 

GQM Metric

This describes something that we would like to measure. Opinions vary as to whether a GQM metric must be directly collectible (e.g., “time in days”) or whether a GQM metric may be something as complex as “productivity.” The argument for directly collectible metrics is that all necessary information for refining a goal into collectible metrics should be captured in the GQM plan. The argument for permitting complex metrics is that a metric need only state clearly what a person would like to know.  The use of complex metrics in a GQM plan implies that some refinement of GQM metrics into directly collectible data items may be necessary before beginning to use a GQM plan. 

Some hold the opinion that a GQM metric need not even be collectible at all; in other words, no refinement may be necessary or even possible. In the case of a non-collectible metric, stating the metric would merely serve to illustrate what is desired, and would thereby reveal the limitations of the metrics for which data are collectible.

 [Differding, 1996]

 

SOURCES (Origins of the Practice):

 

The GQM approach was originally developed by Victor Basili, et. al. [Basili 1984] at the University of Maryland, to evaluate defects for NASA projects.  It was later expanded to apply to a larger context and now, often serves as a cornerstone for many process/product quality improvement initiatives. 

 

RECOMMENDING SOURCES:

[This section identifies segments from specific DoD, industry, or academic sources that recommend, specify or otherwise support adoption of the subject practice either directly, or as part of a broader best practice initiative, or a practice with a different name.]

§        Capability Maturity Model Integration (CMMI), Version 1.1, Software Engineering Institute, CMU/SEI-2002-TR-011, TR-012, March 2002[CMU/SEI-2002-TR-011], [CMU/SEI-2002-TR-012]

 

The CMMI identifies the Measurement and Analysis key process area (KPA) as part of CMMI Level  2.  The purpose of this KPA is to develop and sustain a measurement capability that is used to support management information needs.  The following text is extracted directly from this KPA.  Although there is no mention of GQM, the goals and practices described herein provide a strong parallel with most descriptions of GQM.  The KPA involves:

 

·         Specifying the objectives of measurement and analysis such that they are aligned with identified information needs and objectives

·         Specifying the measures, data collection and storage mechanisms, analysis techniques, and reporting and feedback mechanisms

·         Implementing the collection, storage, analysis and reporting of the data

·         Providing objective results that can be used in making informed decisions and taking appropriate corrective actions

 

 

§        Report of Defense Science Board Task Force on Defense Software, November 2000

 

Although it does not specifically mention GQM, the report refers to establishing meaningful metrics as a commercial best practice that needs to be adopted by the DOD.

 

Specifically, DSBTF (2000) found that one of several recommendations from the 1987 DSB Task Force on Military Software, namely, “that the DoD should develop metrics and measuring techniques for software quality and completeness and incorporate these routinely in contracts”, had not yet been implemented in DoD practice.

 

The Task Force also found that among numerous software development best practices utilized on commercial programs was the practice of ”incentivizing development teams based on meaningful, quantifiable metrics”.

 

Although the DoD is making increased use of other commercial best practices, this practice is not well established in DOD projects.

 

§         GAO Report on Defense Acquisitions, March 2004 [GAO, 2004]  

 

The consensus of this report was that stronger management practices are needed to improve DOD’s software-intensive weapons programs.  The report goes on to state that the success (quality software product, on time and within budget) of leading software companies is attributable to creating a manageable evolutionary development environment, using disciplined processes, and collecting and analyzing meaningful metrics to measure progress.  The report does not specifically mention use of the GQM approach but it is generally accepted in the software community that GQM provides a way to establish meaningful metrics.

 

§         Software Program Managers Network (SPMN) 16 Critical Software Practices for Performance Based Management

 

The SPMN was established in 1992 to identify proven industry and government software best practices and convey these practices to managers of large-scale DoD system acquisition programs.  The SPMN 16 Critical Software Practices^TM specifically address underlying cost and schedule drivers that have caused many software intensive systems to be delivered over budget, behind schedule and with significant performance shortfalls.

“Use Metrics to Manage” is one of the 16 practices.  Although it does not specifically mention the GQM approach, many of the elements of GQM are evident within the following list of essentials:

1.       All programs should have in place a metrics program to monitor issues and determine the likelihood of risks occurring.

2.       Metrics should be defined as part of definition of process, identification of risks or issues, or determination of project success factors.

3.       All metrics definition need to include description, quantitative bounds, and expected areas of application.

4.       All programs need to assign an organizational responsibility for identification, collection, analysis, and reporting of metrics throughout the program's life.

5.       Metrics information should be used as one of the primary inputs for program decisions.

6.       The metrics program needs to be continuous.

 

GLOSSARY

 

Abstraction Sheet	A tool used to capture the perspective of individuals regarding the quality focus, variation factors, baseline hypothesis, and the impact of the variation factors on the quality focus.  They are typically used as part of an interview and show differences of perspective  of the individuals involved in a project. [Gresse, 1995].
Analysis Plan	Focuses on how to analyze and aggregate measurement data into presentation formats that others can easily interpret
Experience Factory(EF)	Logical and/or physical organization for continuous learning from experience, including an experience base for the storage and reuse of knowledge [Conradi, 2003]
Facilitation	The art of leading people through processes toward agreed-upon objectives in a manner that encourages participation, ownership, and productivity from all involved
GQM Approach:	Synonymous with GQM Paradigm                                  [Differding, 1996]
GQM Paradigm	Goal Question Metric (GQM): method used to define measurement on a software project, process and product.  GQM defines a measurement model on three levels: (1) Conceptual level (goal):  A goal is defined for an object, for a variety of reasons, with respect to various models of quality, from various points of view, and relative to a particular environment.  (2) Operational level (question): A set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.  ( 3) Quantitative level (metric):  A set of metrics, based on the models, is associated with every question in order to answer it in a measurable way.
GQM Plan	A single goal plus the sets of questions and metrics that provide an operational definition of that goal. A GQM plan documents the refinement of a precisely specified measurement goal via a set of questions into a set of metrics. Thus, a GQM plan documents which metrics are used to achieve a measurement goal and why these are used - the questions provide the rationale underlying the selection of the metrics. On the other hand, the GQM plan is used to guide analysis tasks because it documents for which purpose the respective data were collected. [Differding, 1996]
Indicator	A device or variable that can be set to a prescribed state based on the results of a process or the occurrence of a specified condition. For example, a flag or semaphore. [IEEE, 1990]
Measurement Plan	When coupled with a GQM Plan, a measurement plan specifies who collects the data required by the GQM Plan, how the data is collected, and when the data must be collected.  A measurement plan usually includes the data-collection forms as well as descriptions of tools that perform online data collection. [Differding, 1996]
Metric	In measurement theory, this is a mapping used to assign a value to some attribute of an entity. [Differding, 1996]
Metric	A quantitative measure of the degree to which a system, component, or process possesses a given attribute. [IEEE, 1990]
Quality Metric	(1) A quantitative measure of the degree to which an item possesses a given quality attribute. (2) A function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which the software possesses a given quality attribute. [IEEE, 1990]
SEL	Software Engineering Laboratory
SPMN	Software Program Managers Network

 

CASE STUDIES from the literature:

A Software Measurement Case Study Using GQM

By Bjorn Lindstrom, Lund Institute of Technology, Lund University, Masters Thesis, 2004

http://serg.telecom.lth.se/education/master_theses/docs/62_Rep.5522.Lindstrom.pdf

 

Description:

Software measurement is about measuring attributes of the process, product and resources in software projects, and the overall objective for it is to get information about the work to be able to improve and make it predictable.  Software measurement is far from a standard procedure in the software industry, mainly due to lack of immediate results.  HP OpenView in Amsterdam, and the team Special Products Engineering, is one example of this, where they have not used software measurement in their previous projects.  Hence, the manager saw a need to evaluate the completed projects with measurement.

 

Measurement always implies some effort and it is therefore important to know why you measure, for what purpose, and for whom.  The Goal Question Metric (GQM) approach is a way to find out why and what to measure and this approach is applied.  The work presented in this report is based on existing information only, which tightly limits the possible measurements.

 

Results:

This statement is extracted directly from Lindstron’s thesis [Lindstron, 2004]:

“The conclusion of using GQM is that it is not suitable when the existing data is very limited because it results in too many questions and metrics that are not possible to answer and measure.  The main conclusion of these measurements is that the way the team has been working is not sufficient for reliable measurement.  Most measures are not possible to validate and one should therefore be careful to draw general conclusions from the result.  Still the results indicate that the organization is immature and the productivity low.”

What is implied by these conclusions is that the effort required to implement GQM may be perceived as wasted if the data needed to establish a baseline for many of the metrics could not be validated.  He cautions against moving forward when existing data is not reliable.  Using it to answer some of the questions is dangerous.  Thus, in this scenario using GQM would mean abandoning existing data and starting from scratch to collect and validate data on a go forward basis – providing a payoff only at some future point in time for new projects.  The study did give HP a much clearer impression of exactly where they were with respect to their measurement capability, however.

 

Integrated Metrics for CMMI and SW-CMM: Lessons Learned

By Gary Natwick, October 30, 2003

http://www.software-engineer.org/article_read.php?article_id=20000939

 

Description:

As organizations move toward the Capability Maturity Model® Integration (CMMI), requiring the integration of technical and management processes across functional disciplines, the tool suites used to plan, manage, and monitor these integrated processes must also evolve to support them.  One example of this is an integrated engineering metrics set to reinforce process deployment, provide effective management oversight, and ensure alignment with organizational business goals.  Harris Corporation used a Goal-Question-Metric approach to develop an integrated metrics set for quantitative management of performance, progress, cost, schedule, and resources across systems, software, and hardware engineering disciplines.  Institutionalization of this metrics approach resulted in achieving CMM for Software Level 4.

 

Results:

Lessons learned from implementing a metrics program and tool within an integrated discipline work force are as follows:

·         One metric does not tell the whole story.  You need integrated, and many times, orthogonal views of metrics to get a complete picture; trending is key.

·         Project planning is key, and data collection is the hardest

·         Having an organizational standard tool is a must for consistency; it should be user friendly with easy access

·         Cultural change is hard, so train everyone about the organizational metrics program and tool to increase acceptance and buy-in

Conclusion:
Integrated engineering metrics are required to provide effective management oversight and to ensure alignment with organizational business goals.  As organizations move toward the CMMI requiring the integration of technical and management processes across functional disciplines, the tool suites used to plan, manage, and monitor these integrated processes must also evolve to support them.

 

Validation of an Approach for Improving Existing Measurement Frameworks

By Manoel G. Mendonca and Victor R. Basili, Fellow, IEEE, Transactions on software Engineering, Vol. 26, No 6, 2000

 

Description:

Software organizations are in need of methods to understand, structure, and improve the data they are collecting. We have developed an approach for use when a large number of diverse metrics are already being collected by a software organization [1], [2]. The approach combines two methods. One looks at an organization's measurement framework in a top-down goal-oriented fashion and the other looks at it in a bottom-up data-driven fashion. The top-down method is based on a measurement paradigm called Goal- Question-Metric (GQM). The bottom-up method is based on a data mining technique called Attribute Focusing (AF). A case study was executed to validate this approach and to assess its usefulness in an industrial environment. The top-down and bottom-up methods were applied in the customer satisfaction measurement framework at the IBM Toronto Laboratory. The top-down method was applied to improve the customer satisfaction (CUSTSAT) measurement from the point of view of three data user groups. The bottom-up method was used to gain new insights into the existing CUSTSAT data. This paper uses the case study and its results to qualitatively compare our approach against current ad hoc practices used to improve existing measurement frameworks (MF).

 

Results:

The top-down approach identified several new metrics for the interviewed groups, and contributed to better understanding the data user needs.  Unexpected associations between key variables prompted new business insights, and revealed problems with the process used to collect and analyze the CUSTSAT data.

 

The case study tested the three parts of the new approach with respect to ten MF improvement objectives considered important or very important by the Toronto Lab data manager. The case study showed that the new approach was effective in achieving eight of these ten objectives.  The case study also showed the CUSTSAT MF is mature and has diverse mechanisms to achieve some of the improvement objectives. Nonetheless, even in this scenario, the AF and GQM-based methods contributed by complementing and expanding the capabilities that already existed to improve the MF. This indicates that the new approach acted in areas that were important but were being ignored in this and possibly in others MFs.

 

 

The Relationship of GQM To Other Paradigms and Frameworks

 

The principles on which GQM is built include [PERFECT 1997]:

·         Set explicit measurement goals and derive appropriate metrics

·         Consider the context of the object to be measured

·         Analyze data according to the goals

·         Involve the people who are experts on the object to be measured

 

Several measurement methodologies have emerged in recent years.  Some are variants of the GQM approach.  Some address the same elements but from a slightly different perspective, or using slightly different terminology.  Some are themselves measurement frameworks and some are organizational frameworks that have an integrated measurement methodology embedded within them [Basili 2005].  This section provides a brief explanation of some of these paradigms, and how they may or may not relate to GQM,   The Figure  below illustrates the relative chronology of these paradigms and methods and relates them to either the measurement or organizational framework depending on their scope.

 

Application of Metrics in Industry (AMI):  The AMI method is the outcome of the AMI project, initiated in the 1990s by the AMI consortium, comprised of experts in software measurement and users from across Europe, applying quantitative methods to the management of their projects.  The goal of the AMI project was to provide a practical and validated approach to implementing quantitative approaches to control and improve the software process.  The project produced the AMI method and the AMI Handbook, which was launched in 1992 and published by Addison-Wesley in 1995. AMI had a wide user base [ESPRIT 1996].

This approach establishes an integrated framework for software process improvement using both analysis and benchmarking techniques to quantify software projects.   AMI consists of four primary activities - Assess, Analyze, Metricate, Improve:

Assess the project environment (with its objectives and problems) to define primary goals for measurement

Analyze the primary goals to derive sub-goals and relevant metrics (make a measurement plan)

Metricate – Implement the measurement plan

Improve  - Compare results with the plan and make adjustments as necessary

Although its activities have different names and are grouped differently, the AMI method embodies the same principles as GQM. However, GQM places greater emphasis on involving the stakeholders throughout the process.

Balanced Scorecard (BSC): The balanced scorecard is both a measurement system and a management system that enables organizations to clarify their vision and strategy and translate them into action.  It was developed in the early 1990's by Drs. Robert Kaplan (Harvard Business School) and David Norton.  BSC views the organization from four perspectives and develops metrics, collects data and analyzes it relative to each of these perspectives.  There must be a balance among the perspectives. Figure 4 provides a visual synopsis of this approach.

 

Source:  Balanced Scorecard Institute web site (see http://www.balancedscorecard.org/  )

Figure 4:  The Balanced Scorecard Approach to Performance Measurement for Strategic Management

 

The BSC methodology builds on some key concepts of previous management ideas such as Total Quality Management (TQM), including customer-defined quality, continuous improvement, employee empowerment, and -- primarily -- measurement-based management and feedback.

BSC is focused on getting from the organization’s vision to specific objectives for achieving that vision.  It is forward reaching in that it assesses not only how you have been doing (“lagging  indicators”), but how well you are currently doing (“current indicators”) and how well you can expect to do in the future (“leading indicators”).  Although BSC is typically implemented at the organizational level and encompasses organizational level goals being identified in addition to measurement defined, GQM can be used within BSC to fill the gap between these higher-level goals and the precise measures that will need to be collected.  BSC provides guidance for the presentation of data and the areas of measurement that need to be addressed, but GQM, or other specific measurement methodologies, can be used to support the implementation of BSC.  BSC is focused on upper management needs and essentially involves management stakeholders, while GQM focuses on getting the right people involved at all levels to ensure the right goals and metrics are identified and get buy in from the stakeholders.  BSC can incorporate the human focus of GQM, although this is seldom addressed in the literature about BSC.

The Department of Energy (DOE) Federal procurement system has been implementing the BSC approach in planning for FY04 and FY05 and has some excellent implementation examples on its website, which include not only the planning but the reporting of results as well.

 See http://professionals.pr.doe.gov/ma5/MA-5Web.nsf/Business/Balanced+Scorecard?OpenDocument

Although the visualization of the BSC model fits nicely on one page and, perhaps, suggests that the actual goals and measurement information fit into this visual space, the volume of information typically needed necessitates a different visual structure. This is exemplified in the BSC documents found on the DOE website noted above.

In some communities of interest the “Balanced Scorecard” terminology is used to reflect the structure of management reports that present several aspects of project data on one page so that a manager can quickly see the financial status, the technical status and any significant issues along with a brief statement of the project primary purpose.  These are sometimes called QUAD charts.  They are monitoring functions that attempt to capture the “balancing” notion of BSC.; however, they could be part of a broader BSC, or GQM, initiative.

Capability Maturity Model (CMM and CMMI): The CMM for Software and its successor, CMMI, provide a process benchmark, which organizations can use to assess their capabilities as software developers.  The maturity level of an organization, with respect to its capability to develop quality software, is quantified on a scale from one to five, with five signifying the highest level.  In order to achieve a high overall rating, the organization must demonstrate that it implements the processes defined in each of the process areas (PAs) of the Model.  The model serves as a benchmark because it reflects processes that are evident in organizations that have repeatedly developed quality software on time and within budget.  One of the KPAs is “Measurement and Analysis”.  The processes within this PA are identified in Figure 5.  Although GQM is not specifically mentioned, the fundamentals of GQM are embodied within this PA.

Source: http://www.osqa.org/documents/siviy.pdf

Figure 5:  CMMI Measurement and Analysis Key Process Area

Goals Questions Indicators Measures (GQIM): GQIM, which emerged in the 1990s, is a 10-step process developed by the Software Engineering Institute (SEI) that helps organizations to identify and define software measures that directly support their business, process-improvement, and project goals.  In GQIM, the user defines indicators and measures based on what is needed to manage the User’s goals and decisions and decision criteria related to managing the user’s goals [Siviy 2004].  GQIM differs from GQM in that GQIM focuses on identifying Indicators up front rather than metrics.  Indicators address the presentation of data in order to optimize what is communicated to the recipient.  Indicators are typically high-level aggregates of measurement data, presented in graphical formats.  Some typical indicators are trend lines and circle graphs of normalized data.  In the SEI publication titled “Goal-Driven Software Measurement - A Guidebook”[Park 1996], the authors define an Indicator as “a display of one or more measurement results that is designed to communicate or explain the significance of those results to a reader.”  GQIM asserts that seeing how measurement data will be displayed (focusing on indicators) helps point to and clarify exactly what we must measure, better positioning us to develop the procedures for the data to be collected [Park 1996].   GQIM avoids use of the term “metric” because there is no standard meaning for it in the industry.   Figure 6 illustrates this slightly different focus of GQIM.

 

Figure 6:  The GQIM Paradigm as presented in [Siviy 2004]

 

Base measures are needed in order to build the Indicators.  GQM also builds indicators but they are often developed as part of the analysis function, which is performed later in the process after the metrics have been identified.  The two methods also differ in their stakeholder focus.  As noted in Figure 6, GQIM is primarily oriented to Senior Managers, while GQM is focused on involving the right people in the measurement process irrespective of the intended recipients of the measurement results.

 

Personal Software Process (PSP) and Team Software Process (TSP):  The PSP, introduced in the mid-1990s by the SEI, enables software engineers to plan their work based on personal data, to measure their work and use their results to continually improve, and to feel personally responsible for the quality of the products they produce.  The SEI TSP enables managers to institute a team environment that supports PSP work and to build and maintain self-directed teams whose members understand business and product goals and who produce their own plans to achieve those goals. The PSP and TSP are powerful tandem tools that foster the necessary skills, discipline, and commitment required for successful software development.  Collectively, these processes assist organizations to climb to higher levels on the software capability ladder, as defined in the SEI CMM and CMMI.

The CMMI identifies what processes are necessary ; PSP and TSP focus on how to implement those processes.

PSP was designed to show software engineers how to apply process principles to their work by infusing them with appropriate experience and exposure to advanced software engineering methods in a controlled training environment.  As part of their training, the engineers write a defined set of ten programming exercises and five reports, which gradually introduce the engineering methods, providing direct exposure to a new way of working.  They measure their own progress and get to see the effect of these new methods on their work [Humphrey 1999].

The focus is on developing estimating, planning and tracking skills and on five basic metric concepts, considered to be fundamental or core metrics for software development, namely, time/effort, size, quality, schedule and productivity.  Further details on PSP and TSP are available on the SEI website (see http://www.sei.cmu.edu/tsp/).

PSP and GQM are not conflicting methodologies; they are indirectly related through their alignment with CMM and CMMI.  Their purpose is different. PSP asserts what metrics are essential to the software development process and focuses on individual skill building, providing specific guidance on what data to gather and how it should be presented.  PSP focuses on improving one’s individual estimating skill with respect to time, effort, size and schedule, and developing procedures for deriving the estimates.  With GQM, such estimates would be identified as metrics in response to questions.  Using GQM, one would assume that those same metrics would be the natural result of the team’s identification of important measurement goals for their project but the GQM process itself does not require identification of those metrics.  Both GQM and PSP emphasize the importance of analysis of the data and presentation and communication of measurement results.  GQM is perhaps more closely related with TSP than PSP because of the GQM focus on involving the right stakeholders and sharing of perspectives and perceptions among the team.  An individual could certainly apply GQM at a personal level, applying it to their specific tasks within a project, and GQM could be applied by a team trained in PSP and TSP.

 

Practical Software Measurement (PSM):  PSM is an information-driven, issue-based approach to software measurement for project management [McGarry 2002].  This measurement process initiative, established in 1994, is funded by the DoD and is freely available at http://www.psmsc.com.  It has been endorsed by government Acquisition Executives, and is a key element of the OUSD (A&T) software initiative.  The methodology is also being adopted by government and industry organizations.

Highlights of the PSM process are:

• Its focus is on cost, schedule, and technical objectives
• It identifies project-specific issues
• Issues are grouped into common software issue areas
• Measurement categories correspond to issue areas
• Each measurement category has a candidate set of proven measures
• Measures are selected based on availability, environment, and other factors.

 

The PSM process defines and uses seven common information categories to facilitate the identification and prioritization of a project’s specific information needs [McGarry 2002].  They are:

• Schedule and Progress
• Product Size and Stability
• Product Quality
• Process Performance
• Resources and Cost
• Technical Effectiveness
• Customer Satisfaction

 

PSM differs from GQM in the following ways:

·         PSM is information-driven while GQM is goal-driven.  PSM does not ignore goals; rather it focuses on issues that result when striving to achieve a goal.  Many of the broad goals relating to cost, schedule and quality are assumed.

·         PSM focuses on using existing measures or metrics rather than creating new ones.  It does this by providing a pick list of measures for each of the various information categories, which is based on best measurement practices of DoD, government and industry programs.  GQM perceives each project as unique and therefore asserts that one should not arbitrarily adopt another project’s plan or measures.  PSM, however, advocates for tailoring the templates and plan to the specific environment.

·         PSM is primarily a tool for the project managers who work with a measurement specialist to establish the measurement program that meets their needs.  Other stakeholders, such as the development team are not directly involved in the planning although they are called upon to provide data and implement data collection procedures.  GQM seeks to involve the stakeholders in the entire process and focuses on the human and cultural issues involved in implementing a metrics program, operating on the principle that involvement at all levels is critical to a successful measurement program.

·         PSM is designed to help put measurement into practice at the project level, thereby providing the data for addressing enterprise level performance, process improvement, and business-related questions.  GQM has a more generic application in that it can be used wherever there is a need to assess satisfaction of goals, irrespective of the subject domain or the level.

 

Quality Function Deployment(QFD):  QFD is not a measurement program, but it is goal-driven as is GQM.  QFD is a systematic process for motivating a business to focus on its customers.  It is used by cross-functional teams to identify and resolve issues involved in providing products, processes, services and strategies that will more than satisfy their customers.  It is an organized, disciplined process for determining the product or service requirements necessary to achieve customer-perceived expressed or unexpressed quality.  It assists the developer with translating goals into measurable observable  product and process requirements and it does have a data collection component and an interpretation component.

 

Although QFD assists decision-makers by quantifying different strategies (using numerical values and weighted scores to help identify which alternatives might offer the best result), it is not a measurement program.  It helps the developer to determine what to do, but it does not measure progress or the results of strategies implemented.  It is more of a planning and estimating tool than a measurement tool.

 

GQM and QFD can be implemented together because they are complementary methods..  While GQM seeks to identify measurement goals and align them with pre-defined organizational goals, QFD focuses specifically on achieving the ultimate goal of customer satisfaction by listening to the customer and soliciting their input.  In QFD, the customer drives goal setting; QFD provides the guidance for prioritization of customer satisfaction goals while GQM provides guidance for identifying appropriate measures and indicators of progress toward those goals.  GQM focuses on how to assess progress toward goals by addressing the questions used to identify the measures; QFD establishes performance thresholds necessary to reflect goal attainment.

Figure 7 presents the visual structure of the first phase of QFD.  When completed, it resembles a house structure (or series of houses) and is often referred to as the “House of Quality”.  The House is divided into several rooms.  Typically, there are customer requirements, design considerations and design alternatives in a 3-dimensional matrix to which one can assign weighted scores based on customer information collected.  QFD originated in Japan in the late 1960s as a quality assurance method that would design customer satisfaction into the product before it was manufactured.  Prior methods were primarily aimed at fixing a problem during or after manufacturing.  The QFD methodology was embraced by a wide variety of industries in the U.S. and Western Europe in the 1980s because of its flexibility and comprehensiveness, and it continues to be used today.  Although it can be applied to software, there are few instances of such implementation in the open literature.

 

Figure 7:  Sample Structure of a House of Quality (QFD Phase 1)

Cross-functional teams typically participate in a four-phase process that consists of matrices that analyze data sets according to the objective of the QFD process.  These matrices, connected together using priority ratings from the previous matrix, are used to translate higher level needs into lower level hows, product requirements or technical characteristics to satisfy the needs.

Further details about QFD and its history are available on the QFD Institute web site (see  http://www.qfdi.org/ ).  See also the DACS Gold Practice titled “Acquisition Process Improvement” (http://www.goldpractices.com/practices/api/#detail ).  The section about Acquisition Process Improvement Techniques describes how QFD could be used to translate customer requirements into a successful acquisition.

 

Quality Improvement Paradigm (QIP)

QIP originated at the NASA Software Engineering Laboratory in the late 70s.  It provides a bottom-up approach for software improvement through experimentation and analysis at the project and corporate levels.  Lessons learned from QIP in the late 70s demonstrated that data collection needed to be goal driven and provided the impetus for the development of GQM in the 80s.  Both GQM and QIP evolved throughout the 1980s into formalized paradigms.  QIP is based on six essential activities [Basili 2005]:

1.       Characterize the current project and its environment with respect to models and metrics.

2.       Set  quantifiable goals for successful project performance and improvement.

3.       Choose the appropriate process model and supporting methods and tools for this project.

4.       Execute the processes, construct the products, collect, validate, and analyze the data to provide real-time feedback for corrective action.

5.       Analyze the data to evaluate the current practices, determine problems, record findings, and make recommendations for future project improvements.

6.       Package the experience in the form of updated and refined models and other forms of structured knowledge gained from this and prior projects and save it in an experience base to be accessed or reused on future projects.

GQM is integrated within the QIP in that it provides the practical guidance for setting measurement goals, identifying the right metrics, and establishing the measurement program that is needed to assess goal attainment.  The same GQM principles are applied both within specific projects and at the organizational level.  Further description of GQM within QIP is provided in  the body of this document.

 

Software Quality Metrics (SQM): SQM is a method formalized by G. Murine [Murine 1983] in the 1980s, based on earlier work by Boehm [Boehm et al.  1978] and McCall [McCall et al. 1977]  Murine describes SQM as follows:

"A method of providing improved management visibility into software development using a metric measurement approach.  ...  Software is defined to include code, data, and documentation. Each of these parts is measurable using a methodology called Software Quality Metrics (SQM). SQM defines specific, high-order, management oriented characteristics (factors) of ‘good’ software which are measurable in terms of their components or ‘criteria.’  The criteria scores, in turn, are determined by measuring individual, countable occurrences called elements. This measurement process may be applied at each of the software’s developmental phases--Requirements Analysis, Design, Code, and Test.  Product (software) status is reported at either the factor or criteria level. Discrepancy status reporting is included."

SQM was developed to allow the customer to assess the product being developed by a contractor.  SQM instructs users to evaluate the quality of software by choosing high-level factors, refining those factors into criteria, and refining those criteria, in turn, into metrics.  The SQM approach defines a fixed set of 12 factors, 23 criteria, and over 400 metrics.  Some emphasis is placed on scoring techniques that will yield a single number representing the “goodness” (author’s terminology) of an artifact.  Murine states that the data gathered for the metrics must be interpreted in light of the criteria and factors, an argument for viewing SQM as a top-down approach to measurement.  However, he also states that the appropriate life-cycle phase and documents must be considered beyond the factors and criteria in order to choose metrics, which suggests that the factors and criteria alone are insufficient for the selection of metrics.

The scope of the SQM approach is limited to assessing the conformance or nonconformance of software products with respect to the predefined factors.  The 12 factors are:

Correctness                                 Reliability

Efficiency                                     Integrity

Reusability                                   Usability

Maintainability                              Testability

Flexibility                                     Portability

Interoperability                             Intraoperability

SQM seeks to produce a single score for each factor.  Factor scores (e.g., 90% correct or 85% reliable) are reportable to management and thus provide a basis for quantifiable software status.

Because the set of factors is fixed, SQM does not support the definition of user-specific measurement goals, as does GQM.  In addition, the predefined refinement of the factor, criteria, and metric hierarchy precludes any tailoring of the refinement hierarchy, which makes this method significantly different from GQM.