SOFTWARE ACQUISITION GOLD PRACTICE

METRICS-BASED SCHEDULING

FOCUS AREA:        MANAGEMENT - Measurement

Description of the practice:

Summary Description

Metrics-based scheduling is about establishing realistic software development or maintenance schedules based on accurate estimates of software size and effort.  The practice necessitates use of a minimum set of four metrics (namely, software size, effort, time/schedule and quality) coined by the Software Engineering Institute (SEI) as the “core” metrics for software development, along with a “productivity” metric that is derived from the core metrics and is often referenced as the fifth core metric [Putnam, 2003].

 

This practice is applicable to both new software development and maintenance projects.  Its scope is very broad, in terms of both content and implementation, since realistic scheduling is contingent upon the successful integration of several other principles, processes and practices supported by the appropriate infrastructure and data collection activities.  These include:

 

• Establishing a goal driven metrics program that supports the organization’s goals as well as individual project goals
• Implementing good requirements engineering practices
• Getting accurate estimates of product size early for use in developing other estimates and tasks that will comprise the project baseline and verifying the feasibility of the project given the environmental constraints
• Storing estimates, along with project metadata in a project repository for use in future estimating and planning
• Capturing and quantifying process productivity
• Implementing risk management early

 

It is seldom successful when implemented in an intermittent fashion.  It requires a management commitment at the organization level that crosses project boundaries, although it may be started with one or two pilot projects.

 

It is important to have valid core metric data (definitions and details provided in the body of this document) from a few completed projects, or access to appropriate benchmark data, in order to begin to develop confidence in the use of metrics for planning and decision-making.  This takes time (the development cycle of the projects) unless such data from previous projects already exists in an accessible form.  Thus, a two or three-year window is a reasonable timeframe for establishing and institutionalizing this practice if you are starting from scratch (with no historical data).  Global repositories of project information exist and can be used to jump-start the process in cases where there is little historical project data within the organization.

 

Estimation of software size and effort plays a significant role in metrics-based scheduling because these metrics are generally used for estimating the cost and schedule as well as planning the development and establishing a baseline.  Estimation accuracy can thus have a major impact on the ability of the organization to establish and then meet its contractual obligations of delivering a quality product on time and within budget.  Much of the success of this practice stems from managers and other decision-makers developing confidence in the validity and accuracy of the core metrics they are using.  The data needs to be strong enough to allow them to make critical decisions relating to the feasibility of what is being proposed and to understand and communicate the risks associated with such decisions.  Accurate project data, from past and current projects, provides the significant evidence they need to support their decisions or recommendations.

 

Another important concept of metrics-based scheduling is understanding the interdependencies of the core metrics.  These interdependencies are often referred to as “the software equation”, a mathematical equation supported by actual project data collected from thousands of projects for a period of more than 25 years.  Refinements and variations (including constants, factors and exponents) of the software equation form the basis for most of the parametric software estimation models used today.  Each model makes a set of assumptions, and requires some initial input that is related to the subject project, or the development environment in which the project will occur.  Therefore, it is important to understand these aspects of each model in order to determine which models are appropriate for your particular situation.  The body of this document contains a more detailed discussion of the software equation and its various implementations in parametric modeling.

 

The literature cites the following benefits from implementing metrics-based scheduling.

• Increased likelihood of project success (quality product, on time, and within budget)
• Satisfied customers
• Improved communication among stakeholders
• The capability to acquire software on the basis of ‘cost per delivered functional unit (in contrast to a fixed price contract)

 

Some of the key practices, identified in this document are summarized as follows:

• Do not let the lack of a formal scope specification stop you from doing an initial project estimate.
• Merely having core metrics does not guarantee success.  You must use them for guidance and decision-making.
• The core metrics are interdependent, and should always be addressed together.
• Every software project has a minimum development time but there is no maximum development time.
• Work to realistic project expectations
• Recognize the impossible; be prepared to stop projects
• Recognize and use industry specialists such as estimation (software sizing and cost estimating) and domain experts
• Consider acquiring software on the basis of cost-per-unit-delivered
• Establish a metric mindset
• Monitor estimation accuracy
• Define measurements in a top-down fashion
• Involve the stakeholders (participants to the measurement process)
• Inform/communicate with the team

 

 

Detailed Description

 

In 1998, the Software Program Managers Network [SPMN 1998] identified Metrics-based Scheduling and Management as one of 16 practices critical to successful software development and provided the following definition:

“Metrics-based scheduling and management is concerned with making cost and schedule estimates based on empirical data and then monitoring the project for process control as it progresses using both product and process metrics.  An important goal of the practice is to identify problems early while there is still time to resolve them or make adjustments”.

 

Thus, the scope of the practice, as defined by the SPMN, is very broad, in terms of both content and implementation, since it encompasses both planning for and management of the project throughout the software development life cycle.  This practice has dependencies with many other effective practices, which are the essence of project management, as illustrated in Figure 1.

 

The DACS has elected to break the content of the original practice, defined by SPMN, into two separate but related Gold Practices.  One practice, the subject of this Gold Practice, which is called Metrics-based Scheduling, focuses on the estimating and planning activities that are necessary to create a realistic schedule and baseline for a project.  The other, called Metrics-based Management, focuses on the project monitoring and control activities that are part of project execution.

Figure 1: Context Focus For Metric-Based Scheduling

All of the project management activities identified in Figure 1 are interrelated (as noted by the dotted connecting lines) but the focus (marked as 1 in Figure 1) of Metrics-based Scheduling is primarily on the relationship of measurement and analysis to the estimating and planning activities that occur in the initial stages of the project and produce the project schedule.  Measurement and analysis, along with the outputs of estimating and planning, (see primary relationships marked two and three in Figure 1), in turn, affect project monitoring and control during execution, which are the focus of Metrics-based Management.  Other relationships and influences, such as requirements management (engineering) and risk management, are discussed as appropriate later on in this document.

 

Metrics-based scheduling is scheduling based on accurate estimates of effort, which are, in turn, based on estimates of software size (the amount and complexity of the software work product) and an understanding of organizational productivity.  It is knowing when a proposed completion date is not possible and having appropriate data to prove it.  It is forward scheduling based on reliable size and effort estimates rather than backward scheduling driven solely by a desired completion date or other constraint.  It is reputation building; it is a mindset that approaches scheduling based on the size and complexity of what is to be built, while considering the requested completion date as a constraint. It is being able to predict with confidence the impact of decisions made with respect to product or process.  It requires having confidence in estimates of the core metrics (size, effort, quality and schedule) and the productivity measures that are derived from the core metrics of past projects.

 

Perhaps the best way to describe metrics-based scheduling is to contrast it with time-based scheduling (sometimes called backward scheduling or reverse scheduling) as presented in Table 1.  Some may argue that all scheduling is, by definition, metrics-based, because it involves scheduling activities over a period of time.  Although time is a metric, when time is not derived from or related to the size, or the effort required for developing the software product, then such scheduling is not metrics-based.

 

Table 1  Comparison of Time-based and Metrics-based Scheduling

Time-based scheduling	Metrics-based scheduling
Scenarios
The customer wants the project completed by a specific date.  Developer commits to the completion date.  Scheduling progresses back from the finish date, theoretically, along a critical path. The result is that the schedule dictates how much time is allowed for developing each of various functions or components of the system, rather than any analysis of effort required based on the functionality to be delivered. The developer expects to add more resources as necessary to ensure that each component is developed within the scheduled time period.  Managers may expect staff to work additional undocumented hours to ensure adherence to the schedule during development. Typically, this type of scheduling is associated with a fixed cost that includes a reserve to accommodate unforeseen needs such as additional resources.	The customer wants the project completed by a specific date.  Contractor analyzes the product requirements to determine an estimate of size.  Then, using process productivity data, or other data from past projects, the contractor estimates the effort (person hours) required to complete the project.  Once the effort is known, the contractor develops the critical path and various scheduling approaches to see if it is feasible to complete the project by the date requested by the customer, and at what cost and risk.  If yes, contractor reviews with the customer the various approaches with respect to risk and cost and they reach agreement about the scheduling strategy.  If not, the contractor goes back to the customer to present the reality of the situation and perhaps negotiates less functionality or an incremental release strategy instead.
Effects
May work with small projects of short duration where accelerated work schedules will be acceptable to the staff for short periods Provides no basis for determining overall productivity unless the organization tracks actual size and effort (including uncompensated effort) Breaks down as the product size and complexity increases Is often perceived as necessary to sustain business May result in loss of expertise over time because skilled software people do not wish to remain in such an environment	Effectiveness depends on having accurate size and effort metrics for past projects, or benchmarking against a repository of similar projects within the same domain, in order to determine process productivity Works for projects of all sizes Provides early identification of schedule problems Provides realistic time period for accomplishing specific tasks of the project Establishes good customer relationships Builds a reputation for delivering quality products on time Reduces project risks (cost and quality) Supports reduction of staff turnover

 

Metric-based scheduling is part of a management strategy developed with a metric mindset that is committed to using appropriate metrics for management decision-making and control of projects of all sizes.

 

Activities of Metrics-based Scheduling

 

Metrics-based scheduling consists of planning and estimating activities, which involve early calculation (estimation) of size and projection of costs and schedules from empirical patterns, in order to verify the feasibility of the project and to establish a realistic project baseline and plan for moving forward.

 

These planning and estimating activities represent a subset of the activities described in the Project Planning Process Area of the CMMI [Chrissis, 2003], which are listed in Table 2.

 

Table 2: Estimating and Planning Activities Identified in the CMMI Project Planning Process Area [Chrissis, 2003]

 

Although all of the activities identified in Table 2 are important to project planning, the activities of estimating project scope (SP 1.1), establishing estimates of work product (SP 1.2), establishing estimates of effort and cost (SP1.4), schedule and budget (SP 2.1), and developing the project plan (SP 2.7) are the focus of this Metrics-based Scheduling practice because they involve estimating the core metrics and using them to develop the resulting project baseline.

 

Figure 2 presents a detailed process flow of these planning and estimating activities to show the sequence in which they are performed along with their inputs and outputs. 

 

As illustrated in Figure 2, requirements influence all of the estimating activities.  They drive the scoping of the project and provide the facts for estimating work product size.  Quality requirements influence the design and testing and therefore influence the effort, scheduling and, ultimately, project costs.  Constraints (a form of requirements), such as a required completion time, influence the quantity of resources and the distribution of effort that, in turn, affects the cost.

 

The parameters of project planning include all information needed to plan, organize, staff, direct, coordinate, report and budget.  Estimates of these parameters need to have a sound basis in order to instill confidence that plans based on them will actually support project objectives.  Thus, when estimating, it is important to document the estimating rationale and the data used to support the estimate.

The outputs of the planning and estimating activities are the elements that comprise the project baseline (WBS, size and effort estimates, project schedule, budget) and the project plan as illustrated in Figure 2.

The following paragraphs discuss each of these planning and estimating activities in detail.

 

Step 1 - Estimate Project Scope

Estimating project scope is the process of translating requirements into actions or tasks, by, analyzing known requirements and identifying a collection of tasks that are necessary to develop a product that meets the known requirements.  This is not the same as product design.  It is a higher-level delineation of tasks, which includes product design as one or more of the tasks.  Note that ‘project’ is not the same as ‘product’ and they should not be used interchangeably.  Some project tasks are necessary to complete the software product but are not part of the product and are not reflected in product size.

The developer builds a top-level Work Breakdown Structure (WBS) that defines in essence, what work has to be done to satisfy the known requirements for a software product.  Requirements include product requirements, organizational requirements (or constraints), customer requirements (or constraints) such as funding or schedule and any other requirements or constraints (whether formal or implied) that might affect the project. The WBS in turn, provides the basic structure to support initial estimates of the size and complexity of the product and for estimating the effort required to complete the project.

The WBS divides the project into a hierarchical set of manageable components, which are logical units of work called ‘work packages” that are the basis for assigning effort, schedule and responsibility and provide the underlying framework for planning, estimating size, executing and controlling the project.

The primary purpose of the WBS is to help gauge the reality of the project work effort.  It needs to be sufficiently detailed to enable realistic effort estimates and schedules thereby minimizing the need for management reserve, the pot of resources allocated to accommodate the ‘unforeseen’.

Estimating scope should be done in concert with defining the project life cycle.  Typically, this is done by selecting a software development model to address interdependencies and appropriate sequencing of software project activities.  The model selected depends on the scope of requirements, estimates of project resources, and the nature of the project.

Most projects need to tailor the selected model to address the particular needs/constraints of the project.  The important issue is that project leaders understand why a particular model should be used and distinguish between the theoretical model and their customization of it.  The choice and tailoring of life-cycle model have an impact on the structure of the WBS. 

Step 2 - Estimate Work Product Size

Software size is a metric that seeks to answer the question, “How big and how complex is the software product?” by quantifying various physical and/or functional attributes of the product as identified in Table 3.

Table 3: Comparisons of Physical and Functional Size Measures

Physical Size Measures

Functional Size Measures

Physical size metrics count some physical entity associated with the software product such as Source Lines of Code (SLOC), number of requirements statements, number of software configuration items (CSCI), pages of documentation or other entities. SLOC are sometimes used as a measure of the “length” of the software.

Functional size metrics quantify the functionality and complexity of the software product based on a standard or widely used consistent set of counting guidelines.  The base measure is a Functional Size Unit (FSU) and each specific functional sizing methodology identifies its FSU with a unique name.  For example, IFPUG, the predominant functional sizing methodology for business applications, calls its base measure a Function Point (FP).  More recently, a methodology called COSMIC –FFP has emerged that was designed for quantifying the functionality of real time and embedded systems.  Its base measure is called COSMIC FSU, ( CFSU). Even though the FSU serves as a standard unit of measure, this does not mean that a single FSU maps one-to-one with a specific software function, or that the FSU of one method is equivalent to the FSU of another method.

 

Because we have evolved the way we build software, the physical size measures (particularly SLOC) may not be as useful as they once were in many organizations.  The boundaries of a software product have expanded beyond the organization.  We 

• Use multiple programming languages in the same project
• Use code generators rather than write the code
• Develop pieces of software systems in concert with other organizations
• Reuse code
• Integrate existing software components or systems rather than develop software from scratch

Many organizations are migrating to Functional Size Measurement for the following reasons:

• It works for all types of software (scientific, business apps, web portals, embedded systems, etc.)
• It works for all types of projects (new development, enhancements, maintenance, etc.)
• It is language independent
• It is technology independent
• It is repeatable(consistent) – two analysts independently estimating the same software product, using the same resources and the same sizing methodology, will arrive at the same size within a few percentage points (± 5%)
• It produces statistically significant results
• It can be applied early in the development life cycle

Although the concept seems simple, estimating size is really quite difficult and often requires the engagement of a skilled professional who is experienced in using the selected sizing methodology.

Size is the primary input to many models used to estimate effort, cost, and schedule, but it is also an estimate.  Therefore, achieving accuracy of the size estimate is critical to the estimation process since so many attributes are based on it.  Size is also an important part of the functional baseline because the product is affected by requirements changes. Tracking size throughout the project, particularly when there are major requirements changes, provides visibility into and heightens awareness of scope creep. Awareness can trigger better control over the project by adjusting effort, schedule, and/or cost before the project reaches a crisis state.

Having an accurate size estimate enables the developer to better estimate effort and cost earlier and with fewer resources.  However, the size estimate, by itself, is of little value. We must use it together with other project attributes and historical data from previous projects.  Projects of similar size will require similar effort when other project attributes match.  The correlation between product size and effort might be strong or weak depending on the match of other project attributes, but as an organization builds its repository of project metadata, it is possible to infer the strength of the relationship based on key project characteristics.  Although it takes time to achieve this state, the benefit is that of achieving sound effort and cost estimates based on accurate size estimates in a minimal amount of time.  In addition, the project repository can be used to estimate the size of a new project based on comparisons with previous projects.

 

The critical elements in choosing the size metrics your organization should use are:

• Accuracy – Over time, your estimates should come closer and closer to the actual size determined at project completion.  Accuracy should get better with each new project.
• Consistency – Choose size measures that can be applied consistently across all projects, now and in the future.  If you are no longer using Ada, having size estimates in terms of the number of SLOC for Ada is not very useful unless you can convert them to an equivalent size relating to the language you are currently using.  If you are using multiple languages, or doing COTS integration, SLOC is probably not as useful as a functional size metric.  If you will never have a need to benchmark against external organizations, then focus on achieving consistency within your organization.  The key is to identify the size metric(s) that provide the best correlation with effort within your organizational culture and development process.
• Predictability – Having an accurate size metric should enable you to predict the effort and cost when used in conjunction with other project data, and to know the degree of correlation. Track the correlation as you proceed.  Functional size may be a good predictor in your particular development environment, but not in another organization.  You have to focus on finding what works for you.  It may be that you need more than one size metric.
• Earliest usage -- Choose a size metric that you can use early in the project with reasonable accuracy.  This allows you to assess the feasibility or reality of completing the project and meeting customer expectations before you have invested significant resources in the project.  Having a size estimate is of little value for project planning if we have to wait until after the design phase is complete, or after testing before we can measure size.

Unfortunately, in many organizations, developers often skip the sizing activity during initial planning.  They may develop the WBS asserting that it reflects the product size through the task components that identify software functionality to be developed.  Many developers tend to equate size with effort or cost.  In fact, they seldom refer to product size at all.  If one asks, “How big is the product?”, both the developer and the acquirer are likely to answer that it is a 3-year effort, or a $2 million dollar project.  They are not likely to say that it is estimated to be 600,000 SLOC or 2,000 functional size units (FSUs).  Many developers perceive that a quantitative size estimate is not obtainable in the initial stages of project planning.  Acquirers concerned with the cost to meet the requirements rather than how functionality is quantified, may perceive early sizing of the product as a waste of time.  The developer focuses on building the WBS and using that to estimate effort and schedule.  Many organizations capture a product size estimate only upon project completion.  The mindset described in this paragraph needs to be overcome if you elect to implement metrics-based scheduling.  Although the entire project team needs to be aware of the product size and how it might be changing, the project should use skilled professionals who are trained in the appropriate size estimating techniques to do the estimating functions.  This results in estimates that are more consistent over time.  Details about specific estimating techniques and estimating practices are addressed under the ‘Estimating Techniques’ section of this document.

 

Step 3 - Estimate Effort

Effort is the amount of work expended in order to complete the software; it is typically measured in person hours (PH), person months (PM) or person years (PY) depending on the scope of the project and the granularity of information that is available.

Note, at this point in the process, our effort estimate will be a nominal estimate, an estimate that is “in the ball park” but not exact;  it assumes variations of staff experience and skill exist.  Its purpose is to assist with determining both schedule and cost to assess the feasibility of the project.  We may know that if we used all highly experienced staff, they might do the work in less hours than an inexperienced staff would take, but we also know that they might cost a great deal more.  That is not the intended purpose of this initial effort estimate. The distribution and allocation of effort among staff resources is addressed as part of scheduling.

There are several ways to estimate effort and the methods you choose will depend on the availability of information and its accuracy.  You should not rely on a single method but instead estimate using at least two methods and compare the results.  Major differences between estimates should trigger investigation as to why they differ, and some resolution before going forward with using the effort estimates for deriving a schedule.  Effort estimation approaches, described in the literature, typically fit into one of the four groups identified in Table 4.  The best way to determine the effectiveness of your chosen method(s) is to monitor the estimation accuracy as you move forward with projects.  You may find that what works best for you is to tailor one of these approaches, or create a composite of these approaches.  Always keep in mind that the effort estimate, like any other estimate, is not free.  Resources must be expended to derive the estimate and those resources translate into time and cost.  It may not be efficient to expend significant time and resources to get an estimate that is extremely accurate, at a time when you do not need that degree of accuracy.  The intended use of the effort estimate should always drive the decision regarding the appropriate estimating methods.  It is noteworthy that with the exception of the bottom-up method, these approaches depend on having some data and/or knowledge of previous projects.  The accuracy of the effort estimate depends on the accuracy of the historical data to some extent.

Table 4:  Effort Estimation Approaches

Approach	Description	Usage Considerations
Top-down	Comparing high-level project composition with previous or similar projects and using the effort from those projects as a starting point.  Adjust the effort up or down based on expert judgment of the risks and differences between the current project and the selected comparison projects.	Easiest and perhaps most acceptable from the perspective of the technical staff, but its value depends on the amount and accuracy of the metadata for projects in the comparison, and the ability of the estimator to make adjustments consistently. Subjective since it depends on the domain knowledge of the estimator and the organizational culture. Breaks down as projects increase in size and complexity
Bottom-up	Detail is added to the WBS to decompose the project into tasks of anticipated short duration. Then individual developers estimate the effort to complete each task.  The rule of thumb is that if a task takes longer than two weeks it should be further decomposed. The aggregate of the individual task estimates is used as the project effort estimate.	Use when historical data from past projects is not available. Use as a reality check against any of the other methods. Studies show that developers tend to underestimate when using this or any manual approach. The organizational culture can impact how individuals estimate, and thus, the validity of the estimate (see Estimation Games)
Parametric Modeling	Formal implementation of the Software Equation using a tool that accepts project size, complexity and other characteristics, and calculates the effort based on a historical relationship among the characteristics of past projects from many organizations.	The tool must be calibrated for the unique characteristics of this organization. Requires skilled estimation specialist with experience using the tool and a thorough knowledge of the metric data collected by the organization
Derivation from Effectiveness Parameters	Applying a version of the software equation to calculate the effort using the size estimate of the current project, and an established productivity rate (or comparable effectiveness measure) from previous projects with similar characteristics	Requires the least time and resources of all approaches Usefulness depends on the strength of correlation between size and effort Depends on the validity of the productivity rate, which is derived from accurate measures of actual effort for past projects and consistent size measures gathered on completion of each project

 

Confidence in the estimates is based on the rationale for the estimation model selection and the nature of the data.  Sometimes, historical data does not apply, such as where efforts are unprecedented, the type of task does not fit available models, the project leadership has changed, or the development environment has changed.

There are now some worthwhile methods to derive accurate effort estimates easily and quickly if your organization collects functional size measures on its completed projects along with project metadata.  It is important to identify the risks associated with using the selected estimation model and the uniqueness of the project to ensure a common understanding of any assumptions that are made in the planning stages of the project.

Ideally, when the organization has sufficient history to derive organizational productivity rates for projects based on specific attributes, the effort estimate can be calculated from the size, but this presumes a consistent mature software development process is in place.

If I estimate the size of this software product to be 1000 FSUs and, historically, my team has produced at a rate of 8 FSUs per person month in this environment, then, with simple arithmetic (1000/8) I can calculate a realistic effort estimate in a matter of minutes.  As a starting point, I can estimate it will take 125 person months to develop this system.  This estimate can be used as a consistency check (reality check) against other effort estimating techniques. If a bottom-up estimate of effort results in only 75 person months, then it should trigger the need for further investigation before committing to a baseline since the difference in the two estimates is significant and a significant difference in effort results in a significant difference in costs as well.

Studies have shown that we (practitioners) tend to underestimate the effort when we are using manual (top down or bottom up) approaches.

If you have been involved in tracking effort, but not estimating it initially, there is a tendency to mistrust the effort estimate, perceiving that perhaps it does not account for the variation of individuals within the team.  You might tend to think that if we used more of person A and less of person B the effort would be less because person A is more experienced than person B is.  Remember that the productivity rate you chose to use was based on actual past projects, with similar or same staffing scenarios and the same environment.  Therefore, that variation is already accommodated in the productivity rate used for the estimate.

You may find that the reality of your situation is that you (the project manager) are given the funding amount and instructed to make the effort estimate fit the funding. You allocate the funding among the tasks of the WBS and adjust the effort in each case.  Then, during project execution, effort is tracked in accordance with this baseline. The project team may be working many additional hours to complete the project but the extra effort is never recorded and not visible once the project is complete.  There is a big difference between the actual effort and the documented effort.  If the documented effort is stored in the project repository and not the actual effort, it can skew the project team and organizational productivity metrics and render them useless. 

There must be a commitment across the organization to capture and record real/actual effort data if your project repository is to have any real value.  Your metrics repository is not a showcase of marketing tidbits. It is a tool for decision-making. Your organization needs to take the responsibility for ensuring that the data posted there is accurate and factual, not inflated.

Step 4 - Estimate Schedule

Scheduling involves distributing an effort estimate among staff resources and across a calendar timeframe to ensure that both product and performance or operational requirements will be met.  It is also the process of planning the appropriate level of staffing.  If our scheduling analysis demonstrates that we need a minimum of eight developers, but we only have five, we know early on that we must adjust the staffing allocation or risk failure.  Schedule is typically expressed in calendar weeks, months, or years, depending on the scope of the project and the desired level of granularity. 

One can use the following technique as a reality check against schedules derived by other means to alert the planner to major inconsistencies early in the planning stages before committing to a particular schedule baseline.

The metrics-based effort estimation process (based on product size) provides us with a nominal effort estimate, perhaps bounded by a meaningful range based on the level of risk, or the uniqueness of project attributes (125 person months + or – 10%).

Assuming a size-related effort estimate of 125 person months, if our team consists of five people, and they each work the same amount of time concurrently, then the minimum calendar schedule would be 25 months.  This is typically the minimum schedule because it assumes that (a) the software critical path can be accomplished by this scheduling scenario and that (b) the experience mix of the staff can address all of the project tasks.  In addition, if any one person works less than full time, the schedule would have to be stretched to offset the additional time of another person.  This is the starting point for scheduling, but more importantly, it provides a reality check against any specified time constraints against the project such as a customer requirement or expectation that the project be completed within 12 months.  This tells us immediately that it is not possible for us to complete the project within the expected timeframe, given the current staffing plan.  Having just three metrics (realistic size estimate, valid project team productivity rates and the number of staff) enables us to get the schedule estimate in a matter of minutes.  It is not intended to be exact.  Its purpose is to let us know the minimum schedule and provide a starting point for refinement.  We would next have to ask, “If we double the staff could we accomplish the project within the 12-month window?” That answer would come only from analysis of the software critical path and the skills and experience needed to perform the tasks.

It is extremely important to define for your organization exactly what constitutes a person hour, person-week and person month.  That information needs to be standardized across the broadest possible range of groups within the organization.   Assuming a person hour is defined as one person working on the project for one hour, then a person-week is likely to be viewed as a person working 40 hours on the project, and a person-month is one person working 160 hours on a project (40 X4).  However, is the person who is full time on the project really working on the project for 40 hours?  Are they spending perhaps one hour a day reading email or other miscellaneous tasks that are not totally tied to the project?  What happens to the person month concept when holidays occur? Long weekends?  Vacations?  How many person hours of each month are realistic. Uncompensated overtime is also a big problem.  In many organizations software professionals are expected to work overtime (uncompensated) whenever there is pressure to meet a deadline or milestone.  Because it is uncompensated, it is usually not tracked or recorded.  Thus, it is never reflected in the effort data reported for a project.  The assumptions related to the schedule metrics need to be documented and stored with the data.  If your department uses 160 hours as the standard person month, but another department uses 176 hours as the standard person month, then there is an impact on scheduling caused by the variation of metric data used to derive the schedule.

Many parametric models have been developed to aid in estimating cost and schedule; they are based on historical data from many projects that may or may not be pertinent to your project.  Because of this, it is important to use multiple models and compare results to ensure a higher level of confidence in the estimate.  Ultimately, any such models should be calibrated using your own repository of historical data.  This is an area where it may be appropriate to bring in an industry specialist, trained to work with such models to maximize their value for decision makers.

Obviously, there is more to scheduling than the simplistic perspective presented here but the focus of this practice is in knowing when and how to use the core metrics to derive the initial schedule and to create this valuable comparison point for assessing the validity of any schedule that might be derived from other methods.

The larger the project (product plus related tasks) the more important it is to establish this initial metrics-based schedule as the starting point for further schedule refinement.

Step 5 - Estimate Costs

In metrics-based scheduling and management, costs are derived from effort and schedule estimates which, in turn, are based on software product size estimates combined with historical project data.

Note that in this planning context, the planner wants to estimate as accurately as possible what the actual costs will be.  This is not necessarily the same as a cost estimate performed for bidding purposes.  The cost drivers that apply to bidding estimates tend to address a particular business strategy.  Hopefully, the two types of cost estimates are well aligned, but they serve different purposes.  The purpose of the estimate should be carried along with the estimate in the project repository.

Just as with effort and schedule, an accurate cost estimate enables the planner to determine the feasibility of meeting cost constraints imposed for the project.  If the cost estimate is $500,000 and the acquirer has set a ceiling of $200,000 then it is highly unlikely that the developer will be able to bring the project in on time for $200,000 without sacrificing quality and/or functionality.  Having this awareness early in a project, and having supporting evidence ( based on the body of historical data which influence the estimates) enables the planner to communicate the reality of the situation to the leaders, and the leaders to make informed decisions about whether to proceed, and if so, how to proceed.  Typically, when there is sound evidence and confidence in the estimating process, the acquirer and developer work together to de-scope such a project to establish a more realistic set of requirements, or to adjust the development cycle in some way.

There are many cost estimating models and approaches to consider and some are most effective when used by a skilled professional.  Generally, the larger the project, the more important it is to employ a skilled professional to do the estimating. References for cost estimating techniques are provided later in this document along with additional reading.

Step 6 - Do a ‘Reality Check’

The reality check essentially asks the following types of questions:

• Are the estimates we have developed realistic?
• How do our estimates align with budget availability?
• Do we have more information now that should trigger re-estimating of some elements?
• Have we considered all the risks and can we live with the level of risk that remains?

This is a critical decision point in the process and will often uncover the very elements that are likely to cause a project to fail at a later point in time.  It is far less costly to perform the reality check than to wait in anticipation of resolving issues during project execution.

Note that you may need to negotiate functionality and repeat iterations of steps 1 – 5 as needed to establish the estimates and WBS that comprise the Project Baseline.  When the baseline is established you can proceed with developing the Project Plan, or, you can evolve the Project Plan as you proceed through the necessary iterations.

One should note that the acquisition process and the contracting policies may actually be barriers to metrics-based scheduling in that they might not have provision for making adjustments based on the reality check.  Consider alternative acquisition strategies and contractual arrangements such as performance-based contracting that provide greater flexibility for making realistic adjustments.

Step 7 - Develop the Project Plan

The Project Plan is a formal, approved document that the developer uses to manage and control the execution of the project.  It is based on the project requirements and the established estimates.

It documents and ties together all activities that are needed to ensure successful development of the product.  It serves as a guide for all project participants and promotes a common understanding of how the project will progress and what processes are to be applied.

The project plan includes the initial budget and schedule that is based on the developed estimates.  Sometimes the schedule is event-driven rather than calendar based. Schedule-driven is most useful for projects that you have done before (for another client, for example), and everyone on your team knows exactly what to do to get the job done. Schedule-driven tends to ensure that things get done according to a known schedule, on time, and on budget, but it is problematic when a project involves a lot of uncertainty.

Event-driven schedule is required for projects that have never been attempted before or that involve a lot of uncertainty. Attempting to stick to a pre-defined calendar based schedule on such a project is very difficult, as new and unforeseen obstacles tend to pop up and slow your progress, leading to continual adjusting of the schedule, and confusion among your client(s). Event-driven methodology breaks the project into sub-tasks, and delineates the order in which sub-tasks must be accomplished before moving on to the next (subsequent tasks are usually dependent on the completion of preceding tasks). An event-driven methodology does not provide a fixed schedule or completion date, but it does tend to ensure that a complex job with a lot of uncertainties is done right the first time.    It is effective in dealing with project risk, especially when resources are limited.

The plan thus details schedule dependencies as well as budget.  It identifies major milestones, schedule assumptions, constraints and task dependencies.

The schedule and budget are tied together because the budget must support the resources that are allocated to complete the tasks in the schedule.  Establishing the project’s budget and schedule includes the following:

• Defining the expected availability of resources and facilities
• Determining time phasing of activities
• Determining a breakout of subordinate schedules
• Defining the dependencies between activities
• Defining the schedule activities and milestones to support accuracy in progress measurement
• Identifying milestones for delivery of product to the customer
• Defining activities of appropriate duration
• Defining milestones of appropriate time separation
• Defining a management reserve based on the confidence level in meeting the schedule and budget
• Using appropriate historical data to verify the schedule
• Defining incremental funding requirements
• Documenting project assumptions and rationale

 

An important part of planning, which sets the stage for effective Metrics-based Management activities as the project progresses, is to allocate an appropriate portion of the budget to each work unit identified in the WBS.  This allows tracking of progress by the accumulated “value” of completed work units, thus integrating actual work performed with planned and actual spending.  This is called “Earned Value Management”.  Although tracking earned value occurs during project execution, it cannot be accomplished if appropriate project planning and budget allocation has not occurred up front. This concept is discussed in detail in a separate Gold Practice titled “Track Earned Value” available on the DACS gold practice website (see www.goldpractices.com).

Additionally, the planner should incorporate a risk management plan, configuration management  plan and quality assurance plan into the project plan and obtain commitment to the plan. For further details regarding these activities refer to the project planning process area of the CMMI guidelines [Chrissis 2003], and to the DACS Gold Practice on Formal Risk Management.

Another recent scheduling methodology, which is applicable for large complex projects, is called ‘Critical Chain Scheduling’. It is innovative and, in many ways, contrary to the techniques described in the CMMI.  It does not specifically address the use of the core metrics, but that does not preclude its usefulness in the execution of the project.  The DACS topic area on Project Management (see www.thedacs.com) has a subtopic on Critical Chain Management that provides some informative articles about what it is, how it differs from more traditional scheduling techniques, and how it should be implemented. 

CONTRIBUTING PRACTICES

As indicated in the definition of the practice, Metrics-based Scheduling is, in reality, a compendium of the body of knowledge relating to the implementation of many other effective practices.  In the context of the preceding pages, these other practices were identified as mechanisms that contributed to effective metrics-based scheduling.  Now, in the remaining pages we will address the salient points of those key practices and provide you with additional references in each case.  They are:

Estimating Techniques

Knowledge of the Software Equation (Core Metrics)

Measurement Program

Requirements Engineering

Risk Management

Estimating Techniques

Note: The explanation of Estimating Techniques provided in the following paragraphs is derived primarily from John McGarry’s book titled “Practical Software Measurement:  Objective Information for Decision Makers” [McGarry 2002].

Estimation produces quantitative projections of key project attributes, which are the basis for initial project plans; it is usually the first type of measurement analysis performed on most projects, often occurring early in the project before the project team is even selected.   It establishes the overall funding and schedule commitments for the project, even though these early estimates are often imprecise.  Poor estimates and misconceptions about the estimating process often contribute to failed projects by leading to planning targets that are impossible to achieve.

Poor estimation can be attributed to the following factors:

• Lack of estimating experience
• Lack of historical data on which to base estimates
• Lack of a systematic estimation process, sound techniques, or models suited to the project’s needs
• Failure to include essential project activities and products within the scope of the estimates
• Unrealistic expectations or assumptions
• Failure to recognize and address the uncertainty inherent in project estimates

Estimation involves using one measure to predict another; product functionality to quantify product size, product size to predict effort; effort to predict schedule; effort to predict cost; product size to predict the number of defects (quality).

There are four basic types of estimation approaches: parametric models, activity-based models, analogy, simple estimating relationships (see Table 5).

Table 5:  Types of Estimation Approaches

Parametric models	Parametric models assume that a specific mathematical relationship exists between size, effort, schedule, and quality and that the relationships are affected by measurable performance factors (parameters) and are based on theoretical reasoning or analysis of historical data. They take the form: Effort =A x (Size) B  x C  Where Effort is expressed in hours or person months, Size represents the amount of software functionality or product, A is a constant, B is an aggregation of performance factors that have a non-linear effect on the output, and C is an aggregation of performance factors that affect Effort in a linear manner.  The non-linear nature of B causes the factors to have more influence on larger software products than those with smaller products.  Performance factors generally fall into two categories, process and product factors.  Estimation models provide as many as 100 factors, but organizations find that only a few factors substantially affect their performance. Parametric models must be calibrated to data from the local development environment for best results. Calibrating all of the parameters takes a great deal of data, however, only a few data points may be required to calibrate a model’s constants to local development conditions, with a resulting improvement in model accuracy. Most parametric models assume that schedule is a function of effort.  Few models support both effort/schedule estimation and quality estimation.
Activity-Based Models	These models are sometimes referred to as activity-based costing or bottom-up estimating because they are derived from engineering estimates of size, effort, and/or schedule for all products and activities in a project.  The low-level estimates for each individual activity or work product are derived from (1) engineering judgment, (2) historical data, or (3) a combination of the two.  Estimates are then aggregated to produce a project-level estimate. This approach requires detailed knowledge of the product and process. It should include effort for all products whether internal or external.
Analogy Approach	This is based on a comparison of the characteristics of the proposed project with other previously completed projects.  Data from similar, or analogous, projects is the basis for the proposed project’s estimates.  Differences are identified and appropriate changes are made to adjust the size, effort, schedule, and quality to fit the new situation. An estimate can be generated from just one similar project; however, it requires a detailed understanding of project contexts and characteristics for both projects.
Simple Estimating Relationships	(Simplified parametric approach) are based on use of local historical data instead of a comprehensive mathematical model. An example would be calculating Effort as the product of size and productivity, where the productivity rate is determined from past projects.  These simple relationships assume that all of the factors that affect effort are similar among past and future projects; therefore, they are seldom appropriate outside of the organization and domain that provided the data.

 

Considerations for selecting an estimation approach are presented in Table 6.

Table 6:  Considerations for Selecting an Estimation Approach

Estimation Approach	Assumptions and Understandings	Data Required	Mathematical Complexity	Applicability
Parametric Models	General understanding of performance factors Might not include all project activities (e.g. requirements analysis) May assume minimum level of documentation Model is based on historical data from many organizations	Historical data for calibration Estimate of size	Complex statistical techniques	Early in life cycle New Development Large projects
Activity-Based Models	Detailed process and product information is available (work breakdown structure	Very detailed historical data for a few projects	Arithmetic	After requirements phase Small and medium  projects
Analogy	Detailed project knowledge of both projects	Historical data for at least one similar project	Arithmetic	Small and medium projects Stable staffing environments
Simple Estimating Relationships	General descriptive information	Historical data from multiple projects within the organization	Simple statistical techniques	Maintenance projects Similar projects

 

Estimating Size, Effort and Schedule

Estimates are typically sequenced as follows: size, effort, schedule, and then quality.

The size metric relates to functional or physical size and sometimes both.  The most prominent unit of measure for functional size is function points. Function points consider the software from an external perspective while source lines of code (typically used to address physical size) consider software from an internal structural perspective.  Function points can be counted early; source lines of code (SLOC) require knowledge of the potential software design.  SLOC works best with the analogy and activity-based estimation methods. SLOC should be applied to the lowest degree of resolution that is economically possible.

Size estimates should be updated throughout the project because they, in turn, impact effort and schedule estimates.

When an organization has a significant project repository of valid data from past projects, it is possible to calculate an effort estimate easily and quickly using the size estimate and a productivity rate from past projects with similar characteristics. Effort is a function of size; schedule estimation is a function of effort combined with project constraints.

A primary mistake in estimating schedule is to ignore critical path dependencies. Parallel development and schedule compression have their limits.  Although sound from a mathematical perspective, they may be infeasible in reality.  Scheduling involves estimating optimum effort distribution or staffing levels but this is relevant only for large projects; many short-duration projects and maintenance organizations operate with a fixed staff.

Table 7 describes some key resources for use in learning about estimating techniques.

Table 7: Some Resources for Learning about Estimating Techniques

Resource	Description and Location
DACS Topic Area on Software Cost Estimation	This web page, which is continuously updated by the DACS, identifies a multitude of resources relating to cost estimating and estimating size, effort and schedule. http://www.thedacs.com/databases/url/key.php?keycode=4
DACS Topic Area on Measurement	This web page addresses measurement in general but has subtopics focused on specific items such as functional sizing and various measurement methodologies that may be utilized to collect the right data for estimating. http://www.thedacs.com/databases/url/key.php?keycode=3
The Common Software Measurement International Consortium (COSMIC) Web Site	This site provides the body of knowledge relating to the COSMIC-FFP method, which was designed for deriving the functional size of non-business software such as real-time and embedded systems. http://www.cosmicon.com/
International Function Point Users Group (IFPUG) Web Site	This site provides the authoritative body of knowledge relating to function point counting and analysis http://www.ifpug.org/
International Society of Parametric Analysts (ISPA) Web Site	ISPA is dedicated to the improvement and promotion of: parametric cost modeling techniques and methodologies risk analysis econometrics design-to-cost technology forecasting cost management http://www.ispa-cost.org/
International Software Benchmarking Standards Group (ISBSG)	This group provides education and training, tools and other resources relating to software estimation, benchmarking, productivity, risk analysis and cost and maintains a database of project data for more than 3000 projects. Developers can use this database to jumpstart their estimation activity when a local project repository is not available. http://www.isbsg.org/
SEI Sage Web Site	This web site provides specific definitions relating to size metrics involving source lines of code (SLOC)

 

Estimating is a broad discipline that goes far beyond the scope of this practice, and subsequently there are many good resources.  Refer to the resources section of this document for additional material.

Knowledge of the Software Equation (core metrics)

 

The software equation, as defined by Putnam and Meyers [Putnam 2003], identifies the generic relationships among some key metrics coined by the Software Engineering Institute (SEI) and others as the “core” metrics.  Specifically, people, working at some level of productivity, produce a quantity of function (size) or a work product at a level of quality (reliability) by the expenditure of effort over a time interval.   The interconnectedness means that when one is adjusted, the others are affected.   To understand the value of the software equation we must attempt to understand the core metrics and their interconnectedness.

Figure 5 identifies and illustrates the relationships among these important metrics.  Some sources assert that there are only four core metrics, size, quality, effort, and schedule, but Putnam [Putnam, 2002] claims that productivity is also one of the core metrics.

Putnam defines these five core metrics as follows:

• Size - Quantity of work or functionality (software product), usually measured in terms of size such as source lines of code or functional size measures, that ultimately executes on the computers; a way of measuring the amount of functionality a project/product represents --- before we embark on it.
• Quality – Level of quality or reliability required for the product, quantified by measures such as the defect rate (or its reciprocal, mean time to defect)
• Time (Schedule)  - the duration of the project, typically measured in calendar months or weeks for shorter projects (often referenced as the schedule)
• Effort – the amount of work expended, typically measured in person-months or person-hours
• Productivity – the rate of progress, expressed in terms of the functionality produced for the time and effort expended.  The conventional measure of productivity has been source lines of code per person-month (SLOC/PM), calculated from past projects. In recent years with increased reuse, and code generation, this measure has become less accurate and is being set aside in favor of Functional Size Units per person-month.

 

Figure 3: Core Metrics and Their Relationships

 

As illustrated in Figure 3 the basic tenets of the relationships among these core metrics are:

• Size and quality estimates represent what must be done (the work) and therefore drive the estimation of effort
• Productivity (derived from performance on past projects) together with size, can facilitate estimating effort and  can be used to identify the minimum possible schedule in scenarios where the number of staff is fixed 
• Effort estimates together with known project constraints drive the development of schedule
• Knowledge of organizational productivity enhances realistic scheduling
• Cost is derived from both effort and schedule

Knowing something about the size (functionality) of the work product, together with the required level of quality supports the developer in estimating the effort that will be required.  Quantifying that functionality in terms of a common metric such as source lines of code or functional size measures facilitates estimating the effort, because the developer can compare it to past projects, or can use parametric modeling to get an estimate of the effort.  In the absence of such historical data from past projects, the developer must rely on functional decomposition that is sufficiently detailed to enable the technical staff to provide realistic estimates of the effort needed to complete each task.  However, such estimates typically fall below what is actually needed for the project.

The software equation establishes a formal mathematical relationship among the core metrics.  Table 8 presents the evolution of the equation from its generic concept to its formal representation as defined by Putnam [Putnam 2003].

 

Table 8:  Evolution of the Software Equation

Work Product (at a Quality level) = Effort over a Time interval at a Productivity level

Size(at implied Quality) = Effort x Time x Productivity Or Productivity = Size/(Effort x Time)

Process Productivity = Size/[(Effort/β)(1/3) x Time(4/3)]

 

Note that this definition of productivity is different from the conventional definition, P= size/effort. Therefore, we need to distinguish it in name as well.  Putnam calls it Process Productivity, the productivity of a project full of all kinds of software people operating over the period of the project.

Putnam [Putnam 2003] has refined the equation over time to reflect what is evidenced by data spanning more than 25 years for thousands of projects from hundreds of companies, many of whom were not aware that such a formula existed when their data was provided.

However, it is not a law of nature; the exponents are approximations, not physical constants.  Beta, β, is a size-dependent parameter that has the effect of giving greater weight to the effort factor in very small systems.  The exponent for the effort term, (1/3), is necessarily less than one; the exponent of the time term (4/3) is greater than one, reflecting the relationship between time and size.   The major difference between conventional productivity and Putnam’s process productivity is that he asserts (through mathematical derivation) that schedule is a factor in productivity; this is not the case for conventional productivity.

Variations of this equation form the basis for most of the parametric models used for cost estimating.

According to Putnam [Putnam, 2003], managers can use the software equation to carry out six basic functions relating to project and process control.  They are:

1.       Estimating time and effort

2.       Setting dates

3.       Supplying resources

4.       Monitoring work progress

5.       Assuring product quality

6.       Measuring process improvement

 

Some key concepts revealed by the software equation are as follows [Putnam, 2003]:

• To get more work product (more functionality, measured by some metric of size) we have to (1) increase the amount of effort, or (2) lengthen the development time, or (3) improve productivity, or (4) augment some combination of effort, time and productivity
• To get a product of higher quality (reliability) we must (1) increase effort, or (2) lengthen development time, or (3) improve process productivity, or, (4) implement some combination of  these increases
• To reduce effort and/or time, we must limit the functionality of the product (reduce its size), accept less quality, or increase productivity
• Improving productivity enables us to achieve more functionality or quality at the same level of effort and time
• With improved productivity we can reduce the effort and time and still achieve the original functionality or quality
• With significant increases in productivity, we can achieve higher levels of the other four metrics ( reduced time and effort, greater functionality and quality)

Here are some of the key facts about the software equation, the core metrics and their relationships:

• Acquire core measures from completed projects where the values of time, effort and size are known (or can be collected).  Productivity is thus obtained through simple calculation. Because it is derived from measured attributes, it is as precise as those measurements.  Productivity should be calculated at the end of every project, but may also be calculated as soon as enough work has been completed to make the metrics valid.
• Productivity (derived from the software equation) stands for the productivity of the entire project organization (not just the productivity of the code writers) during the time period the measurements cover
• There is also a relationship between quality and the other core metrics but it is not necessarily simple math.  Although defects tend to occur at a rate derived from historic data, the core metrics tend to influence the defect rate (typical measure of quality).  Larger projects generate proportionally more defects; high productivity results in fewer defects; allowing adequate time for development reduces defects.
• Planning: Given size and process productivity, we can estimate project time and effort from the relationship between size and process productivity
• Project Control: given the planning projection of time and effort, management can compare actual achievement against what was projected, and act to harness an out-of-control project
• Defect Tracking:  Given a projection of the defect rate, derived from the other core metrics, management can track defects found against those expected
• Process Improvement: Given the productivity value on a series of completed projects, management can assess its progress toward increasing capability and determine whether its improvement efforts are succeeding.
• Merely having core metrics does not guarantee success.  You must use them for guidance and decision-making.
• The core metrics are interdependent, and should always be addressed together.
• Every software project has a minimum development time. There is always a limit to how much the schedule can be compressed.  It varies from project to project depending on size, domain, developer skill, and developer environment. Determining that time, for a new project, involves keeping the core metrics on past projects and having at least a rough size estimate. Extending development time beyond that elusive minimum allows one to reduce the effort and realize improved quality, but the amount of reduction or improvement is based on the knowledge provided by the basic metrics. A reasonable amount is somewhere in the middle.
• There is no maximum development time, but Putnam’s data shows that beyond 130% of minimum development time the trade-off between time and effort lessens sharply, therefore, there is little economic point in extending development beyond 130% range.
• Productivity seldom improves in the short term; data shows that it typically stays the same within the time range of a project.  Productivity gains come slowly; for business projects, the best has been 8% per year, while for engineering and real-time software it has seldom exceeded 5% per year, with many organizations reporting no gains at all.
• Productivity relates to the project/process as a whole, never to the efforts of individuals within the project.  Individual productivity is not one of the core metrics.
• As new development practices emerge, new measures of functionality may be needed.
• We should choose a size (functionality) metric for which we have adequate data from past projects.
• We calibrate the estimate of a product by using the size metrics from past projects to estimate the size of the new project.
• Some (but not all) software activity is of a research nature (requires creativity); in those cases the project should not be constrained by a firm schedule (and fixed price bid).
• Standardize the five core metrics within the scope of a single project and preferably, within the organization. Solid definitions make cross comparisons possible and facilitate education, training and the mobility of people.
• Integrate the metrics within the software development process

 

There are a multitude of articles and resources that relate to the core metrics and their relationships. Table 9 identifies a few resources to get started with.  Others are listed in the resources section of this document.

Table 9:  Some Resources for Learning about the Software Equation and core Metrics

Resource	Description and Location
DACS Topic Area on Software Cost Estimation	This webpage, continuously updated by the DACS, identifies a multitude of resources relating to cost estimating. http://www.thedacs.com/databases/url/key.php?keycode=4
DACS Topic Area on Measurement	This webpage addresses measurement in general but has subtopics focused on specific items such as functional sizing and various measurement methodologies that may be utilized to collect the right data for estimating. http://www.thedacs.com/databases/url/key.php?keycode=3
Crosstalk Article, Feb. 2006	“Software Estimating Models: Three Viewpoints” This article compares the approaches taken by three widely used models for software cost and schedule estimation. The comparisons illustrate significant differences between the models, and show significant differences in the approaches used by each of the model classes. http://www.stsc.hill.af.mil/crosstalk/2006/02/0602JensenPutnamRoetzheim.html
Crosstalk Article , Aug. 2002	“Control the Software Beast With Metrics-Based Management” The five core metrics enable managers to estimate and bid software projects, and control progress during the project. Higher executives and clients need to understand the relationships among them because, otherwise, they are the prime source of unrealistic expectations and the resulting failures.   http://www.stsc.hill.af.mil/Crosstalk/2002/08/putnam.pdf

 

Measurement Program

 

Throughout this document, we have focused on the need to collect and use the core metrics.  In order to do this it is necessary to establish a measurement program.  The program is necessary to ensure consistency and accuracy of the data.  Gopal [Gopal, 2002] notes that many companies find software measurement to be a complex and difficult undertaking, citing studies in the 1990s showing that fewer than 10 percent of the industry classified metrics programs as positive and that two out of three metrics initiatives did not last beyond the second year.  Metrics program implementation is a long and complex journey confronted with dilemmas stemming from contradictory demands.

Metrics can help guide the organization toward informed decisions by providing indicators regarding the quality, adequacy and progress of projects, processes and products and by enabling the organization to recognize the sum of its collective capabilities, which can lead to consistently realistic and achievable plans for providing and delivering products and services.

Metrics also promote teamwork and improve team morale by linking efforts of individual team members to the overall success of the project and ultimately, the success of the organization.

They can also serve as a clear and objective basis for communication with stakeholders.

Metrics, in and of themselves do not impart any value.  The ROI from metrics is the value of the action a project professional takes, with the assistance of the metrics, to manage the issue at hand.  People make the decisions; metrics simply provide the foundation and rationale for such decisions.

McGarry [McGarry, 2002] asserts that fundamental software measurement analysis concepts can be categorized as follows:

• Estimation:  Producing estimates of size, effort, schedule, and quality
• Feasibility Analysis:  Assessing the feasibility of project plans during initial planning and re-planning activities
• Performance analysis:  Assessing the project’s actual performance against plans throughout the project

Levin [Levin 2004] describes a measurement framework comprised of three sets of project related issues: things, people, and the organization.

• Things metrics describe the deliverable of the project and the efficiency with which it is being produced.  This involves the WBS and metrics dealing with element cost, project cost and project schedule.  Delivery issues include risk management, product quality, and contracts.  It includes metrics to assess and track scope and quality and contractor performance with respect to cost, schedule and quality.
• People metrics address the team members’ relationships with one another and attempt to quantify or characterize the behavioral attributes of people. They also measure the friendliness of the organization toward the project team and the team toward itself.
• Organization-Oriented Metrics address the environment in which the project team must operate.  Metrics are tailored to the organization’s important issues and strategic objectives.  The organization provides the environment for the project and, at the same time, is the direct beneficiary of the success of the project.

It is important to strike a balance between the organizational culture and the best practices of project management and establish metrics that cover all three areas.

Gopal [Gopal 2002] conducted some research into factors that affect metrics program success. Success is measured with two variables ---- the use of metrics information in decision-making and improved organizational performance. 

Gopal identifies technical factors influencing use of metrics in decision-making:

• The frequency with which metrics are collected has a positive and significant influence on their use in decision-making.
• The presence of a systematic process for identifying and collecting metrics across the organization increases the use of the metrics in decision-making.
• The benefits from collecting and using the more advanced metrics are greater than the benefits from using the basic metrics; however, the basic metrics must be captured and used first.
• The rigor and usefulness of the analyses done on the raw metrics data increases the use of metrics in decision-making.
• Better communication processes for dissemination of metrics analysis showed a positive effect on use of metrics in decision-making but was not statistically significant.
• The other variables (technical factors) were not statistically significant, preventing statements asserting that they, in fact, positively influenced use of metrics in decision-making.

Gopal also identifies organizational factors influencing organizational performance:

• Involvement of upper management and project managers as well as metrics personnel in setting goals and identifying metrics increases organizational performance.  The institutional belief factor is positively associated with organizational performance and is statistically significant.  The organization that believes that a metrics program is beneficial has higher organizational performance than an organization without such a set of beliefs.
• Companies with software development as their primary activity are more likely to report improved performance resulting from their metrics program than companies where software development is only a small percentage of their business.
• Resource sufficiency and maturity level were not statistically significant.

In general, Gopal’s research results highlight the need to cultivate a culture within the organization to promote metrics usage to ensure eventual success of the metrics program.  Software managers first need to focus on the technical factors and provide incentives for developers to use the metric information in daily operation.  Once metrics data are used at the project level, managers can then focus on the organizational performance improvement in the second stage.

There are several measurement methodologies (developed in the 1980s and1990s but still valid) that one can follow when implementing a measurement program.  There is no need to start from scratch.  The two predominant methodologies are:

 

1.       Goal Question Metric (GQM)

2.       Practical Software Measurement (PSM) – recently renamed as Practical Software and Systems Measurement

 

GQM is a methodology for identifying the right metrics by a process of specifying goals that are aligned with organizational goals, asking questions to refine and clarify the goals, and identifying the metrics that result from responding to the questions posed.  The generic methodology can be applied at various levels, but it is intended to help the organization establish a metrics program that is closely aligned with the goals of the organization.  In most cases, the core metrics would be identified as a result of applying this process, but it might reveal other metrics as well that are unique to your project and/or organizational culture.  GQM relies on participation of all stakeholders for its success.  This is consistent with Gopal’s findings.  For further details, refer to the DACS Gold practice on GQM.  See http://www.goldpractices.com/practices/gqm/index.php

PSM is an information-driven, issue-based approach to software measurement for project management [McGarry 2002].  This measurement process initiative, established in 1994, is funded by the DOD and is freely available at http://www.psmsc.com.  It has been endorsed by government Acquisition Executives, and is a key element of the OUSD (A&T) software initiative.  The methodology is also being adopted by government and industry organizations.

Highlights of the PSM process are:

• Its focus is on cost, schedule, and technical objectives
• It identifies project-specific issues
• Issues are grouped into common software issue areas
• Measurement categories correspond to issue areas
• Each measurement category has a candidate set of proven measures
• Measures are selected based on availability, environment, and other factors.

 

The PSM process defines and uses seven common information categories to facilitate the identification and prioritization of a project’s specific information needs [McGarry 2002].  They are:

• Schedule and Progress
• Product Size and Stability
• Product Quality
• Process Performance
• Resources and Cost
• Technical Effectiveness
• Customer Satisfaction

 

PSM differs from GQM in the following ways:

• PSM is information-driven while GQM is goal-driven.  PSM does not ignore goals; rather it focuses on issues that result when striving to achieve a goal.  Many of the broad goals relating to cost, schedule and quality are assumed.
• PSM focuses on using existing measures or metrics rather than creating new ones.  It does this by providing a pick list of measures for each of the various information categories, which is based on best measurement practices of DoD, government and industry programs.  GQM perceives each project as unique and therefore asserts that one should not arbitrarily adopt another project’s plan or measures.   PSM, however, advocates tailoring the templates and plan to the specific environment.
• PSM is primarily a tool for the project managers who work with a measurement specialist to establish the measurement program that meets their needs.  Other stakeholders, such as the development team are not directly involved in the planning although they are called upon to provide data and implement data collection procedures.  GQM seeks to involve the stakeholders in the entire process and focuses on the human and cultural issues involved in implementing a metrics program, operating on the principle that involvement at all levels is critical to a successful measurement program.
• PSM is designed to help put measurement into practice at the project level, thereby providing the data for addressing enterprise level performance, process improvement, and business-related questions.  GQM has a more generic application in that it can be used wherever there is a need to assess satisfaction of goals, irrespective of the subject domain or the level.

PSM was the basis for the development of ISO/IEC 15939 and has influenced several other standards as noted in Figure 4 (extracted from the PSM website in June 2006): 

• The purpose and outcomes of the measurement process from ISO/IEC 15939 have been added to the revision to ISO/IEC 12207, Software Life Cycle Processes, within a new supporting process, entitled Measurement. 
• Measurement concepts have also been added to ISO/IEC 15288, System Life Cycle Processes. 
• The new measurement terminology has also been coordinated with the revisions to ISO/IEC 9126, Software Product Quality, and IOS/IEC 14598, Evaluation of Software Products, so that all these standards will use a common set of measurement terminology, once the revisions are complete. 
• In addition, the purpose and outcomes of the measurement process have been added to ISO 9000-3: Application of ISO 9001:2000 to Software.

Figure 4:  The Influence of PSM on Various Measurement Related Standards

The draft international standard ISO/IEC 15393, in turn, was used as an input to the Measurement and Analysis (MA) process area of the CMMIsm.  The MA process area provides a methodology for assessing whether a project's measurement program is compliant with the international standard, in addition to providing relevant information on CMMI-based process improvement activities.  Overall, the CMMI helps organizations to institutionalize their measurement and analysis activities, rather than addressing measurement as a secondary function.  In the MA process area, the activities of "Plan Measurement" and "Perform Measurement" are detailed in two specific goals that must be implemented and eight specific practices that are considered important in achieving the associated specific goals.  The activities of "Evaluate Measurement" and "Establish and Sustain Commitment" are considered through the generic goals, with elaborations specific to the MA process area.

The coordination of these documents means that the software and systems engineering communities have a consistent set of information-driven standards and guidance for implementing project and process measurement.

Some common themes or lessons learned that have emerged from the literature about measurement programs include:

• Establish a metrics mindset
• Create and foster an atmosphere conducive to the acceptance of metrics technology by providing ways to monitor cost effectively and efficiently
• Find meaningful incentives (with short-term payoffs) to motivate managers to establish a metrics program
• Create awareness: build confidence through success stories
• Emphasize continuous data collection and awareness
• Address the question of organizational culture.  Can a metrics program be effective, or will change management be needed?
• In initial planning for the metrics program, assess whether the skills exist internally and whether there is time to initiate the program. Alternatively, are outside consultants needed?
• Involve practitioners through feedback
• Key measures need to be clearly defined
• Determine up front how to evaluate the success of the metrics program. According to Gopal [Gopal, 2002], past studies have quantified the success of metrics programs by using either the longevity or the budget allocated to it.  He asserts that these measures capture a limited dimension of success; longevity and budgeted allocations are meaningless if the metrics information is not, in reality, used.
• Recognize that a metrics program is, in fact, extra work.  The extra work involves:
• Determine what is important /span>
• Assess what data is needed /span>
• Determine how to collect the data /span>
• Define the reporting requirements /span>
• Design the processes /span>
• Collect the data /span>
• Analyze the data /span>
• Implementing the metrics program is itself a project, but the process or reporting metrics over time is operations.  Project managers must sell the program to upper management so it can sponsor it over time; must also sell to the rank-and-file who will be responsible for implementing it.
• Recognize that it is possible to select the wrong metrics.  Choosing the wrong metrics causes extra effort in measuring, collecting and reporting activities and often results in having to collect more relevant metrics after-the-fact.
• Metrics chosen should be specific to what you are trying to measure, not generic metrics that you collect because they can be collected.
• Recognize that there can be bias in data collection and take precautions to minimize it.
• Collect Metrics in a standardized fashion
• Report on metrics consistently
• Explain to those responsible for collecting the metrics how the metrics will be used
• Set standards to ensure that metrics are comparable across projects.

Some general principles for determining how much measurement should be standardized within a project or organization are [McGarry, 2002]:

• Standardize on base measures rather than indicators. Base measures can be combined in many ways to produce different indicators that address different information needs.  Base measures are tied to specific entities – which tend to be relatively persistent.
• Promote flexibility in the indicators.  No single view of the data works for all decision makers throughout the project.  Indicators are tied to information needs, which tend to change frequently.

 

Table 10 provides some key resources for enriching your understanding of establishing a measurement program.

Table 10:  Some Resources for Learning about Establishing a Measurement Program

Resource	Description and Location
DACS Topic Area on Measurement	This webpage addresses measurement in general but has subtopics focused on specific items such as functional sizing and various measurement methodologies. http://www.thedacs.com/databases/url/key.php?keycode=3
DACS Gold Practice Goal Question Metric Approach	This document explains in detail the Goal Question Metric Approach and provides a detailed list of resources for additional reading. http://www.goldpractices.com/practices/gqm/index.php
IEEE Transactions on Software Engineering, Vol. 28, No. 9, September 2002	Gopal, Anandasivam, Krishnan, M. S., Mukhopadhyay, Tridas, and Goldenson, Dennis R., “Measurement Programs in Software Development: Determinants of Success” Gopal’s research seeks to identify success factors for metrics programs and study their direct and indirect effects on metrics program success, which is characterized with two variables – use of metric information in decision-making and improved organizational performance.  He identifies many case studies as well.  The paper can be purchased from IEEE.  http://ieeexplore.ieee.org/xpl/abs_free.jsp?arNumber=1033226
White Paper by Frank Parth, 2003	Parth, Frank R. and Gumz, Joy, “How Project Metrics Can Keep You From Flying Blind”, Project Auditors, LLC, December 2003 Click here
McGarry Book, 2002	McGarry, John, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark, Joseph Dean and Fred Hall, Practical Software Measurement: Objective Information for Decision Makers, Addison-Wesley, 2002 This book is a comprehensive guide for getting started with a measurement program using the Practical Software Measurement methodology.
Practical Software and Systems Measurement Web Site	This website provides a wealth of information about the Practical Software Measurement (PSM) methodology, now expanded to include both Software and Systems measurement. http://www.psmsc.com/

Requirements Engineering

 

Knowledge (awareness) of requirements is necessary in order to do the estimating and planning that result in realistic estimates of size, effort and project schedule.  The term “requirements” has generic meaning in this context.  It may encompass such things as a ‘scope of work’ description, functional requirements, quality requirements, performance requirements, operational requirements, and constraints relating to architecture, completion time, cost and life cycle.  The requirements could be something stated as simply as:

 

“Build an open source application that has all the key features of our current custom-built in-house content management application.  Business needs demand that Functions A, B and C have to be operational in the new system in four months; the other features can be implemented on an incremental roll out after we cut over to the new system.”

 

This very brief paragraph implies that the acquirer has the goal of migrating to an open source architecture.  The statement contains a multitude of implied and referenced requirements and constraints, all needing to be analyzed in order to gain sufficient knowledge of the real requirements to enable estimating the size and complexity of the product and to understand the risks associated with the task.

Requirements may appear in a formal specification document; they might take the form of use cases, stories, a collection of emails, meeting notes, or notes from a phone conversation.  They may exist as customer expectations or as domain assumptions.

 

Regardless of the source, or state of representation, the developer must make the requirements explicit, through communication with key project stakeholders.  Requirements drive the decision-making about the development (maintenance) approach and ultimately affect the composition of any level of the WBS.  In the absence of documented requirements, both developers and estimators make implicit assumptions about the requirements and constraints. Those assumptions are rarely communicated and typically surface late in the project when it is more difficult to correct them.  Such assumptions also affect the accuracy of the size estimates.

 

Davies [Davies 2005] talks about requirements engineering within an agile development environment.  Here are some of the highlights.

• Agile methods evolve both requirements and the system under development by planning the development in short cycles (weeks rather than months).  Agile teams use conversation as the primary means of communicating requirements.  The medium of conversation creates the possibility for information flow between all parties. The problems imposed by communicating via documents drop away.  All team members are partners in this conversation. They discuss scenarios and development options to explore the requirements before development is planned. The beauty of exploring requirements through team conversation is that we develop a shared understanding and have the opportunity to offer our own ideas. Having a two-way conversation between business and IT and flushing out misinterpretation early may prevent developing code that has to be thrown away.
• The use of Stories is a primary practice of Extreme Programming (XP). Stories must be small enough to implement in a single development cycle (XP teams use a one-week cycle). Each story card represents a conversation that has taken place.
• Other agile methods also emphasize that requirements descriptions need to be short and loose, the bare-minimum needed to plan development. Typically, requirements are listed as one-line descriptions maintained in a prioritized list. Features may be swapped in and out of this backlog and it is assumed details about each requirement will be determined as part of each development cycle. Stories are developed with the goal of achieving a shared understanding. Understanding the big picture can help programmers to develop software solutions that are a better fit.

 

Leaders responsible for estimating need to ensure that the real requirements are sufficiently documented to provide common guidance for all those involved in estimating at any level.

 

It is also a reality in many development situations that some requirements are unknown at the start of the project and that they evolve as more knowledge is gained.  The planner needs to recognize this and address it as a risk.  Then such risks are taken into account when doing the estimates.

 

In addition to ensuring a common understanding of the real requirements up front, requirements engineering is about managing requirements as the project progresses because requirements usually change.  The project needs to be able to recognize when new requirements emerge or requirements change, analyze the impact of the change to the project, communicate with the acquirer/customer regarding that impact and make decisions regarding the prioritization of those requirements.  Analysis of impact and communication are the critical components of requirements engineering.  Those activities may have varying formats depending upon the development paradigm followed, but, regardless of format, they are critical to the development process.  From a metrics perspective, one would want to monitor requirements churn or requirements volatility as well as the increase in requirements.  There are a number of resources that can guide you in implementing a requirements engineering program and in identifying and implementing key metrics for tracking requirements changes.  Table 11 provides some initial resources.  Refer to the resources section of this document for additional material.

 

Table 11:  Some Resources for Learning about Requirements Engineering

Resource	Description and Location
DACS Gold Practice on Requirements Management	This document describes effective practices for requirements management and identifies a wealth of resources for additional reading and detailed investigation. http://www.goldpractices.com/practices/mr/index.php
DACS Topic Area on Measurement	This web page addresses measurement in general but has subtopics focused on specific items such as functional sizing and various measurement methodologies that may be utilized to collect the right data about requirements. The site is updated continually with new material. http://www.thedacs.com/databases/url/key.php?keycode=3
DACS Gold Practice on Requirements Trade-offs/Negotiations	This document describes the practice of making tradeoffs in requirements to ensure that a balance exists between what is desired or expected and what is realistically possible. http://www.goldpractices.com/practices/rto/index.php
Ralph Young’s Web Site	Ralph Young, renowned author of several books, including the Requirements Engineering Handbook, 2004, Artech House, Inc., maintains this web site, which is dedicated to improving requirements practices by providing ideas, suggestions, recommendations, and resources for those who must understand and manage customer requirements in the systems and software engineering world. http://www.ralphyoung.net/
Software Program Managers Network (SPMN)  Web Site for 16 Critical  Software Practices	One of the 16 critical software practices identified by the SPMN is Manage and Trace Requirements.  This web page discusses effective practices and provides some insight into requirements related metrics that are useful in tracking a project. http://www.iceincusa.com/16CSP/content/8_requir/anamtrgt.htm

 

Risk Management

Risk management and metrics-based management are integrated processes.  Software Risk Management is a proactive approach for assessing and controlling the uncertainty and potential loss associated with a project. 

Some symptoms that organizations are not effectively practicing risk management include:

• A continual state of project instability
• Constant fire-fighting
• Multiple schedule slippages because of recurring surprise factors
• Constantly operating in a high-stress, crisis management environment

Ironically, these same symptoms would be observed in a project considered “out-of-control”.

Capers Jones assessed several hundred organizations and observed over 100 software-related risk factors (of which 60 he discusses in detail in his book [Jones, 1994]).  He observed, however, that few projects have more than 15 active risk factors at any one time, but many projects have approximately six simultaneous risk factors.  According to Jones, some of the most serious risks factors are:

• Creeping user requirements
• Excessive schedule pressure
• Inaccurate cost estimating
• Inaccurate metrics
• Inadequate measurement
• Poor project management (malpractice)

Noteworthy, is the fact that metrics-based scheduling, as described herein, is, in fact, a risk mitigation strategy for these risk factors.

While further investigation of risk management is beyond the scope of this document, the list of resources provided in table 12 should be very helpful in learning how to implement risk management for your next project.

Table 12:  Resources for Learning about Risk Management

Resource	Description and Location
DACS Gold Practice on Formal Risk Management	This document describes effective practices for risk management and provides a wealth of resources for more detailed investigation. http://www.goldpractices.com/practices/frm/index.php
DACS Topic Area on Risk Management	This web page, continually updated by the DACS, contains a wealth of literature, and other resources for risk management, including training, tools, experts and service providers and consultants. http://www.thedacs.com/databases/url/key.php?keycode=270
Capers Jones book on the Assessment and Control of Software Risks	Jones, C., Assessment and Control of Software Risks, Prentice Hall, ISBN 0137414064, 1994 This book is highly recommended for anyone wanting to understand the relationship between managing risks and developing software.
Software Program Managers Network (SPMN) website for 16 Critical Software Practices	One of the 16 critical software practices identified by the SPMN is “Adopt Continuous Program Risk Management”.  This webpage discusses effective practices and provides some insight into why and how to implement a formal risk management program. http://www.iceincusa.com/16CSP/content/1_risk/rskfrm.htm

 

Characteristics of implementation

 

Summary characteristics

Enabling Practices:           Link to Metrics-based Scheduling Interrelationships Diagram

Enabled Practices:            Link to Metrics-based Scheduling Interrelationships Diagram

Impact Areas:                      Cost, Schedule, Risk, Technical Performance

Life Cycle Phase:               Concept and technology development (requirements and design) and support/maintenance of existing systems

Scope/Authority:                Project-level and above

Applicability:                       Acquisition; development

Use Indicators:                   Whenever there is motivation to improve estimates and/or establish better control of a  project

Use Inhibitors:                    Lack of management commitment to capturing core metrics

Appropriate Programs:    Software-intensive acquisitions, maintenance or operational projects

Inappropriate Programs: Small, single product, one-of-kind projects

Barriers:                               Mistrust of metrics; Lack of management commitment to collect core metrics; Poor quality (inaccurate) project metrics and metadata from past projects; Lack of project metadata; Rigid contractual agreements that freeze requirements and  time; Inadequate resources allocated to requirements engineering

Facilitators:                          Strong leadership; Skilled estimators; Project metadata repository; Metric Mindset; RM tools; Knowledge of functional sizing methods and benefits; Flexible contracts; Iterative development approach; Resources and budget allocated to initial planning

 

ANTICIPATED BENEFITS OF IMPLEMENTATION:

• Maintain control of software quality and cost
• A more accurate/realistic project baseline
• Better estimates at less cost over time
• Improved organizational performance
• Increased likelihood of project success (quality product, on time, and within budget)
• Satisfied customers
• Improved communication among stakeholders
• The capability to acquire software on the basis of ‘cost per delivered functional unit (in contrast to a fixed price contract)

 

Detailed characteristics

Key Characteristics of the “Metrics-based Scheduling” Gold Practice

Characteristic	Comments
Based on empirical data	Project metadata and product size, along with size, effort and quality data collected from previous projects are used to create the initial schedule An organization must determine what combination of measures are the best predictors of effort within their organizational environment. Product size, project metadata, and process productivity, derived from similar projects, are used to predict effort for a proposed project.
Captures, quantifies and uses process productivity	Productivity rate is derived from accurate effort and size measures from past projects, with similar characteristics. Productivity is expresses as a quantity of work per calendar unit of time (e.g. 8 functional size units per person month, 30 source lines of code per person week) Productivity rate X estimated work product size = effort estimate Use productivity rates to get reasonable effort estimates in a brief amount of time
Takes time to implement	Need to develop skill in estimating size and build up a repository of project metadata Apply estimating  across several project life cycles Monitor estimating accuracy (by comparing to actual measures or estimates at completion) Establish a metrics program to support gathering the right data Expect to take two – three years to establish a sound practice
Relies on software sizing	Software size is the key measure for estimating effort  and determining productivity , which, in turn, are key factors in determining the initial schedule Software size is an estimate. Requirements growth is reflected in size changes as the project progresses; need to estimate size more than once Functional sizing methods provide consistent estimates with minimal resources early in the project life cycle Physical size estimates, such as source lines of code, may still be valuable in a stable environment The benefits of using a skilled size estimator are significant.
Enhanced by requirements engineering practices	Real requirements (vs. formal documented requirements) should drive the size estimating process Requirements volatility may impact project effort more than product size. The Sizing process can flush out areas where requirements are not clear. Managing requirements in light of expected change can provide objective evidence to support schedule adjustment before the project reaches a crisis state
Presumes risk management	The project manager must determine how much confidence they have in the size, effort, and productivity estimates they are expected to use Tracking estimation accuracy over time and across projects should lead to actions that improve accuracy over time and ultimately, the confidence intervals of those estimates. Multiple estimating methods should be used as a reality check Size metrics, collected throughout the project life cycle, can be used as risk mitigators to the risk of requirements creep
Requires management commitment	The metric program needed to support metrics-based scheduling is unlikely to be successful without management commitment and a true focus on gathering the right data (the real effort data) If management ignores or bypasses the metrics data collection and estimation processes and elects to use other means to establish effort estimates, then the metric program has no real purpose and is viewed by most as simply extra extraneous work.  It will not be effective in providing valid data.
Facilitated by contract flexibility	In many cases, the benefits of metrics-based scheduling are nullified by strict inflexible contracting arrangements which fix the time and or cost.  If there is no mechanism for addressing requirements changes and their impact on schedule and cost, then the use of metrics-based scheduling is restricted to developing reasonable proposal bids, not controlling the project. Having accurate size metrics and reliable productivity metrics enables the developer and acquirer to engage in performance-based contracting where the contract specifies a cost per delivered functional unit, with perhaps a ceiling and both parties address the requirements priorities after contract award.

 

Relationships to other practices:

 

The figure below represents a high-level process architecture for the subject practice, depicting the nature of the influences on the practice (describing how other practices might relate to this practice).  These statements are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may “influence” (or are influenced by) the ability to successfully implement other practices.  A brief description of these influences is included in the table below.  The arrangement of the statements in the figure and corresponding table does not imply any particular order or sequence in which they should be addressed.

 

Process Architecture for the "Metrics-Based Scheduling" Gold Practice

 

Note: In the table below, specific Gold Practices with strong relationships to this practice are identified with italics.

Summary of Relationship Factors

INPUTS TO THE PRACTICE
Identify Appropriate Metrics	In order for Metrics-Based Scheduling to work, the organization must be collecting the right metrics.  Developers must have confidence in the metric data collected and perceive that data to be valid and valuable for decision-making.  Therefore, there should be a specific purpose or goal, which justifies or drives the data collection for each specific metric.  Goal-Question-Metric (GQM) focuses on aligning metrics with goals by involving the right people, identifying and prioritizing the goals, asking questions to converge on appropriate metrics for monitoring attainment of those goals.
Address the Uncertainties of Estimates	Formal Risk Management (FRM) is a major contributor to metrics-based scheduling because it allows us to recognize and provide mitigation for the risks associated with our estimates of size, effort, quality, schedule, and costs.  Without FRM, estimates are often perceived as simple mathematics and get used as though they were exact.   Many projects today involve integration of COTS/NDI and reuse of existing components rather than custom development. This may impact the size estimate by introducing inconsistencies in the derivation of size, and affecting the correlation between size and effort, which, ultimately, affects the productivity metric.   Two Gold Practices, Leverage COTS/NDI and Assess Reuse Risks and Costs, address the effects of reuse and COTS on software development. Reuse and COTS integration are important characteristics to be captured and stored with other project metadata because it can affect which projects should be used for making future estimates.
Establish an Ongoing Measurement Program	A measurement program provides data to, or gathers data from every process or practice that contributes to a project.  Its structure should facilitate the collection of core metrics, which are the lifeline of metrics-based scheduling, throughout the project life cycle.  The quality and consistency of the measurement practices and the resulting measures collected are essential to build the repository of project metadata that supports schedule planning and decision-making. The primary methodologies (practices) for establishing a measurement program are the Goal-Question-Metric(GQM) approach and the Practical Software Measurement methodology.  Metrics are an essential element of Develop and Maintain a Life Cycle Business Case , which, in turn, can provide a basis for sound negotiation when a reality check reveals that the initial schedule is not realistic in light of identified constraints.
Analyze requirements	Metrics-based Scheduling depends on having measures of size and effort along with other attributes (metadata) from past projects and estimates of size and effort for the subject project.  Knowledge of requirements (specified, implied and inferred) drives the estimating of product size, not only in the initial phases but also throughout the project as requirements change.  Therefore, a formal process for Managing Requirements (also called Requirements Engineering) is a critical pre-requisite for Metrics-based scheduling.  Often, the basis for identifying good metrics lies within the project and product requirements.  Projects that Manage Requirements throughout the project are more likely to identify and execute meaningful metrics.  Because metrics-based scheduling is premised on consistent and accurate estimating practices it alerts the stakeholders to situations where what is expected (scope of work and other requirements/constraints) is not realistic. Such information should prompt the parties to engage in making trade-offs.  The Gold Practice Requirements Trade-offs/Negotiations addresses that process.  If stakeholders ignore early results from the reality checks that are part of Metrics-based Scheduling, then, they are essentially nullifying the value of the practice.
OUTPUTS FROM THE PRACTICE
Capture Product, Project and Process Data for Future Planning	Data captured in order to estimate the initial schedule helps build the project repository of metadata, which can be used for estimating and planning future projects but also sets the stage for Quantitative Progress Measurement such as tracking growth in project size. The data collected also supports the program-wide communication of progress (see the Gold Practice titled “ Program-wide Visibility of Progress vs. Plan”)
Use Metrics for Decision-making	Metrics-based Scheduling establishes a realistic framework for using metrics for decision-making, namely, Metrics-based Management, which relies on Quantitative Progress Measurement techniques such as Tracking Earned Value.
Establish Project Baseline	Metrics-based scheduling involves not only defining, but also verifying the project baseline.  Thus, it provides a solid foundation for starting the project, monitoring progress over time and making adjustments as necessary with the knowledge of how those adjustments will impact the project. Thus, Metrics-based Management, Quantitative Progress Measurement, Track Earned Value practices all depend on it for their success.
Collaborate/Communicate on Barriers to Success	The core metrics and other estimates, collected and used to establish the initial project schedule, can be used to support  Requirements Tradeoff/Negotiation , to Develop and Maintain a Life-cycle Business Case for the project, and facilitate Program-wide Visibility of Progress vs. Plan .

RESOURCES:

 

• Web Sites
• Software Program Managers Network (SPMN) web site:  This site identifies 16 critical software practices, two of them relating directly to this practice.  Practice no. 2 is titled “Estimate cost and schedule empirically”; practice no. 3 is titled “Use metrics to manage” 

  http://www.spmn.com/16CSP.html
  
  

• Software Program Manager’s Network (SPMN) Lessons Learned (1998):  This web page describes the reality of implementation of best practices in the DOD, identifying cultural, managerial and technical problems.  It can serve as a sanity check for managers of what not to do.

  http://www.spmn.com/lessons.html
  
  

• International Software Benchmarking Standards Group (ISBSG) web site: The mission of the ISBSG, established in 1997, is to help improve the management of IT resources by both business and government through the provision and exploitation of public repositories of software engineering knowledge, which is standardized, verified, recent and representative of current technologies.  The Group currently maintains a software development repository of more than 3000 projects, and has recently established a repository of software maintenance and support projects.  It provides many resources relating to software estimation, benchmarking, productivity, risk analysis, and cost information for software developers and businesses that are based on the data within these repositories.

  http://www.isbsg.org/ISBSG.nsf/weben/Home
  
  

• Practical Software and Systems Measurement (PSM) web site:  PSM was developed to meet today's software and system technical and management challenges. It is an information-driven measurement process that addresses the unique technical and business goals of an organization. The guidance in PSM represents the best practices used by measurement professionals within the software and system acquisition and engineering communities.

  http://www.psmsc.com/
  
  

• Software Cost Estimation: Metrics and Models web site:  this site is actually a web book about cost estimation maintained by Kim Johnson, Department of Computer Science, University of Calgary, last updated in 1998.  It is an excellent introduction to cost estimation for project managers and other professionals who have a need to understand the essential elements.

  http://sern.ucalgary.ca/courses/seng/621/W98/johnsonk/cost.htm
  
  

• DACS web site:  The DACS maintains research areas (topics) related to software.  Each topic is further subdivided into sub topics and information categories such as case studies, literature, service providers, training, tools and more.  New resources are listed on a continual basis.  You may want to visit the following research areas that are particularly relevant to this practice:  Cost Estimation, Measurement, Risk Management

  http://www.thedacs.com/
  
  

• Tools and Methods

[The fact that a tool or tool vendor is listed (or not listed) here does not constitute an evaluation or endorsement of the product or vendor by the DACS.  These are simply links to tools currently available relative to the subject practice. To suggest other tools, please email the DACS: dacs@dtic.mil]

There is no single tool or method that directly addresses all aspects of metrics-based scheduling.  Rather, the tools and methods tend to be associated with the categories of measurement, cost estimating and project management.

Measurement Related Methods

Name	Description
Balanced Scorecard	This method is useful method for translating strategic goals into actions.  [Parth, 2003 ] The balanced approach means that you cannot measure in only one area.  You have to measure in all the areas that are important to the business.  Use a mix of financial and non-financial measurement, linking metrics regarding customers, internal processes, employees, and financial results to long term financial success.  Key to the balanced scorecard is collecting perspective from differing viewpoints: the customer, the employee, the financial manager and the marketing manager.
Goal/Question/Metric Approach (GQM)	GQM is a three-level structured approach for identifying appropriate metrics that are aligned with the goals of the project or organization. It was developed by Vic Basili in the 1980s and has been refined over time. It consists of (1) expressing goals of the organization, (2) generating questions to meet those goals, and (3) analyzing questions to define the metrics and assess the feasibility of collecting the metrics.  Similar approaches include Issue/Category/Measure presented by J. McGarry and others in 1997 and Factor/Criteria/Metric proposed by Walters and McCall in 1977 [McGarry et. al., 2002].
Practical Software Measurement (PSM)	PSM is an approach to fact-based software management based on integrating the concepts of a Measurement Information Model and a Measurement Process Model.  The Information Model defines the relationship between the information needs of the manager and the objective data (measures) to be collected; the Process Model describes a set of related measurement activities that are generally applicable in all circumstances, regardless of the specific information needs of any particular situation.  They combine to form a viable information-driven measurement process.   The basic concepts and terminology of PSM are formalized in the ISO/IEC Standard 15939 – Software Measurement Process (2001) and have been adopted as the basis for the Measurement and Analysis Process Area in the Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI) [McGarry, 2002].
Software Metrics Plan	This plan, for the Project Planning Key Process Area of the CMM  for the SENG 623 Company,  is a sample plan that can be used as a template for developing your own metrics plan http://sern.ucalgary.ca/courses/seng/621/W97/johnf/metplan.htm
Value-based Six Sigma	Six sigma is a business improvement methodology commonly used to obtain detailed information regarding customer demands and then using this information to improve process and product design.  It is a methodology for controlling a process to the point of plus or minus six sigma (standard deviations) from a centerline.  Its basic methodology, DMAIC, consists of the following five steps: Define the process improvement goals that are consistent with customer demands and enterprise strategy. Measure the current process and collect relevant data for future comparison. Analyze to verify relationship and causality of factors. Determine what the relationship is, and attempt to ensure that all factors have been considered. Improve or optimize the process based upon the analysis using techniques like Design of Experiments. Control to ensure that any variances are corrected before they result in defects. Set up pilot runs to establish process capability, transition to production and thereafter continuously measure the process and institute control mechanisms.

 

Measurement Related Standards

Name	Description and URL
ISO/IEC -15939:2002(E)	Software Engineering - Software Measurement Process This standard identifies the activities and tasks that are necessary to successfully identify, define, select, apply, and improve software measurement within an overall project or organizational measurement structure.  It also provides definitions for measurement terms commonly used within the software industry.  The standard is described in terms of the purpose and outcomes of a compliant process, along with associated activities and tasks.  It also defines the measurement information model and associated terminology.  The ISO/IEC 15939 covers measurement activities, required information, the application of measurement analysis results, and determining if analysis results are valid.  It can be used by both systems suppliers and acquirers.  It is available for purchase from ISO or the American National Standards Institute(ANSI) but subject to strict copyright restrictions for distribution and use. Click Here
ISO/IEC 19761:2003	Software Engineering -- COSMIC-FFP -- A functional Size Measurement Method  ISO/IEC 19761:2003 specifies the set of definitions, conventions and activities of the COSMIC-FFP Functional Size Measurement Method. It is applicable to software from the following functional domains: 1.       Application software which is needed to support business administration 2.       Real-time software, the task of which is to keep up with or control events happening in the real world 3.       Hybrids of the above Click here
ISO/IEC 14143: Series 1 - 6	Information technology -- Software measurement -- Functional size measurement: Parts 1 – 6   Part 1 (1998): Definition of Concepts Part 2 (2002): Conformity evaluation of software size measurement methods to ISO/IEC 14143-1:1998 Part 3 (2003): Verification of functional size measurement methods Part 4 (2002): Reference model Part 5 (2004): Determination of functional domains for use with functional size measurement Part 6 (2006): Guide for use of ISO/IEC 14143 series and related International Standards http://www.iso.ch/iso/en/StandardsQueryFormHandler.StandardsQueryFormHandler?langu ageCode=en&keyword=14143&lastSearch=true&title=true&isoNumber=&isoPartNumber=&i soDocType=ALL&isoDocElem=ALL&ICS=&stageCode=&repost=1&stagedatepredefined=&st ageDate=&committee
ISO/IEC 20926: 2003	Software Engineering -- IFPUG 4.1 Unadjusted Functional Size Measurement Method -- Counting Practices Manual   Specifies the International Function Point Users Group (IFPUG) Release 4.1 unadjusted Functional Size Measurement Method.  It provides a clear and detailed description of function point counting; a foundation to ensure that counts are consistent; guidance to allow function point counting of Functional User Requirements from the deliverables of popular software development methodologies and techniques; a framework to enable automated support for function point counting Click here

 

Tools That Support Metrics- based Scheduling and Management

 

 

• Experts and Contact Points

 

The DACS maintains a list of Experts for each of its research areas.  In some cases contact information is provided.

 

DACS List of Experts for Cost Estimation

http://www.thedacs.com/databases/url/key.php?keycode=4:1942

 

DACS List of Experts for Measurement

http://www.thedacs.com/databases/url/key.php?keycode=3:1945

 Dr. Ginger Levin is a project management consultant with more than 25 years of experience. She is the co-author of Achieving Project Management, Success with Virtual Teams, Advanced Project Management Office - A Comprehensive Look at Function and Implementation, People Skills for Project Managers, Assuring Project Success with Metrics-Based Management.

http://www.govg4i.com/content.cfm?pageid=3664

 

• Training Opportunities

  The fact that a course is listed (or not listed) here does not constitute an evaluation or endorsement of the course by the DACS.  These are simply links to courses currently available relative to the subject practice.  In the interest of keeping the document manageable, the following types of training have not been included:

• Pure consulting services
• Training courses primarily based/offered outside of the continental US.
• Courses focused on broader areas with elements of this practice as a component
• Tool –specific training courses

 

Many of the vendors offer a virtual classroom alternative to the traditional classroom training and many offer on-site training opportunities.  Click on the links provided to get this type of information.  The first table below lists training opportunities focused on metrics and measurement.  The second table below lists training opportunities relating to estimating

Courses on Measurement and Measurement Programs

 

Courses relating to software estimation

 

 

• Bibliography

 

 

APPENDICES

 

DEFINITIONS:

Metrics-based Scheduling - establishing realistic software development or maintenance schedules based on accurate estimates of software size and effort and other empirical data.

SOURCES (Origins of the Practice):

.

Although the use of metrics for scheduling and management has been practiced to some degree for a long time, the practice was formally identified in 1995 by the Arlie Council, (a group of software experts and industry leaders, that met under the aegis of the Software Program Managers Network (SPMN)), as one of nine principal best practices that, if implemented, would improve software development and maintenance productivity and quality, reduce cost and improve user satisfaction.

RECOMMENDING SOURCES:

[This section identifies segments from specific DoD, industry, or academic sources that recommend, specify or otherwise support adoption of the subject practice either directly, or as part of a broader best practice initiative, or a practice with a different name.]

• Capability Maturity Model Integration (CMMI), Version 1.1, Software Engineering Institute, CMU/SEI-2002-TR-011, TR-012, March 2002 [CMU/SEI-2002-TR-011], [CMU/SEI-2002-TR-012]

   

  The Project Planning process area of the CMMI includes a specific goal for establishing and maintaining estimates of project planning parameters, which include all information needed by the project to perform the necessary planning, organizing, staffing, directing, coordinating, reporting and budgeting.  It specifically states that estimates should have a sound basis to instill confidence that the plans based on them will support the project objectives.  It identifies as a specific practice establishing estimates of work product size as the primary input to models for estimating effort, cost and schedule.  It further identifies Measurement and Analysis as a support process area where the measurement program provides supports objective planning and estimating by providing objective results that can be used in making informed decisions.
  
  

• Section 804 of the National Defense Authorization Act (Enacted Dec. 2, 2002)

   

  Section 804 mandates improvement of the DOD’s software acquisition processes to address cost overruns, schedule slippages, and performance difficulties. This legislation directly instructs the secretaries of each military department and heads of selected defense agencies to establish software acquisition process improvement programs – an apparent message of frustration with the way software improvement had been handled in the past.  Software acquisition process improvement program requirements include the following:



• A documented process for software acquisition planning, requirements development and management, project management and oversight, and risk management
• Efforts to develop appropriate metrics for performance measurement and continual process improvement
• A process to ensure that key program personnel have an appropriate level of experience or training in software acquisition
• A process to ensure that each military department and select defense agency implement and adhere to established processes and requirements relating to the software acquisition


• GAO Report on Defense Acquisitions, March 2004 [GAO, 2004]  

   

  The consensus of this report was that stronger management practices are needed to improve DOD’s software-intensive weapons programs.  The report goes on to state that the success (quality software product, on time and within budget) of leading software companies is attributable to creating a manageable evolutionary development environment, using disciplined processes, and useful metrics such as earned value.

 

GLOSSARY

10% Value Chart	A chart that lets stakeholders and the project team discuss and capture vital information about relative priorities in the triple constraint - schedule (time), performance (scope), and resources (cost).  See [Barker, 2000].
Airlie Council	Refers to a group of experts convened by the Navy’s Software Program Manager’s Network (SPMN) in 1995 who established/identified nine best practices.  These practices have been augmented with other practices since 1995, and in current literature are referenced as the original Airlie best practices.
Baseline	Defined quantity or quality used as starting point for subsequent efforts and progress measurement that can be a technical, cost, or schedule baseline. See Performance Measurement Baseline (PMB) and Acquisition Program Baseline (APB). [DAU,2003]
Baselining	A process whereby all managers concerned collectively agree on the specific description of the program, requirements, funding, and make a commitment to manage the program along those guidelines. [DAU,2003]
Acquisition Program Baseline (APB)	Prescribes the key cost, schedule, and cost constraints in the phase succeeding the milestone for which it was developed. (CJCSI 3170.01C) See Key Performance Parameter (KPP).
Product Baseline	The initially approved documentation describing all of the necessary functional and physical characteristics of the Configuration Item (CI); any required joint and combined operations; the selected functional and physical characteristics designated for production acceptance testing; and tests necessary for deployment/installation, support, training, and disposal of the CI. This baseline is usually initiated at the Critical Design Review (CDR) and finalized at the Physical Configuration Audit (PCA), and normally includes product, process, and material specifications, engineering drawings, and other related data.
Best Practice	A documented practice aimed at lowering an identified risk in a system acquisition and is required or recommended by a bona fide DoD, industry, or academic source. -    [Turner, 2002]   Methodologies and tools that consistently yield productivity and quality results when implemented in a minimum of 10 organizations and 50 software projects, and are asserted by those who use it to have been beneficial in all or most of the projects -    Jones, “Software Assessments, Benchmarks, and Best Practices”, 2000
COTS	Commercial Off-The-Shelf
Green Fields	Technology areas in which the organization has not completed a project
Harvesting	The process of combining data from a measurement source with its measurement schema and transferring the resulting object to its storage destination, usually, a data repository [Lawler, 2003]
IDEF	IDEF is short for ICAM DEFinition language, an initiative managed by the US Air Force in the 1970s.  ICAM is short for Integrated Computer-Aided Manufacturing.  These "definition languages" have become standard modeling techniques.  They cover a range of uses from function modeling to information, simulation, object-oriented analysis and design and knowledge acquisition
Management Reserve	An amount of the Total Allocated Budget (TAB) withheld for management control purposes, rather than designated for the accomplishment of a specific task or set of tasks.  It is not a part of the Performance Measurement Baseline (PMB). Synonymous with reserve. From GLOSSARY DEFENSE ACQUISITION ACRONYMS AND TERMS Eleventh Edition September 2003
Measurement Protocol	A description of conditions under which a measurement will be taken, defining the who, when, and how of any data collection activity. [Kitchenham, 2001] A measurement protocol defines and thereby controls its software environment to hold as much as possible constant.  It ensures the repeatability of a measurement so that any difference from one measurement to the next is due to the factor under measurement and not to other confounding variables [Lawler, 2003].
Metric	A mathematical expression composed of one or more atomic measurements or other metrics that can include conditional selection statements [Lawler, 2003]
Requirements Negotiation	Engaging in the explicit trade-off between required functionality, schedule, time, project/product stability, and risk without compromising overall system objectives.
Risk	A standard definition of risk is an uncertain event that would cause an uncertain impact on project schedule, cost, or quality.
SME	Subject Matter Expert
Software Metric	A method of quantitatively determining the extent to which software processes, product, or project possesses a certain attribute [Gopal, 2002]
SPMN	Software Program Managers Network

 

CASE STUDIES from the literature:

Return on Investment from a Software Measurement Program

 By George E. Stark, the MITRE Corporation, 1997

 

http://members.aol.com/geshome/ibelieve/ROI/ROI.htm

 

Description:

The Missile Warning and Space Surveillance Sensors (MWSSS) program management office became responsible for the software maintenance of seven products in November 1994. The total inventory consists of 8.5 million source lines of code written in 24 different programming languages.  The first management decision was to define a variable staff/variable schedule process for software maintenance releases and to implement a software measurement strategy to help understand, manage, and improve the process. This paper describes the software release planning and implementation process and the measurement strategy. It examines the costs of developing and sustaining the measurement program over the past two years as well as quantitative and qualitative returns on that investment.

 

Results:

• The measurement program has returned 187% of the two-year investment in cost savings and 418% in cost avoidance.
• It has also improved understanding of the software release planning process and allowed the organization to answer recurring questions more efficiently.

 

 

Software Project Management Practices: Failure Versus Success©

By Capers Jones, Software Productivity Research LLC, 2004

 

http://www.stsc.hill.af.mil/crossTalk/2004/10/0410Jones.html

 

Description:

An analysis of approximately 250 large software projects between 1995 and 2004 shows an interesting pattern. When comparing large projects that successfully achieved their cost and schedule estimates against those that ran late, were over budget, or were cancelled without completion, six common problems were observed: poor project planning, poor cost estimating, poor measurements, poor milestone tracking, poor change control, and poor quality control.  By contrast, successful software projects tended to be better than average in all six of these areas. Perhaps the most interesting aspect of these six problem areas is that all are associated with project management rather than with technical personnel.

 

Results:

Two working hypotheses emerged:

• Poor quality control is the largest contributor to cost and schedule overruns
• Poor project management is the most likely cause of inadequate quality control

 

 

Finland – Ministry of Justice - Prison Information System

Described in Software Development Projects in GOVERNMENT:

Performance, Practices and Predictions, pages 22 -32

By International Software Benchmarking Standards Group (ISBSG), 2004

 

http://www.isbsg.org/html/Govt%20Sector%20Report%20Rel%201.3%20%201912031.pdf

 

Description:

This case study describes the circumstances, environment and project management that occurred from 1998 to 2004 on this rather complex project, which transitioned the project targeted for cancellation into a successful project.  A major contribution to the project’s success was the introduction of functional sizing and changing from a fixed price model to a model based on a number of Euros per function point delivered.  Functional sizing revealed that the system grew from 3,000 fp in 1998 to 20,500 fp in 2003.   The customer representative to the Project Steering Committee asserts the project could never have succeeded without functional sizing and scope management support provided by the customer’s IT organization.

 

Results:

Summary of the lessons learned

• If possible, ensure that the requirements specifications are complete and contain all the required functionality
• Have in-house consulting support available to the project customer to provide advice if the project runs into trouble
• Where requirements are difficult to define in total prior to the start of a project, or there is the possibility that requirements will be added, do not attempt to work on a fixed price quotation basis
• Use an external scope manager from the beginning of the project to provide balance and objectivity
• Be very wary of rough functional size estimates, use these only as a very broad, initial indication of the possible size of the project
• Be aware of unusual functional complexities, in this case a high number of algorithmic rules or a complex user interface
• Spend the time and money needed to establish an accurate functional size from detailed specifications. (This is a good opportunity to check the completeness of the specifications, if a detailed functional size measurement cannot be completed from the specifications provided then the specifications need more work).

 

“A Comparison of Size Measures for Predicting Web Design and

Authoring Effort”, IEEE Proceedings on Software, June 2002

By Emilia Mendes, Nile Mosley, and Steve Counsell

 

http://www.eicstes.org/EICSTES_PDF/PAPERS/A%20Comparison%20of%20Size%20
Measures%20for%20Predicting%20Web%20Design%20and%20Authoring%20Effort%
20(Mendes).pdf

 

Description:

 

The first half of this paper describes a case study evaluation in which size metrics characterizing length, complexity and functionality were obtained and used to generate effort prediction models for Web authoring and design. The second half describes the comparison of those size metrics as effort predictors by generating corresponding prediction models and comparing their accuracy using box plots of the residuals. Results suggest that in general all categories presented a similar prediction accuracy.  The results from this study somewhat contradict what has been purported in the literature about the value of software size as a predictor of effort.

 

Results:

Analysis revealed the following results:

• With one exception, the estimates were biased towards over-estimation
• None of the models produced reasonable accurate estimates of effort
• There was no single technique that produced a better fitting model than the others
• For the Web applications used in our dataset it seems that measuring the functionality of those applications was very much related to the number of links that the application has.
• Further investigation is clearly necessary to determine whether or not functionality is highly correlated to connectivity

 

There is an urgent need for adequate Web development effort prediction at an early stage in the development; effort estimation can contribute significantly to the reduction of costs and time involved in developing Web applications.

 

 

  “Estimation Models for Software Maintenance Based on Functional Size”, DACS Software Tech News, September 2006

By Alain Abran

https://www.softwaretechnews.com/stn_view.php?stn_id=6&article_id=17

Description:

This paper illustrates the use of functional size measures in building estimation models for maintenance projects implementing small functional enhancements in existing software. More specifically, it reports on estimation models built with 19 maintenance projects on a single a real-time embedded software application in the defense industry. Functional size measures were collected using the COSMIC-FFP functional size measurement method and the maintenance projects were classified into two classes of project difficulty to identify sub-sets of projects with greater homogeneity in their relationship of project effort with respect to functional size.

 

Results:

• For this data set, even though there is a clear positive relationship between functional size and project effort, such a relation was not strong enough to derive good estimation models using this single independent variable with either the average unit costs models or the linear and nonlinear forms of simple regression models.
• The multiplicative (multiple regression) models graphically mapped the two classes of projects much better (low-high) with respect to the other independent variable, functional size, and their criteria as good models of the relationship between size and work effort were much better.
• A process for deriving estimation models has also been illustrated, with both an objectively derived quantitative variable (functional size), obtainable early on in the project life cycle, and a categorical variable, difficulty. Once such models are derived, and their quality analyzed, then they can be quite useful in estimating subsequent maintenance projects involving adding or modifying functions to these software applications.