SOFTWARE ACQUISITION GOLD PRACTICE^™

FORMAL RISK MANAGEMENT

FOCUS AREA: RISK – Risk Management

Definition and Summary: Software Risk Management is a proactive approach for minimizing the uncertainty and potential loss associated with a project. A risk is an event or condition that, if it occurs, has a positive or negative effect on a project’s objectives. The three common characteristics of risk are (1) it represents a future event, (2) it has a probability of occurring of greater than 0%, but less than 100%, and (3) the consequence of the risk must be unexpected or unplanned for. Future events can be categorized as opportunity-focused (positive risk) if their consequences are favorable, or as threat-focused (negative risk) if their consequences are unfavorable.

“If you can’t afford to mitigate the risk now, be absolutely sure that you can afford to resolve the problem later when it happens.” [Universal, 2002]

Providing insights to support informed decision making is the primary objective of Risk Management. In practice, Risk Management concentrates on performing bottom-up, detailed, continuous assessment of risk and opportunity. It focuses on addressing the day-to-day operational risks that a program faces. Risk Management follows a two-stage, repeatable and iterative process of assessment (i.e., the identification, estimation and evaluation of the risks confronting a program) and management (i.e., the planning for, monitoring of, and controlling of the means to eliminate or reduce the likelihood or consequences of the risks discovered). It is performed continually over the life of a program, from initiation to retirement.

Effective implementation of formal risk management is based on the following set of benefits resulting from the process:

§ Appropriately tailored risk management strategies are defined and implemented

§ Potential problems (risks) that could impact project success are identified

§ The likelihood and consequences of these risks are understood

§ A priority order in which risks should be addressed is established

§ Mitigation alternatives appropriate for each potential problem are carefully considered based on project circumstances

§ Optimized mitigation techniques for all risks above their thresholds are selected

§ Contingency plans in case the risk mitigates are developed proactively, rather than as a result of fire-fighting

§ Information to improve risk management policies is captured, analyzed and acted upon

§ Risk management processes/procedures are systematically and periodically reviewed and improved to further reduce risk

DESCRIPTION OF THE PRACTICE:

SUMMARY DESCRIPTION

There are a variety of risks that confront the global software industry, as illustrated in the figure below. The characteristics of the legal, social, economic and competitive environments impose constraints and opportunities that help to define the nature of the risks (and their exposure levels) for suppliers, buyers, and other stakeholders in the software acquisition and development process.

Rather than meeting risks head on, however, and despite increased efforts to promote sound risk management processes, the software industry, as a whole, tends to be in denial about whether risks exist on their programs and, if they do, how they should go about addressing them. The figure below, adapted from the Software Program Manager’s Network (SPMN) “The Little Book of Bad Excuses” [SPMN, 1998] highlights some of the most common traps that all too many programs have fallen into over the years, and continue to fall into today. As a result, symptoms that organizations are not effectively practicing risk management continue to flourish, including a continual state of project instability, constant fire-fighting, multiple schedule slippages because of recurring surprise factors, and constantly operating in a high-stress, crisis management environment.

(click to enlarge)

A formal risk management process is a continuous process for systematically addressing risk throughout the product/project life-cycle. Risks can be introduced (or latently reside) at the very earliest stages of the software life-cycle. The ability to identify risks earlier translates into earlier risk removal, at less cost, which promotes higher project success probability.

DETAILED DESCRIPTION

INTRODUCTION

There are a variety of risks that confront the global software industry, as illustrated in Figure 1 [McManus, 2004], which will be discussed in more detail. The characteristics of the legal, social, economic and competitive environments impose constraints and opportunities that help to define the nature of the risks (and their exposure levels) for suppliers, buyers, and other stakeholders in the software acquisition and development process.

Figure 1: Risk and the Global Software Industry [McManus, 2004]

Rather than meeting them head on, however, and despite increased efforts to promote sound risk management processes, the software industry, as a whole, tends to be in denial about whether risks exist on their programs and, if they do, how they should go about addressing them. Figure 2, adapted from the Software Program Manager’s Network (SPMN) “The Little Book of Bad Excuses” [SPMN, 1998] highlights some of the most common traps that all too many programs have fallen into over the years, and continue to fall into today. As a result, symptoms that organizations are not effectively practicing risk management continue to flourish, including a continual state of project instability, constant fire-fighting, multiple schedule slippages because of recurring surprise factors, and constantly operating in a high-stress, crisis management environment.

Figure 2 (click to enlarge):
The Stubborn Face of Risk Denial (adapted from The Little Book of Bad Excuses) [SPMN, 1998]

There are a number of good resources that helped to initially define and add rigor to software risk management approaches. Capers Jones provided some historical perspective on the evolution of software risk assessment processes through the evolution of Software Productivity Research (SPR) activities into key process areas of the Software Engineering Institute (SEI) Capability Maturity Model (CMM). Additional rigor was added to the software risk management approach through the contributions of Barry Boehm (“Software Risk Management” [Boehm, 1989]) and Robert N. Charette (“Software Engineering Risk Analysis and Management” [Charette, 1989] and “Application Strategies for Risk Analysis” [Charette, 1991]).

There are numerous reasons as to why formal risk management processes have not been implemented more widely, however, or enjoyed more success. Some of the more interesting suggestions have been made by Kontio [Kontio, 2002], who suggests that the success of the business market (before the dot.com bubble burst) coupled with the expected rigor of a formal risk management process was felt to be, simply, not worth the investment in time or money (see Table 1).

Table 1: Reasons for Lack of Progress in Software Risk Management [Kontio, 2002]

Reasons for Lack of Progress	Proposed New Agenda
The .com boom was a false perception of success: Companies were not concerned about risks; only opportunities. Investors were uninterested in conservative, low-risk investments.	Business attitudes in software companies need to take both risk and opportunities into account
The wrong risk management techniques were being used: Simple techniques (checklists, scoring tables, complex calculation tools) deployed in 1990’s were not necessarily based on sound theories, good assumptions, or accurate mathematics. Industry practitioners may have felt that spending time on risk management was not cost-effective.	Biased and unsound software risk management methods should be replaced by better methods
Risk management was too bureaucratic: Numerous report templates and standards were defined, and many lengthy reports produced, but not necessarily read or acted upon. Therefore, risk management created high overhead cost while producing little of value.	Software risk management needs to be made cost effective and easy, while still based on sound principles
Risk management was too standard: Many risk management approaches/tools aimed to make their use easier by standardizing on some aspects of their models. Risks often valued only through money, calendar or effort impacts. Similar checklists are applied to large defense contracts and small start-up companies. Realities are much more complex than this.	Software risk management needs to be application or situation specific. There are no “silver bullets” or “one size fits all” solutions.
Risk management was not transparent: Many risk management methods/tools use complex calculations and forms to document/analyze risk information. Little attention was paid to make sure all decision-makers understood the process and results. Since they didn’t understand them, they didn’t have confidence in them and could not communicate the results well.	Risk management methods must emphasize communication and understandability of results
Researchers were too theoretical and consultants too naïve: Researchers ignored practical implications or constraints. Lack of interdisciplinary research (risk management methods not leveraged from psychologists, economists, civil engineers, etc.). Consultants have used/reused the same old simple methods without true concern about their serious limitations.	Researchers need to provide more practice results. Consultants need to upgrade their knowledge and gain a deeper understanding of methods/tools.

The literature has, however, reported some success in risk management’s effect on projects. In one study conducted by the Project Management Institute (PMI) [Voetsch, 2004], the underlying rationale was to gain an appreciation of the role of risk in relation to other project management success factors. In order to accomplish this, the study used a population with a demonstrated sensitivity to risk, namely PMI’s Risk Special Interest Group. Based on the 187 responses received during the winter of 2002-2003, the research showed that risk management success followed when adequate support and resources were applied to implement it. Specifically, conclusions were drawn that:

· Senior management’s sensitivity to risk increases the frequency of risk management practices

· Availability of adequate resources increases the use and effectiveness of risk management tools

· More frequent use of formal risk planning practices increases the use and effectiveness of risk monitoring

· Senior management support for up-front risk planning increases the probability of project success

The referenced report contains some relatively positive quantitative findings regarding the use of risk management practices, but on the negative side:

· Less than 5% of respondents reported “almost always” receiving encouragement for risk-taking vs. 18% who “rarely” received such encouragement

· Only 12% of respondents reported “almost always” receiving adequate resources for risk management, while only 7% reported that project team members “almost always” receive training in risk planning and impact analysis

· Only 20% of respondents reported that they “almost always” used formal risk reviews, and only 8% said that they used risk audits during their projects

· Only 21% of respondents reported that their projects are completed according to their specifications or the original Statement of Work (SOW)

· Only 30% of respondents reported “almost always” completing their projects on time, and only 26% reported completing them within budget

· Only 40% of respondents reported that their projects “almost always” are received with customer satisfaction upon delivery

Clearly, there is significant progress that needs to be made in order to use risk management techniques effectively.

The Concept of Positive Risk

Although there has been considerable debate regarding the proper definition of software risk, the traditional consensus has been that risk always involves two characteristics: uncertainty and loss. While uncertainty certainly still holds true, there is a growing perception that risk can also involve opportunity as well as loss.

Positive risk refers to risk that we initiate ourselves because we see a potential opportunity along with a potential for failure (the negative risk associated with “loss” of the opportunity). There are several kinds of opportunities that can be leveraged in projects if responses to them are well-timed and prompt action is initiated [Kahkonen, 2001]. These include:

· Business opportunities, e.g., product development, customer care during the project life cycle, and focused attention on high profit margin activities

· Operational opportunities, e.g., value-added, do what is important, minimize rework

· Systemic opportunities, which typically mean long-term savings resulting from improved safety, insurance, etc.

Two examples of positive risk follow [Mochal, 2002]. Example 1 is a software development project that is scheduled to take 90 days. The customer would prefer earlier delivery, and would get more value from an earlier delivery, but understands and concurs with the 90-day development period. Utilizing a new software-testing tool might allow delivery in 60 days, but this is the first time that your organization has used the tool (lack of expertise; learning curve required). If the tool doesn’t work out, delivery might take 110 days, rather than 90.

In our second example, it is estimated to take six months to complete a medium-sized software development project. You may be able to complete the project sooner using Extreme Programming techniques, delivering the first iteration in three months, then monthly updates after that. There is a risk, however, that new techniques will not work well, or that staff may resist the changes, causing a delay in the project.

Although most of the remaining discussion will focus on the traditional, negative aspects of risk, it would be beneficial to think about how the techniques of formal risk management presented here could be tailored to successfully leverage positive risk.

As we get further into our discussion on formal risk management processes and techniques, keep in mind that there are always trade-offs that need to be considered in terms of how much to invest in risk management versus the expected payback over the total software project life-cycle. Figure 3 helps to illustrate this point.

Figure 3: The Role of Risk in Software Affordability

This figure illustrates the concept that a relative optimum point of investment exists for deciding how much money and other resources should be applied to a formal risk management program. Proceeding to the right on the x-axis represents decreasing risk, and C_TLCC represents the total software life-cycle cost, C_RR is the cost associated with investment in risk reduction activities, and C_OM are the total operating and maintenance costs associated with the ability to successfully reduce risk. Starting at the origin, increasing investment in formal risk management has the impact of decreasing the project risk while simultaneously decreasing the total life-cycle cost of the software. Beyond our optimal point, however, continuing to invest in risk reduction activities will still continue to reduce risk, but the cost of doing so will exceed the degree of improvement experienced in operating and maintenance costs. Even though we’re reducing risk, the total life-cycle cost of the software will increase. These are the types of decisions that must be made in determining the scope of a formal risk management program, and they influence our choices in how we choose to handle risk. Issues regarding risk avoidance, mitigation, transferal, etc., will be discussed later, but there is no silver bullet that can guide the decision-making process. Each project (and the needs of the business at specific points in time and under certain market conditions) will most likely require specifically tailored risk management solutions.

The Most Common/Serious Software Risks

There are numerous reasons as to why formal risk management is difficult to implement effectively. These include the sheer number of risk factors that have been identified in the literature. For example, Capers Jones assessed several hundred organizations and observed over 100 risk factors (of which 60 he discusses in detail in [Jones, 1994]). He observed, however, that few projects have more than 15 active risk factors at any one time, but many projects have approximately six simultaneous risk factors.

Another reason for the relatively low implementation of formal risk management methods in practice are, according to [Kontio, 1998], the fact that risk is an abstract or fuzzy concept for which users lack the necessary tools to more accurately define risk for a deeper analysis. In addition, many risk management methods may be based on risk quantification. Users may not have the ability to provide accurate estimates for probability and loss/opportunity projections required for a reliable risk analysis. Table-based approaches can sometimes be too biased or too coarse for proper risk prioritization. Risks may also have different implications for different stakeholders (or, conversely, be perceived differently by different stakeholders). Existing risk management methods may not provide support for dealing with these differences. Risks may also affect a project in more than one way. For example, most risk management approaches focus on cost, schedule or quality risks, but there may be combinations of risks or other characteristics such as future required maintenance, company reputation, or potential liability/litigation that should be considered important in influencing the decision-making process. Finally, many current risk management techniques may be perceived as too costly or too complex to use. Simple, straightforward risk management techniques that require an acceptable amount of time to produce results might be the answer.

So, what kinds of risks are likely to be encountered in the acquisition and development of software?

There are a number of sources in the literature that expound upon the different types of software risks and problems. References and, where available, Internet links are included in the bibliography section of this document. These references include [Jones, 1994], [Boehm, 1991], [McManus, 2004], [GAO, 2004a], [GSAM, 2000], [DAU, 2003], [STSC, 2003], [Buttigieg], [Kontio, 2002], and [Weigers, 1998].

Since there is a great deal of overlap between them (although some have some slightly different perspectives on potential software project risks) only a few of them will be addressed in the body of this document, primarily because the references provide some additional insight beyond just identifying the potential software program risks. Readers should use these references as a trigger for identifying risks, causes and potential solutions that should be considered in support of their own projects.

Table 2 is of particular interest because of the discussions of each risk in the context of other potential related problems on the project, or in the larger organization. It provides an interesting summary perspective on the concept of compound risks. Jones book goes into much more detail regarding these risks. An expanded version of this table has been prepared and made available by the DACS at http://www.goldpractices.com/practices/frm/SoftwareRisks-Jones1994.doc.

Table 2: The Most Serious Software Risks [Jones, 1994]

Software

Risk

Characteristics

Possible Compounding Risks

Canceled Projects

A. Software projects that are terminated prior to delivery to their intended customers

· Almost always associated with Friction with Users and Friction with Senior Management

· Strong association with Low Employee Morale, Low Managerial Morale and High Staff Attrition

· Strongly associated with Lost Business Opportunities and Low Market Shares

· Canceled projects can be caused by Inadequate Planning, Inadequate Cost Estimating, Missed Schedules, Long Schedules, Excessive Time to Market, Cost Overruns, Inexperienced Management, Management Malpractice, Inexperienced Clients, Creeping User Requirements, Inadequate Development Processes, Low Productivity, and Low Quality

Creeping User Requirements

A. New requirements or significant modifications to existing requirements that are made after a basic set of requirements have been mutually agreed to

B. Widespread failure to anticipate changing requirements, with no plans for how to deal with them

· Associated with Friction with Users, and sometimes with Friction with Senior Management

· Causative factors of Missed Schedules, Excessive Time to Market, and Cost Overruns

· Contributing factor to Excessive Schedule Pressure and Low Staff Morale

· Creeping requirements can be caused by Inexperienced Users, Inexperienced Management, Inadequate Methodologies, Inadequate Cost Estimating and Inadequate Measurement

Excessive Schedule Pressure

A. Coercive demands by software clients to deliver software applications in a shorter time period than is technically achievable

B. Coercive demands by software executives or project managers to deliver code, specifications or documentation in a shorter time period than is technically possible

· Significant contributor to Low Staff Morale and High Staff Turnover

· Observed on a majority of Cancelled Projects and those with Cost Overruns

· Main contributor to Low Quality

· Excessive schedule pressure can be caused by Creeping User Requirements, Inadequate Estimating, Inadequate Measurement, Inadequate Planning and Management Malpractice

Inaccurate Cost Estimating

A. Automated estimating methods that vary by >30% when calibrated against fully-measured control projects

B. Automated partial estimating methods that omit, ignore or fail to include >30% of the activities and tasks associated with software projects unless specifically limited to selected activities

C. All forms of manual estimating over 1000 Function Points

· Associated with Cost Overruns, Long Schedules, Missed Schedules, Excessive Time to Market, Lost Business, Friction with Senior Management, Friction with Users, Inaccurate Quality Estimates, Inaccurate Reliability Estimates, Low Quality and Cancelled Projects

· Inaccurate cost estimates can be caused by Corporate Politics, Creeping Requirements, Inadequate Measurement, Inadequate Software Management Curricula, Inadequate Software Engineering Curricula, Inexperienced Management, Lag Time in Technology Adoption, Inaccurate Metrics, Poor Support Structures and Poor Technology Transfer

Inaccurate Metrics

A. Common software metrics that violate standard economic assumptions

B. Common software metrics that behave in paradoxical, counterintuitive or unpredictable ways

C. “SLOC” and “KLOC” metrics when used as general-purpose software quality or productivity metrics

D. “Cost per defect” when used as an indicator of quality costs

E. “Software Science” metrics when used as general productivity or quality indicators

F. Ratios and percentages when used as general productivity indicators

· Associated with Poor Technology Investment, False Productivity Claims, Inadequate Cost Estimating and Inadequate Measurements

· Causative factors to Missed Schedules, Long Schedules, Low Productivity and Cancelled Projects

· Contributing factors to Friction with Executives, Friction with Users, Friction with Contractors and Low Management Morale

· Characteristic (C) is associated with Poor Technology Investments, False Productivity Claims, Low Productivity, Inadequate Cost Estimating and Inadequate Measurements

· Characteristic (D) is associated with Poor Technology Investments, Low Quality and Inadequate Defect Removal

· Inaccurate metrics can be caused by Academic Malpractice, Management Malpractice, Inadequate Management Curricula and Inadequate Standards

Inadequate Measurement

A. Failure to record the basic and special factors that influence software projects, coupled with either inaccurate or missing quantitative data on size, staffing, schedules, resources, costs and quality, also coupled with the use of paradoxical or unsound metrics

· Root cause of Low Productivity, False Productivity Claims, Low Quality, Long Schedules, Missed Schedules, Excessive Time to Market, Inadequate Estimating, Friction with Senior Management, Friction with Users, Low User Satisfaction, Canceled Projects, Inadequate Methodologies, Inadequate Tools, Poor Organization Structures, Inadequate Specialization, Inadequate Capital Investment and Inadequate Office Environments

· Inadequate measurements can be caused by Inadequate Software Management Curricula, Inadequate Software Engineering Curricula, Inexperienced Management, Lag Time in Technology Adoption, Inaccurate Metrics, Poor Organization Structures, Poor Technology Transfer and Poor Corporate Culture

Low Productivity

A. Software projects that are significantly below U.S. industry averages for their class, type and size

B. Any project that averages less than five Function Points per person month in net project productivity (the “current” U.S. average – Note that this is the U.S average as of the 1994 publication date of Caper Jones’ book)

C. Any project perceived as being of low productivity by its clients or by senior management

· Basic contributor to Friction with Users and Friction with Senior Management

· Causative factor to False Productivity Claims

· May sometimes be a factor leading to Canceled Projects and Excessive Time to Market

· Low productivity can be caused by Low Levels of Reusability, Inexperienced Management, Inexperienced Staff, Inadequate Capital Investment, Inadequate Office Environments, Inadequate Methodologies, Inadequate Tools, Inadequate Measurements, Inadequate Metrics and Short-Range Improvement Planning

Low Quality

A. Software where significant numbers of users are dissatisfied with quality and/or software receiving more than 0.25 user defect reports per Function Point per year

· Key contributor to Low User Satisfaction and Friction with Users

· Contributing factor to Low Market Shares and Excessive Time to Market

· Causative factor to Friction with Senior Management, Low Staff Morale, Error-Prone Modules and High Maintenance Costs

· Often associated with Low Productivity and Long Schedules

· Low quality can be caused by Inexperienced Management, Inexperienced Staff, Creeping User Requirements, Inadequate Estimating, Inadequate Planning, Inadequate Defect Removal, Inadequate Measurements and Excessive Schedule Pressure

Malpractice (Project Management)

A. Performing managerial functions so poorly that harm/damage occurs to projects which are under the manager’s scope of authority

B. Failure to exert proper authority to prevent harm/damage to projects which are under the manager’s scope of authority

C. Repeated/significant errors (more than 35%) in performance of basic software management tasks of schedule planning and cost estimating

D. Repeated delivery of software packages with excessive defect levels

E. Failure to keep adequate measurement records of projects produced under their authority

· Significant contributor to High Staff Attrition Rate and Low Staff Morale

· Strongly associated with Canceled Projects, Corporate Politics, Cost Overruns, Excessive Schedule Pressure, Excessive Time to Market, Friction with Users, Long Schedules, Low Quality, Missed Schedules, Technical Malpractice and Silver Bullet Syndrome

· Management malpractice can be caused by Inadequate Management Training, Inadequate Career Planning, Excessive Schedule Pressure, Executive Malpractice, Resistance to Change and Creeping User Requirements

· May be caused by Inadequate Compensation Plans

Silver Bullet Syndrome

A. The concept that a single tool or methodology will be the “silver bullet” that cures software productivity or quality problems

· Significant contributor to Friction with Users and Friction with Senior Management

· Causative factor for Canceled Projects, Excessive Time to Market, Long Schedules and Cost Overruns

· Contributes to High Staff Attrition, Low Staff Morale, Low Productivity, Low Quality and Short-Range Improvement Planning

Table 3 is derived from the Guidelines for Successful Acquisition and Management of Software-Intensive Systems (GSAM)”, Version 3.0 [GSAM, 2000]. It is included here due to its more direct focus on the software acquisition process, and is noteworthy for its focus on the technical aspects of a project, rather than being limited to cost, schedule and organizational risks.

Table 3: Significant Acquisition Risks to Address in the Acquisition Strategy [GSAM, 2000]

Risk Area	Significant Acquisition Program Risks
Threat	· Uncertainty in threat accuracy and stability · Sensitivity of design & technology to threat · Vulnerability of system to threat countermeasures · Vulnerability of program to intelligence penetration
Requirements	· Operational requirements not properly established or vaguely stated for program phase · Requirements are not stable · Required operating environment not described · Requirements do not address logistics and suitability · Requirements are too constrictive – identify specific solutions that force high cost
Design	· Design implications not sufficiently considered in concept exploration · System will not satisfy user requirements · Mismatch of user manpower or skill profiles with system design solution or human-machine interface problems · Increased user skills or more training requirements identified late in the acquisition process · Design not cost effective · Design relies on immature technologies or excessive use of COTS to achieve performance objectives
Test & Evaluation	· Test planning not initiated early in program · Testing does not address the ultimate operating environment · Test procedures do not address all major performance and suitability specifications · Test facilities not available to accomplish specific tests, especially system-level tests · Insufficient time to test thoroughly
Modeling & Simulation	· Same risks as those identified for Test & Evaluation (T&E) · Modeling & Simulation (M&S) not verified, validated or accredited for the intended purpose
Technology	· Program depends on unproven technology for success, or there are no alternatives · Program success depends on achieving advances in state-of-the-art technology · Potential advances in technology will result in less-than-optimal cost-effective system, or make system components obsolete · Technology has not been demonstrated in the required operating environment · Technology relies on complex hardware, software or integration design · Program lacks proper tools and M&S capability to assess alternatives
Logistics	· Inadequate supportability late in development or after fielding, resulting in need for engineering changes, increased costs, and/or schedule delays · Life-cycle costs not accurate because of poor logistics supportability analysis (LSA) · LSA results not included in cost-performance trade-offs · Design trade studies do not include supportability considerations
Development	· Development implications not considered during concept exploration · Development not sufficiently considered during design · Inadequate planning for long-lead items and vendor support · Development processes not proven · Prime contractors do not have adequate plans for controlling subcontractors · Sufficient development tools not readily available for cost-effective production · Contract offers no incentive to upgrade tools, improve processes, or reduce costs
Concurrency	· Immature or unproven technologies will not be adequately developed prior to system production · Development funding will be available too early, i.e., before the development effort has sufficiently matured · Concurrency established without clear understanding of risks
Developer Capability	· Developer has limited experience in specific type of development · Contractor has poor track record relative to costs and schedule · Contractor experiences loss of key personnel · Prime contractor relies heavily on subcontractors or Commercial Off-The-Shelf (COTS) items for major development efforts · Contractor will require significant capitalization to meet program requirements
Cost/Funding	· Realistic cost objectives not established early · Marginal performance capabilities incorporated at excessive costs; satisfactory cost-performance tradeoffs not done · Excessive life-cycle costs due to inadequate treatment of support requirements · Significant reliance on software · Funding profile does not match acquisition strategy · Funding profile not stable from budget cycle to budget cycle
Schedule	· Schedule not considered in tradeoff studies · Schedule does not reflect realistic acquisition planning · Acquisition Program Baseline (APB) schedule objectives not realistic and obtainable · Resources not available to meet schedule
Management	· Acquisition strategy does not give adequate consideration to various essential elements, e.g., mission need, test and evaluation, technology, etc. · Subordinate strategies and plans are not developed in a timely manner or based on the acquisition strategy · Proper mix (experience, skills, stability) of people not assigned to Program Management Office (PMO) or to Contractor team · Effective risk assessments not performed or results not understood and acted upon

Table 4, although it reverts back to the more “traditional” high-level risk concerns of cost, schedule and organization, is useful because it includes information related to symptoms, potential root causes of the risk, quantitative leading indicators that the risk has manifested, and generic mitigation techniques to address the risk.

Table 4: Standard Risks for Project Failure [STSC, 2003]

Risk

Symptom

Root

Cause

Leading

Indicators

Mitigation Techniques

Undersizing & Unrealistic Schedules

· Projects are late

· Disconnect on schedule status

· Schedules based on product need instead of required engineering effort

· Size estimates typically low, resulting in schedule estimates that are too short

· Earned value (SPI lagging – can’t make up more than 5%)

· Staff (not management) comfort with schedule

· Rational planning process

· Understand and track estimation range over time

· Understand and track size/schedule estimation accuracy over time

Unconstrained Requirements Growth

· Development staff can’t keep up with requirements changes (change rate too high or scope of changes too big)

· Requirements change not managed

· Requirements change rate (1% is OK; 2% means stop and stabilize requirements; 0% - does anyone want your product?)

· Don’t start development until there is a reasonably stable set of requirements

· Plan for change

Dysfunctional Organizational Culture

· High personnel turnover

· Frequent personnel reassignment

· Personnel assignment to multiple projects

· Excessive/over-attended meetings

· Poor IPT implementation

· Poor work environment

· Low productivity

· Lack of motivating work environment

· Poor management project prioritization

· Project staff turnover vs. historical norms, vs. staff turnover plan (>10% per year is red flag)

· Productivity (lower than planned)

· Understand/measure current status and address issues

· Difficult to address because root cause is, by definition, risk management

Dysfunctional Organization

· Staff lacks key skills and experience

· Key roles not filled by staff

· Lack of experience in leading software intensive projects by management, leading to incomplete teams

· Project technical status can’t be explained by staff

· System and/or software architecture can’t be explained

· No version or configuration control of product

· Outside expertise to assess staffing plan (peer managers, formal assessments

Poor Software Quality

· High test defect counts

· Excessive test event duration

· High quality warranty rework

· Lack of dedication to software quality (misguided focus on schedule; quality focus on testing, not early defect removal)

· Lack of disciplined work environment

· Inspection yield

· Test defect density

· Defect discovery and closure profiles

· Create a development quality plan that focuses on inspection rather than test, that estimates defect counts during each life cycle phase

· Measure and track against quality plan

· Cultivate a focus on software quality

Specification Breakdown

· Inability to produce a real specification

· Development started before requirements are reasonably stable

· Conflict within the stakeholder community

· Lack of discipline on the development team

· Real specification should be nominally complete (1) when 15% of planned project manpower has been expended or (2) before 20% of the schedule has elapsed

· Triggers are early indication of impending project failure, but often ignored

· Consider project cancellation if triggers are tripped

· Ensure that specification is baselined early in the project (but don’t force it – stakeholders must resolve conflicts)

Underperformance

· Productivity lower than planned

· Application complexity

· Development environment changes from historical baseline

· Other activities (e.g., process improvement program)

· Development process not capable

· Underachievement

· Earned value (CPI)

· Productivity (actual vs. planned vs. product norms)

· Understand (measure) the realized productivity

· Shield staff, review the development process, adjust plans if necessary, recognize underachievement as an early warning sign of a dysfunctional culture

Belief in Magic (Unproven Technologies)

· Silver bullet mentality

· Unquestioned project reliance on unproven technology (e.g., COTS, subcontracting, outsourcing)

· Actual technology use vs. plan

· Quality of the technology products (do they work?)

· Productivity (are intended technology benefits realized)

· Risk plan for the new technology

· If it sounds too good to be true…

· Better risk management of the project and of technology-specific issues

Long Schedule

· Development schedules >2 years

· Maintaining control of changing requirements over long time periods

· Improperly assuming software schedule must match hardware schedule

· Schedule development takes >2 years

· Rework schedule using evolutionary or spiral techniques so that frequent deliveries are scheduled

If you’re wondering whether any or all of these risks are really applicable to you, consider one example of a relevant software risk factor. There is an emphasis on the use of COTS software to develop the DoD’s Future Combat System (FCS) (and isn’t this true of most combat/weapons systems these days). There is a very legitimate concern that if there are too many COTS software products, versions and releases contained in the various FCS elements the software will not integrate, causing serious delays. This concern is based on the fact that COTS software products typically undergo new releases every 8 or 9 months, and that COTS software products generally become unsupported after 3 new releases. Therefore, within a 30-month period, the risk of unsupported software on FCS is considered high. Strategies for mitigating this risk include the development of contract/subcontract incentives to ensure interoperability across delivered COTS products, and development of a management tracking scheme for all COTS products in all FCS software elements.

A Rapidly Emerging Critical Software Risk

A key software risk area, and one that has an immediate and potentially catastrophic impact on software projects, is related to the development of COTS and outsourced software. In a report entitled “Knowledge of Software Suppliers Needed to Manage Risk” [GAO, 2004a], the Government Accountability Office (GAO) was asked to examine DoD’s efforts to identify software development suppliers and manage risks related to foreign involvement in software development on weapon systems. This concern covers not only contracted software development work, but also is important in the procurement of COTS software.

Unfortunately, the GAO study identified that DoD acquisition and software security policies do not fully address the risk of using foreign suppliers to develop weapon system software. Current acquisition guidance allows program managers discretion in managing foreign involvement in software development, without requiring them to identify/mitigate such risks. Other policies that are intended to mitigate Information System vulnerabilities focus primarily on operational software security threats (such as external hacking and unauthorized access to information systems), but not on insider threats such as insertion of malicious code by software developers.

As a result of this study, GAO recommended that DoD better define software security requirements and require program managers to mitigate associated risks accordingly. While the DoD shared the concerns raised by this report, they felt that the GAO recommendations placed too much responsibility for risk management with the program managers. As a result, GAO has broadened the scope of their recommendations.

Given the rising threat to software security as a result of terrorist activities, industrial espionage, etc., coupled with the increasing proliferation of software viruses, worms and trap-door vulnerabilities, the emerging discipline of Software Assurance is becoming increasingly critical to DoD’s ability to identify and mitigate software security and vulnerability risks.

Risk Management Process

In the previous section we identified what seemed like an endless list of all possible risks that need to be considered in a software acquisition and development project. Knowing what those risks could be, however, is not the same as knowing what they are on your particular project under your particular circumstances. There are any number of unique or inherent internal/external scenarios that may influence the possible risks that your project or organization is susceptible to. The key to successful risk management lies in the ability to tailor a formal risk management process that addresses the complementary needs of the business and its customers.

Hall [Hall, 1997] and others view software risk as being comprised of two key areas (management and technical), categorized into project, process and product (see Figure 4). Software project risk defines the operational, organizational and contractual software development parameters, and is seen as primarily a management responsibility. Software project risks may include resource constraints, external interfaces, supplier relationships, contract restrictions, unresponsive vendors and lack of organizational support. Software project risks are typically difficult to manage due to a perceived lack of control over project external dependencies. Funding is the most significant project risk reported in many risk assessments. [Hall, 1997]

Software process risk includes both management and technical work procedures. In management procedures, process risk may be found in activities such as planning, staffing, tracking, quality assurance and configuration management. In technical processes, it may be found in activities such as requirements analysis, design, code and test. Planning is the management process risk that is most often reported. The most reported technical process risk is the software development process itself. [Hall, 1997]

Software product risk contains intermediate and final work product characteristics. Primarily a technical responsibility, product risk may be found in requirements stability, design performance, code complexity and test specifications. Product risk is difficult to manage because software requirements are often perceived as flexible. Requirements are the most significant product risks reported in risk assessments. [Hall, 1997]

Figure 4: Software Risk Classifications [Hall, 1997]

Beyond the technical and management sources of risk, there may also be sources external to the project or organization (e.g., the nature of the marketplace, the business culture, etc.). The base of the pyramid in Figure 5 highlights what some of these external risk sources might be.

Figure 5: Sources of Project Risk

The complexities of software risk, then, require a formal methodology for planning, identifying, assessing, monitoring and controlling them – a formal risk management process.

One of the most straightforward representations of a formal risk management process was published by Boehm (Figure 6, [Boehm, 1989]) and still reflects the necessary basic elements and structure for understanding and implementing a successful risk management process today. The figure describes not only the basic steps and hierarchical nature of the process, but it also identifies some of the specific artifacts and activities that can be used to support the process. Some of these artifacts and activities will be covered in more detail in the next section.

Figure 6: Software Risk Management Steps [Boehm, 1989]

In general, a risk management process should consist of the following activities, as defined in IEEE Standard 1540-2001 which supports the acquisition, supply, development, operation and maintenance of software products and services [IEEE, 2001]:

· Planning/implementing a risk management process

· Managing the project risk profile

· Performing risk analysis

· Performing risk monitoring

· Performing risk treatment (a.k.a., risk mitigation)

· Evaluating the risk management process

[It should be noted that the IEEE has revised its 1540 standard so that it aligns with existing ISO/IEC software engineering standard ISO/IEC 16085.]

The IEEE Software Risk Management Process Model is illustrated in Figure 7. Steps 1, 2, 3, 6 and 7 of the IEEE process coincide with the “Risk Control” branches of the Boehm model, while Steps 4 and 5 reflect those activities included under the “Risk Assessment” structure of Boehm’s Risk Management Process.

Figure 7: Software Risk Management Process Model [IEEE, 2001]

Technical and managerial processes define the risk information requirements that get passed to Step 2, where policies regarding project-specific risk management guidelines are defined, as well as procedures to be used, specific techniques to be applied, etc. Step 3 captures historically-based management decisions based on the perceived risks at the beginning of the project. The risk analysis activity (Step 4) identifies specific project risks; determines their likelihood and consequences; determines their risk exposures; and prepares risk action requests recommending appropriate risk mitigation (or treatment) if their risk threshold is exceeded. Mitigation recommendations, along with the status of other risks and their corresponding mitigation activities, are reviewed by management in Step 5, where decisions for mitigating unacceptable risk are made. All risks are continually monitored until they are no longer relevant (Step 6) and potential new risks are sought out. In order to ensure the effectiveness of the risk management process, periodic evaluation is performed on information and data captured during Step 7 in order to improve the process or the ability to manage risk, as necessary and appropriate. Any defined improvements re-enter the process at Step 2, and any number of iterations of the risk analysis activity of Step 4 may be performed in order to increase knowledge of potential risks.

An extremely thorough and insightful look into all of the elements that comprise risk management and its associated processes, including detailed discussions related to Risk Management Discovery, Risk Management Process, Risk Management Infrastructure, Risk Management Implementation and case studies of People in Crisis and Control, is included in Dr. Hall’s book “Managing Risk” [Hall, 1997]. The book provides as much of a humanistic viewpoint for understanding risk management as it does an approach to the risk management process itself.

Hall’s representation of the risk management process (Figure 8) conforms to the structure of Boehm’s model and maps into the appropriate areas of the IEEE Model. The blue-shaded areas (Identify Risk and Analyze Risk) include those activities that are required to assess risk. The yellow-shaded areas (Plan Risk, Track Risk and Resolve Risk) cover the activities that are necessary to control risk.

Figure 8: The Risk Management Process [Hall, 1997]

The potential artifacts and activities associated with each element of the process are identified. The emphasis on strategies, tools, techniques, measurements, metrics and databases should be noted. No risk management process can truly succeed without them. A Risk Management Capability Matrix developed by Odegard [Odegard, 2002] and reflecting a similar philosophy as that of the SEI’s CMM, reflects that emphasis (Figure 9).

Figure 9: The Risk Management Capability Pyramid [Odegard, 2002]

In developing her P²I² approach (Process, People, Infrastructure, Implementation) to risk management and her Risk Management Map, Hall addressed two perspectives that hinder routine risk management [Hall, 1997]. The first is that risk is viewed as an extra activity, layered on top of assigned work. The danger in perceiving risks as less important than assigned work is that risks may not be addressed when work priorities elevate. The second is that risk is perceived as an outside activity that is someone else’s responsibility. The pitfall occurs when the “responsible” person is not around, so that risk management ceases.

The details of Hall’s P²I² process are included in her book, but her Ishikawa diagram illustrating the process is shown in Figure 10. The elements of this model represent the factors that need to be addressed in terms of assessing the capability of an organization’s risk management process.

Figure 10: Success Factors in Risk Management – The P²I² Process [Hall, 1997]

Similar in concept to the risk management capability pyramid presented in Figure 9, Hall also developed a Risk Management Map to help chart a course for increasing the capability to manage software risk. The Risk Management Map contains five evolutionary stages of risk management capability, defined as:

· Problem Stage: Describes circumstances when risk identification is not seen as positive. Characterized by lack of communication which causes a subsequent lack of coordination. Crisis management is used to address existing problems. Risks ignored or tracked in ad-hoc fashion.

· Mitigation Stage: Details a shift from crisis management to risk management. People become aware of risks but do not systematically confront them. There is uncertainty as to how to communicate risks. Risks are usually recorded, tracked and handled as discovered.

· Prevention Stage: Discusses the shift of risk management as solely a manager’s activity to risk management as a team activity. This is a transitional stage from avoidance of risk symptoms to identification and elimination of root cause of risk, characterized by team, and sometimes customer, involvement. For risk management to succeed it must occur at each level within an organization. This stage represents a turning point from a reactive to a more proactive approach to risk management. Risks systematically … analyzed, planned, tracked and resolved.

· Anticipation Stage: Describes the shift from subjective to quantitative risk management, through the use of measures to anticipate predictable risks, that is characterized by the use of metrics to anticipate failures and predict future events. This stage involves the ability to learn from, adapt to, and anticipate change, representing a completely proactive approach to risk management. Quantified analysis used to determine resolution cost/benefit for the project.

· Opportunity Stage: This represents a positive vision of risk management that is used to innovate and shape the future. Risks are perceived as an opportunity to save money and do better than planned. Risk, like quality, is everyone’s responsibility. A continuous process of identifying, communicating and resolving risks in an open and non-threatening environment is used. Admissions that some things are not known are acceptable and allowances are made for their existence using a best-case, worst-case scenario. Risk statistics are used to make organizational/process improvements.

The conceptual view of the Risk Management Map is presented in Figure 11. The map helps to (1) lay the foundation by building the capability needed for the next stage, (2) ease the transition between phases by using an incremental strategy, and (3) stay the course by basing risk management activities on a detailed plan that evolves over time.

Figure 11: Risk Management Map [Hall, 1997]

Between each stage are visions, goals and strategies that govern how the organization intends to transition between the stages. A vision represents an ideal state of the practice that guides the journey, acting as a driving force that provides the motivation needed to continue improvement of the risk management effort. Goals are established and accomplished to achieve the vision. Once a current position is understood, the goals will determine the scope of the next tasks. Finally, a strategy specifies an approach to accomplish the goals, containing important information on how to achieve goals and defining activities that will help check for results. If results do not support goal attainment, tactical adjustments can be made to the strategy. The Risk Management Map divides responsibility between the four major factors of risk management capability and provides the five goals for each major factor of the P²I² Success Formula [Hall, 1997].

Risk Management Tools and Templates

This document, so far, has identified a multitude of risks that a software project may be vulnerable to, and provided an overview of formal risk management processes that can be used and tailored to identify, analyze, prioritize, plan, monitor and resolve/mitigate those risks. This section, then, identifies some of the tools and templates that are available (or can be tailored) to help projects and organizations formally manage their risks.

Checklists are one of the more important tools in helping to manage risks in software projects, and they are not necessarily limited to risk identification issues. For example, Table 5 represents a relatively high-level checklist for the types of things that should be addressed when dealing with software acquisition risk management issues. The appointment of a risk officer and establishment of a risk database are important precursors to ensure that all project acquisition roles are addressed. There are also a number of questions that should be asked relative to the conduct of the risk management program in order to ensure that all risks will be adequately addressed and resolved.. It is important to be in a position to not only monitor and track previously identified risks through the entire life cycle, but also to be sensitive to the emergence of new risks as the life cycle progresses from development, through testing and production, and into fielded operation. Each risk should have its own risk profile that documents its probability of occurrence, the impact of that occurrence, the severity of the impact, and the resulting delays that impact the project. These profiles should be updated on a regular basis, as circumstances warrant. A formally documented Risk Management Plan should contain explicit provisions to alert stakeholders and key decision makers in the event that a risk is considered imminent

Table 5: Acquisition Risk Management High-Level Checklist

· Has a Risk Officer been appointed?	______
· Has a risk database been established?	______
· Do risk assessments clearly impact program plans/decisions?	______
· Is risk assessment update frequency/timeliness consistent with project decision updates?	______
· Are objective criteria used to identify, evaluate and manage risks?	______
· Do information flow patterns and design criteria support risk identification by all project personnel?	______
· Are risks identified throughout the entire software life cycle?	______
· Is there a management reserve to fund risk resolution?	______
· Is there a risk profile for each risk?	______
· For each risk profile, are the probability of occurrence, impact, severity and resulting delay regularly updated?	______
· Does the Risk Management Plan have explicit provisions to alert decision-makers regarding imminent risks?	______

In addition, the McManus reference [McManus, 2004] contains self-assessment checklists that address (1) vision and objectives, (2) accountability, (3) procedures, (4) methods, (5) risk classification, (6) risk statements and reviews, (7) stakeholders, (8) risk assessment approach, (9) risk planning, (10) risk management tools, (11) risk mitigation and implementation, and (12) monitoring and tracking risk

Another common element across all activities in formal risk management is the implementation of a Risk Management Database. Suggested minimum elements of such a database are defined in Table 6 [GSAM, 2000].

Table 6: Suggested Risk Record Elements [GSAM, 2000]

Element	Description
Risk Identification	A description of the perceived risk
Risk Source	A description of the source of the risk. The source can be identified by using a risk factor checklist or risk taxonomy.
Risk Assessment	The risk assessment rating (green, yellow, red), including the probability of occurrence and potential damage, as extracted from the risk assessment matrix.
Aversion Strategy	Recommended aversion strategy with associated mitigation technique (research, accept, avoid, mitigate, transfer) for the risk item. Also, note risk items requiring no action.
Integration Strategy	An integration strategy for individual risk aversion plans (with attention to combining action plans for more than one risk item)
Resource Allocation	Assignment of resources and responsibilities needed to implement the risk aversion strategy (including personnel, cost, schedule and technical considerations)
Criteria for Success	The criteria that must be met for the risk to be considered successfully mitigated
Monitoring Approach	The approach to be used to monitor progress in mitigating the risk
Implementation Schedule	The schedule for implementing the mitigation effort, including key milestones
Mitigation History	A record of the mitigation steps taken, the date taken, who took the steps, and the results
Contingency Plan	The action(s) to be taken if, in spite of all mitigation efforts, the risk does actually manifest.
Cost and Effort	The cost and effort expended in managing the risk

RISK IDENTIFICATION

The Universal Risk Project [Universal, 2002] identifies two basic types of risk statements that can be used to identify whether a set of circumstances represents a risk to the project:

· IF-THEN Statements:

o “IF technology is not available, THEN we will not meet the requirement”

o “IF we cannot hire sufficient qualified software engineers, THEN we cannot meet the planned development schedule

· CONDITION-CONSEQUENCE Statements:

o Given the “condition”, there is a likelihood that the “consequence” will occur

o “Given that this specific test fails (the CONDITION), the CONSEQUENCE is that the planned schedule will slip”

One of the more formal tools to help in risk identification is the SEI Taxonomy-Based Risk Identification report [Carr, 1993]. The report defines three major areas that need to be considered for risk management in a software project. These areas are:

· Product Engineering

o Subdivided into Requirements; Design; Code & Unit Test; Integration and Test; and Engineering Specialties (e.g., reliability, maintainability, etc.)

· Development Environment

o Subdivided into Development Process; Development System; Management Process; Management Methods; and Work Environment

· Program Constraints

o Subdivided into Resources; Contract; and Program Interfaces

Appendix B of the SEI report includes a Taxonomy-Based Questionnaire which can be put into checklist format to ensure that the appropriate elements of software risk are covered on a specific project.

Another tool that can be used to aid in the identification of risks on a software project are strategically defined software measures. In the document “The Top Eleven Ways to Manage Technical Risks” [Navy, 1998], the Navy proposes the use of a software measurement process as a proactive approach for identifying risks before they become problems. The document discusses 42 different measures, each formatted to cover (1) the definition of the measure, (2) guidance for selecting that measure, (3) specification guidance and (4) the questions that the measure is intended to answer. The potential measures for technical risk described in this document include:

1. Milestone Dates

2. Software Component Status

3. Requirements Status

4. Test Case Status

5. Paths Tested

6. Problem Report Status

7. Reviews Completed

8. Change Request Status

9. Build Content – Components

10. Build Content – Function

11. Effort

12. Staff Experience

13. Staff Turnover

14. Earned Value

15. Cost

16. Resource Availability Dates

17. Resource Utilization

18. Lines of Code

19. Components

20. Words of Memory

21. Database Size

22. Requirements

23. Function Points

24. Change Request Workload

25. Problem Reports

26. Defect Density

27. Failure Interval

28. Cyclomatic Complexity (Logic Paths)

29. Rework Size

30. Rework Effort

31. Capability Maturity model Level

32. Product Size/Effort Ratio

33. Functional Size/Effort Ratio

34. CPU Utilization

35. CPU Throughput

36. I/O Utilization

37. I/I Throughput

38. Memory Utilization

39. Storage Utilization

40. Response Time

41. Achieved Accuracy in Software Performance

42. Nondevelopmental Item (NDI) Utilization

RISK ANALYSIS

After risks are identified, they should be partitioned into categories such as technical, cost, schedule, management, etc. Note that some risks may fall into multiple categories.

Why do risks need to be partitioned? First of all, some risks are more important than others. Also, different stakeholders may be concerned about different risks, or different personnel may bear responsibility for tracking/monitoring different risks. Finally, different risk types may require different mitigation strategies.

The initial activity in risk analysis is to identify contributing factors, then establish a hierarchy of those contributing factors. Figure 12 illustrates a hierarchy of how a project might fail, given the contributing factors of Staffing, Funding, Performance Failures, and so on. The Staffing factor is further broken down to show, first, how staffing may become a contributing factor to project failure and second, what the contributing factors might be that result in insufficient staffing (subsequently leading to project failure). All contributing factors defined within the hierarchy would be broken down to a correspondingly meaningful level of detail.

Figure 12: A Sample Hierarchy of Contributing Factors

Similarly, for positive risk, a hierarchy of contributing factors could also be created, this time highlighting those elements for which risk is being undertaken in order to leverage a perceived opportunity for the project, such as “Schedule Completion Will Be Early”.

There are any number of ways that risk can be partitioned, analyzed and quantified. The approach taken and method(s) used should always be tailored to meet the needs of the business, the customer and the project. If you reference the Tools and Methods section of this document, you can review several different MS Excel-based tools developed by the DACS from information available in the literature, each having its own partitioning scheme and offering a different approach to quantifying project risk.

Graphical tools such as Project Risk Maps or Project Opportunity Maps [Kahkonen, 2001] can also be used to highlight whether specific risks represent a potential negative (risk) or positive (opportunity) impact on the project. These maps highlight (1) where attempts should be made to move a “negative risk” out of the red-shaded risk area (by decreasing their probability of occurrence and/or their degree of impact) and (2) which “positive risks” should be pursued, if possible, into the green-shaded opportunity area (by increasing their probability of occurrence and/or their degree of impact) (see Figure 13). Note that for “positive risks”, a.k.a., opportunities, risk mitigation techniques should be defined to maximize the probability that the opportunity will occur (rather than the traditional minimizing the probability that a loss will occur). For our previously discussed positive risk example of using a new software tool to deliver a product earlier than required, a mitigation technique could be to partially outsource the software development effort to someone who is familiar with the tool in question (i.e., no learning curve required), thereby increasing the probability of achieving the opportunity.

Once you have successfully partitioned and quantified project risk, prioritization of risk becomes the next logical activity.

Figure 13: Project Risk and Project Opportunity Maps [Kahkonen, 2001]

RISK PRIORITIZATION

Risk prioritization is a critical characteristic of the formal risk management process, as it provides the opportunity to apply what are typically limited project resources to those risks having the largest potential impact on the project. For many risk prioritization approaches, risks are ranked and prioritized based on some combination of probability (i.e., how likely is it that the risk will occur) and impact (i.e., what are the consequences if the risk does occur). This can be done qualitatively in a risk prioritization matrix or quantitatively using some type of composite probability-impact score. Both of these basic approaches are illustrated in Figure 14.

Figure 14: Qualitative and Quantitative Risk Prioritization Example

Note that the qualitative matrix has some basis in quantitative values for frequency and impact, although these can be subjectively defined and tailored to suit specific sets of circumstances. The highest priority risks would be those falling in the red (intolerable) region. The quantitative example of risk prioritization, shown as a bar graph, highlights the combined probability-impact scores for specific elements of the project. Of course, these are very high-level risks to try to deal with effectively. By the way, there is no connotation of a qualitative risk level (low-medium-high) implied by the color of the bars in this chart, although there could be. Also, the total risk value can be represented by a simple arithmetic average, as it is in this case, or it can be obtained through any desired complexity of category weighting schemes, depending on the level of detail required by the project. In this example, the highest priority risk is schedule failure, followed by technical failure, cost failure, etc. The MS Excel^®-based tools provided by the DACS in the Tools and Methods section of this document show how high-level risk areas can be further broken down to quantify and prioritize very specific elements of risk.

Another technique that can be used to help prioritize risk is the calculation of risk exposure. Risk exposure is defined as:

(Probability of an unsatisfactory exposure) * (Loss if unsatisfactory outcome)

An example of a risk exposure calculation is presented in Figure 15, where risk is being prioritized based on a decision as to whether software regression testing should be performed or not.

Figure 15: Example of a Risk Exposure Calculation [STN, 1998]

The combined risk exposure number associated with performing regression testing ($1.975M) indicates that it should be considered a much lower priority than if regression testing is not going to be performed at all ($16.725M) based on the quantitative probability (P) and loss (L) associated with each potential undesired outcome (UO).

McManus identifies numerous techniques for performing both qualitative and quantitative analyses of risk exposure [McManus, 2004], including brainstorming, checklists, questionnaires and peer interviews in the qualitative category, and symbolic models, probability analysis, consequence analysis, decision trees, Monte Carlo analysis and cost-benefit analysis in the quantitative category. Of particular interest in the qualitative analysis of risk exposure is SWOT (Strengths, Weaknesses, Opportunities and Threats) Analysis, which can be used to analyze and prioritize threats and resultant risks within a project. SWOT Analysis supports the concept of positive risk discussed earlier in this paper, and is exemplified by some of the potential opportunities and threats presented in Table 7.

Table 7: Example of SWOT – Opportunities and Threats [McManus, 2004]

Potential Opportunities

Potential Threats

· New hardware technology

· New methods of analysis

· New development tools

· New testing tools

· Performance issues

· Competency issues

· Learning curves/productivity

· Performance issues

RISK MANAGEMENT PLANNING

Risk management planning provides the basis for the identification of the monitoring procedures that should be put in place for each risk, including how to tell if a risk is going to manifest as a real problem, and how frequently each identified risk should be monitored. Risk planning also takes into account risk aversion planning (i.e., what actions will be taken to mitigate risk before it occurs) and contingency planning (i.e., how to react if a risk actually manifests).

IEEE Standard 1540-2001 [IEEE, 2001] was introduced earlier in this document. It contains a number of templates that can be used in support of formal risk management planning, including

· Risk Management Plan Outline (Annex A)

· Risk Action Request Outline (Annex B)

· Risk Treatment Plan Outline (Annex C)

The purpose of a Risk Management Plan is to define how risk management activities are implemented and supported during a project. It is a key output of the planning process, and serves as the mechanism for implementing software risk management.

The Risk Action Request serves as a mechanism by which risk information can be captured and communicated to the stakeholders. An effective risk management process requires the creation of risk action requests for any risks that exceed their quantified risk threshold value.

Finally, the Risk Treatment Plan is used to define how risks that are found to be unacceptable are to be treated. It serves as the mechanism for implementing a selected recommended alternative defined within a risk action request. Note that “treatment” and “mitigation” are considered synonymous.

An outline that highlights the minimum recommended requirements for each of these three software risk documents is shown in Figure 16. The details can be found within the IEEE Standard.

Figure 16: Outlines of IEEE Std 1540-2001 Software Risk Documentation [IEEE, 2001]

The document “Risk Management Survey of Department of the Navy Programs” [Navy, 1997] contains samples of Risk Management SOW Contract Clauses (Section A); a Risk Status Report format (Section B - illustrated); a product/process combination risk management process (Section B); a technical risk assessment form (Section B); a technical risk status form (Section B); a risk management process and analysis methodology (Section B); an event-driven risk mitigation report (Section C); and risk management survey references (Section D).

RISK RESOLUTION

The “Software Acquisition Risk Management Guidebook” [Gallagher, 1997] contains a software risk planning decision flowchart (Figure 17) that, among other things, highlights the major options to be considered when developing response plans for threats, namely “Keep”, “Delegate” and “Transfer” if personal risk responsibility is appropriate; and “Research”, “Accept”, “Mitigate” and “Watch” (or “Monitor”) if further involvement in risk resolution is warranted.

Figure 17: Planning Risk Decision Flowchart [Gallagher, 1997]

Odegard and Sifri reinforce these risk decision elements by defining the options that need to be considered as part of a risk resolution strategy [Odegard, 2002] [Sifri, 2003]. They include:

§ Research: establishing a plan to research the threats

§ Acceptance: deciding to “accept” the threat(s) and documenting the rationale behind the decision

§ Avoidance: threats can be avoided by, for example, altering the original plan concept or changing contractors

§ Mitigation or Reduction: allocating resources and assigning actions in order to reduce the probability and/or potential impact of threats after the risk happens

§ Protection: allocating resources and assigning actions in order to reduce the probability and/or potential impact of threats before the risk happens

§ Reserves: Using previously allocated schedule or budget slack

§ Transfer: threats can be transferred partly or fully from the customer to the contractor or vice-versa, using contractual incentives, warranties or penalties attached to project performance, cost or schedule measures

There are a number of key questions that can help facilitate the effectiveness of risk allocation and risk transfer approaches. These are highlighted in Table 8 and supplement the planning risk decision flowchart just presented.

Table 8: Key Questions for Risk Allocation and Risk Transfer [OGC, 2002]

Do we understand the risks?
	· Have all of the key risks relating to this project or operational service been identified? · Has a thorough assessment been made of each risk (probability of occurrence, its likely impact and cost)? · Are the interdependencies between risks understood? · How do these risks affect the key objectives? · Has a long-term view been taken to identify possible future risks? · What is the overall risk exposure?
What can be done about the risks before deciding how to allocate them?
	· Has the best way to deal with each risk been considered (i.e., minimize, mitigate, accept, transfer or build in contingencies)? · Should other steps be taken, i.e., improving quality assurance processes?
What are the options for allocating risk?
	· Which risks should be managed internally, and why? o Because they can be controlled better internally? o Because it is not cost-effective to allocate them to others? o Because their likely impact will not affect critical objectives? · Which risks should be managed externally, and why? o Because external resources are better placed to influence the outcome? o Because cost-effective payment incentives can be identified that will deliver value for money? o Because the cost is affordable and reflects the ability/willingness of external sources to control risk?
How can risk transfer be negotiated with suppliers?
	· Can complete risk transfer be obtained, or an optimum balance achieved between the benefits of transferring a risk and the cost of compensating a supplier for taking it on? · Is it necessary to obtain multiple alternatives in order to decide on an optimum offer? · Have negotiations been held with each supplier to achieve the optimum balance of risk, costs and benefits? · Are risk allocation decisions based on a realistic assessment of the way in which risks will be managed? · Does the entire supply chain have a shared understanding of the risks and consequences if the risks materialize? · Have risk plans been validated by obtaining proposals and prices from suppliers, assessing risk and its price, and taking into account: o The nature of the requirement (high or low risk)? o The expected length of the contract (long- or –short-term potential for recovering the development costs)? o The likelihood of predicted service volumes being exceeded, with the opportunities for increased revenue?
Have risks been allocated to the correct parties in the supply chain?
	· Is there certainty that the wrong risks have not been transferred, leading to poor value for the money and unacceptable risk exposure? · Is there certainty that only risks that are commercial in nature have been transferred, where the suppliers can influence the outcome? · Where risks have been transferred, are the suppliers genuinely in a position to manage them?
Can “taking back” of transferred risks be avoided?
	· Is there a danger that transferred risks could be “taken back”, i.e., getting too involved in suppliers’ business and the solutions they provide, preventing them from managing the risks that they have agreed to take on? · Is there certainty that risks have not been taken back by: o Attempting to define a technical solution for suppliers? o Attempting to define how a service should be provided? · Has suppliers’ freedom to propose alternatives been preserved? · Will suppliers have the freedom to choose how to handle and minimize risks?

RISK MONITORING

There are a number of measurements/metrics and tools that can be used to monitor and report status of project risks. Table 9 presents some of the basic measurements and metrics that can be considered to meet these needs.

Table 9: Important Risk Measurements/Metrics

Risk Measurements/Metrics

· Number of Identified Risks

· Number of Active Risks

· Number of Risks Assessed “High”, “Medium” and “Low”

· Total “Probability X Impact” (PI) Score for All Risks

· Average PI Score

· Expected Value (= Probability x Cost Impact)

· Percent Likelihood of Meeting Target Schedule and Budget

· Overall Project Criticality Index

· Overall Project Cruciality Index

· Relative Risk Exposure Index

· Risk Reduction Leverage (= Reduction in Cost Impact divided by Cost of Response)

The number of identified and active risks provides a picture of how many risks have been mitigated, and how many remain as open items. Prioritized risks can be categorized as the number considered “high”, “medium” and “low”. Combining risk probability and impact into a total composite score is another way to identify, track and monitor high priority risks. Identifying an “average” probability impact (PI) score subdivides the risk population into “greater than” and “less than” average risks, providing a very general guideline on which risks to address first. Another way to look at impact is risk probability times cost impact, rather than consequence (or severity) impact, in order to calculate the expected value of each potential risk. A useful metric for assessing project risk is the percent likelihood of meeting the target schedule and/or budget. Any significant negative deviation from this metric would not bode well for the success of the project.

There are several indexes that can be used to measure risk, including project criticality, project cruciality, and risk exposure. The Criticality Index identifies which resources (or resource types) are most constrained and how those constraints may impact the project schedule. Activities with a high criticality index are likely to determine the overall project duration. The Cruciality Index, which is similar to the Criticality Index, factors in how critical the resources are to meeting the project schedule constraints. Activities with a high cruciality index have a direct bearing on the variability of the overall schedule duration. The relative Risk Exposure Index, which we discussed earlier, is the probability of unexpected loss (or risk) and the size of the loss. Finally, the Risk Reduction Leverage refers to the corresponding reduction in cost impact or cost of response as a result of reducing the risk level in other areas, such as performance, reliability, schedule, etc. Its value is determined using the equation:

Risk Reduction Leverage = (Risk Exposure Before – Risk Exposure After) divided by Risk Reduction Cost

There are also a number of risk management audit measurements/metrics that are appropriate to monitor and track progress on controlling project risk and relate well to SEI CMM concepts. These include factors such as (1) the number and ratio of scheduled and actual risk management audits, (2) the effort required to support risk management audits, (3) the total number of risk-related problem reports (and the number of open and closed reports) that are measured in each risk management audit, and (4) the number and ratio of actual and deferred corrections that are reported in each risk management audit. This number of corrections can be further broken down into categories of major and minor corrections and, as with the number of actual corrections, can be assessed against the amount of effort required to implement them.

Some general comments about risk monitoring and tracking:

· The frequency with which risks are measured should be dependent on the risk priority (higher priority means measure more often)

· At a minimum, risk measurements should be made/communicated at all key project milestones

· Resist the temptation to perform measurements too precisely or too frequently

· Track all actions to closure (and with demonstrated verification, if possible)

One of the better tools to use in the risk management process is Risk Status Reports. They represent a best practice in that they allow the communication of appropriate risk metrics to all stakeholders on the project. Risk Status Reports can include such things as a listing of the top ten risk items, the number of risks resolved as of the report date, the number of new risk items introduced since the last report, the total number of current unresolved risk items, the presence of unresolved risk items that lie on the project critical path, and assessment of the probable cost of unresolved risk in comparison to the amount of risk reserve. Figure 18 provides an example of how the Navy has used a Risk Status Report architecture to convey the level of risk associated with various elements of a product development process.

Figure 18: Sample Risk Status Report Format [Navy, 1997]

A second useful tool for monitoring risk and reporting risk status is the Event-Driven Management Report. As described by McManus [McManus, 2004], this tool can be represented by a generic waterfall chart, where risks are identified and documented on an Integrated Product Team (IPT) watchlist. The probability and severity of individual risks are assigned and appropriate Risk Mitigation Plans are established and tracked. An example of this type of report is given in Figure 19. One general caution or observation regarding the Event-Driven Management Report, however. Be aware that risks may not always follow the waterfall, i.e., high- or medium-level risks could conceivable pop up anywhere in the life cycle, depending on how effective the overall risk management process is. Event-driven risks will not necessarily conveniently decrease in risk level as the life cycle evolves.

Figure 19: Sample Event-Driven Risk Management Report [Navy, 1997]

CHARACTERISTICS OF IMPLEMENTATION:

SUMMARY CHARACTERISTICS

NOT AVAILABLE

ANTICIPATED BENEFITS OF IMPLEMENTATION: