|
SOFTWARE ACQUISITION GOLD PRACTICE ™ ACQUISITION PROCESS IMPROVEMENT FOCUS AREA: PROCESS Definition and Summary: The analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies, including specific process-oriented tools, methods, models, etc. (such as the SEI SA-CMM ® and CMMI-AM®) to aid in process improvement.
“ Organizations that have a strong, consistent evolutionary environment and practices for setting product requirements, maintaining a disciplined development process and using metrics to oversee development progress achieve favorable cost, schedule and quality outcomes .” This conclusion [GAO, 2004] should serve as the rallying point for any and all acquisition organizations who are struggling to improve the results from their software acquisition processes. “Decisions to begin programs and make significant investments are made [by DoD] without matching requirements to available resources and without demanding sufficient knowledge at key points.” Taken from the same report, this statement provides the realization that there is still a long way to go and that, unfortunately, some may not have yet started on the journey.
Successful implementation of software acquisition process improvement can result in: § Realistic, incremental and continuous improvements to software acquisitions § Adherence to well-understood, well-defined, manageable requirements § Following of disciplined, structured software management and development processes that provide predictable results § Well-informed, well-disciplined and knowledgeable software acquisition teams § Increased probability of software acquisition successes and “best value” awards
DESCRIPTION OF THE PRACTICE:
Acquisition process improvement has been defined as the analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies [Eslinger, 1999]. The definition provided in Turner’s dissertation [Turner, 2002] incorporates the concept of specific process-oriented tools (such as the SEI CMM in 2002, and expanded upon by the SA-CMM® in 2003 and CMMI-AM® in 2004) to aid in the acquisition improvement process. The Navy [Pracchia, 2004] uses, perhaps, the most comprehensive definition of software acquisition process improvement, defining it as: • A documented process for software acquisition planning, requirements development/management, project management/oversight and risk management • Efforts to develop appropriate metrics for performance measurement and continual process improvement • A process to ensure that key program personnel have an appropriate level of experience/training in software acquisition • A process to ensure that each military department and defense agency select, implement and adhere to established processes and requirements relating to software acquisition One of the better descriptions of the characteristics of a software acquisition process that is successfully being used by leading software acquirers/developers is contained in the “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” report from the Government Accountability Office (GAO – formerly the General Accounting Office) [GAO, 2004]. Three fundamental management strategies are recommended within this report: · Evolutionary Environment · Disciplined Development Processes · Collecting/Analyzing Meaningful Metrics These strategies limit software development efforts to what can be managed and result in decisions based on sufficient systems engineering knowledge to adequately assess risks. In addition, business decisions are consciously made to invest time/resources in achieving high software process maturity levels. Strong upper-level management support is a strong enabler for ensuring that these combined strategies are successful.
“Organizations that have a strong, consistent evolutionary environment and practices for setting product requirements, maintaining a disciplined development process and using metrics to oversee development progress achieve favorable cost, schedule and quality outcomes.” This conclusion, taken from a 2004 report entitled “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” [GAO, 2004], should serve as the rallying point for any and all acquisition organizations who are struggling to improve the results from their software acquisition processes. “Decisions to begin programs and make significant investments are made [by DoD] without matching requirements to available resources and without demanding sufficient knowledge at key points.” Taken from the same report, this statement provides the realization that there is still a long way to go and that, unfortunately, some may not have yet started on the journey. The “Acquisition Process Improvement” Gold Practice begins with some observations of what constitutes a software acquisition process, then looks at some of the fundamental management strategies that have been defined as successful in the software acquisition process and assesses their characteristics and potential impact on the process. In the context of this Gold Practice document, the following definitions are used: Acquisition The conceptualization, initiation, design, development, test, contracting, production, deployment, Logistics Support (LS), modification, and disposal of software systems, supplies or services to satisfy customer needs. Acquisition environment Internal and external factors that impact on, and help shape, every software acquisition program. These factors include political forces, policies, regulations, reactions to unanticipated requirements and emergencies. Acquisition life-cycle The life of a software acquisition program consists of phases, each preceded by a milestone or other decision point, during which a system goes through Research, Development, Test and Evaluation (RDT&E) and production.
Acquisition program A directed, funded effort over the software acquisition life-cycle that provides a new, improved, or continuing materiel, weapon or information system or service capability in response to an approved need. Acquisition process The steps involved, and practices employed, in the planning, contracting, implementation, acceptance and follow-on activities of a software acquisition. The acquirer will bear most of the responsibility for the results from the planning, contracting, and follow-on phases of the acquisition, while the developer is most accountable for the results coming out of the implementation and acceptance phases of the acquisition. Development process The process of working out and extending the theoretical, practical, and useful applications of a basic design, idea, or scientific discovery, characterized by the design, coding, modification or improvement of software.
The Software Acquisition Process It is necessary to first understand what a software acquisition process is before steps can be taken to improve it. IEEE Std 1062, 1998 Edition, entitled “Software Acquisition Process Steps” [IEEE, 1998] describes a generic nine-step process for software acquisition, summarized in Figure 1.
Figure 1: IEEE Std 1062, 1998 Edition Software Acquisition Process Steps [IEEE, 1998]
There are three steps that comprise the planning phase of a software acquisition. During this phase, the acquisition strategy should be planned based on a review of the acquirer’s objectives. A formal Acquisition Strategy document is developed to guide the acquisition. Implementing the acquisition strategy into the organization’s process is the next step. Appropriate contracting practices need to be included in the process. The final step of the planning process encompasses the determination of software requirements. In addition to defining the software being acquired, this step includes the preparation of the quality and maintenance plans for accepting the supplier’s/developer’s software. The release of a Request for Proposal (RFP) is the completion milestone for the planning phase of the software acquisition life-cycle. The contracting phase of the software acquisition is also comprised of three steps. The first step is the identification of potential suppliers. Suppliers should be selected who will provide documentation for their software, demonstrate its performance, and submit formal proposals. Failure to perform any of these actions may serve as a basis for rejecting a potential supplier. Past performance data from previous contracts should be reviewed for all potential suppliers. The next step is to actually prepare the contract requirements. The quality of work (acceptable performance and acceptance criteria) should be described, and contract provisions that tie payments to deliverables should be prepared. Legal counsel should also be sought to review and approve contractual language/requirements. Finally, proposals will be evaluated and a qualified supplier(s) will be selected. If appropriate, an alternate supplier should be negotiated with, primarily as a risk reduction measure. Supplier selection completes the contracting phase of the software acquisition life-cycle, which started with the release of the RFP and ends with the contract signing. Once the contract is signed, software development can begin and the product implementation phase of the software acquisition life-cycle is initiated. During this phase, the acquirer must manage supplier performance. The supplier’s progress is monitored to ensure all milestones are met. The acquirer also must approve all appropriate work products. Note that, during this phase, the acquirer has the responsibility to provide all acquirer deliverables to the supplier when required. The product implementation phase of the software acquisition is completed when the software product is delivered to the acquirer (pre-approval). The next phase of the software acquisition life-cycle is product acceptance. Adequate testing needs to be performed, and a process for certifying that all discrepancies have been corrected and all acceptance criteria have been satisfied must be established. Acceptance of the software product by the acquirer based on mutually agreed-to pre-defined criteria signals the end of the product acceptance phase. The follow-on phase of software acquisition covers the period between software acceptance and software retirement, when it is no longer in use. A follow-up of the software acquisition contract should typically be performed to evaluate contracting practices, record lessons learned, and evaluate user satisfaction with the software. During this final phase of the software acquisition life-cycle, the acquirer should record and retain supplier performance data to use for the acquisition of future software products. More recently, the Software Engineering Institute (SEI) has released the Software Acquisition Capability Maturity Model (SA-CMM®) [SA-CMM®, 2002] to represent the important phases and elements of the software acquisition processes. Table 1 highlights the correlation between the five major IEEE Std 1062 software acquisition process steps discussed above [IEEE, 1998] and the Level 2 Key Process Areas (KPAs) of the SA-CMM®. The Level 3 through 5 KPAs are equally important for specific acquisition phases, or for all acquisition phases. Knowledge of the basic concepts behind the SEI CMM modules is assumed. The relationship of the SA-CMM® to the more recently released CMMI-AM® process [CMMI-AM®, 2004], which imbeds the concepts of Integrated Product and Process Development (IPPD) into the software acquisition process, thereby emphasizing the concept of acquisition process improvement, will be discussed later. Note that the IEEE Std 1062, SA-CMM® and the CMMI-AM® processes all focus on what should be done, rather than how it should be done, representing one of the biggest challenges being faced by software acquirers in achieving software acquisition process improvement. Table 1: SA-CMM® Key Process Areas
What is Software Acquisition Process Improvement? Acquisition process improvement has been defined [Eslinger, 1999] as the analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies. The definition provided in Turner’s dissertation [Turner, 2002] incorporates the concept of specific process-oriented tools (such as the SEI CMM) to aid in the acquisition improvement process. The Navy [Pracchia, 2004] uses, perhaps, the most comprehensive definition of software acquisition process improvement, defining it as:
• A documented process for software acquisition planning, requirements development/management, project management/oversight and risk management • Efforts to develop appropriate metrics for performance measurement and continual process improvement • A process to ensure that key program personnel have an appropriate level of experience/training in software acquisition • A process to ensure that each military department and defense agency select, implement and adhere to established processes and requirements relating to software acquisition
How is DoD Software Acquisition Process Improvement Being Addressed? In a presentation on Software Acquisition Process Improvement, Borden [Borden, 2004] addresses nine target areas for improvement which pretty much cover the entire acquisition/development life-cycle: · Acquisition planning · Requirements development/management · Configuration management · Risk management · Project management/oversight · Test & evaluation · Integrated team management · Solicitation/source selection · Performance measurement Dr. Barry Boehm [Boehm, 2004] has described a number of general software acquisition goals (defined as critical success factors) that should be addressed to support software acquisition process improvement: · System/software objectives and constraints are adequately defined/validated · System/software acquisition strategies are appropriate/compatible · Success-critical stakeholders have committed adequate software capability/resources to perform their software-related tasks · Software product/process plans are feasible/compatible · Feasibility Rationale provides convincing evidence that software progress is satisfactory There are a number of other published reports that contain more specific recommendations on improvements that can be made to the DoD software acquisition process. These improvements include (1) documenting agreements between the software developer and acquirer that contain baseline requirements based on systems-engineering knowledge, (2) gated reviews during the software development process, (3) obtaining meaningful metrics from the developer for managing the program, (4) attaining knowledge about the technology that is planned early in the development process, (5) ensuring an appropriate balance between requirements and available resources, (6) ensuring that the software design is mature before it is released, and (7) having all processes under control prior to release. In general, the more that acquisition managers know about software development processes and metrics, the better equipped they are to improve the processes for software acquisition. The Navy is also in the process of developing a formal Software Acquisition Process Improvement Plan (SAPIP) [Pracchia, 2004]. The burden is placed on ASD/C3I and OUSD(AT&L) (analogous to “senior management”) to (1) provide applicable improvement program administration and compliance guidance and ensure that secretaries of the departments and selected agencies comply with that guidance and (2) assist departments/agencies with their respective improvement programs by ensuring the use of applicable source-selection criteria and access to information regarding software acquisition/development best practices in both the public and private sectors. One of the better descriptions of a software acquisition process that is successfully being used by leading software acquirers/developers is contained in the “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” report from the GAO [GAO, 2004]. Three fundamental management strategies are recommended within this report: · Evolutionary Environment · Disciplined Development Processes · Collecting/Analyzing Meaningful Metrics These strategies limit software development efforts to what can be managed, and they result in informed decisions based on sufficient systems engineering knowledge such that risk can be adequately assessed. In addition, business decisions are consciously made to invest sufficient time and resources in order to achieve high software process maturity levels (high quality, on-time, within budget). Committed upper-level management support is an effective enabler for ensuring that these combined strategies are successful. The DoD has, in many software acquisitions, achieved poor results due to inconsistent practices in these areas, as decisions to begin programs and make significant investments are made without ensuring that adequate resources are available to meet requirements, and without demanding critical knowledge at key decision points. While the DoD has made attempts to improve software acquisition processes by establishing a framework for an evolutionary environment in its requirements generation and acquisition policies, the GAO report concludes that the DoD policies do not contain the controls needed to ensure that individual programs will adhere to disciplined requirements engineering and management processes, nor do they include the necessary metrics to do so. The report also observed that the Services’ and Milestone Decision Authority (MDA) planning documentation did not include practices proven successful in leading commercial firms. Suggestions for strengthening these plans included specific criteria to ensure that: · Requirements baselines based on systems engineering are documented/agreed to by both the acquirer/developer before a program is initiated · Cost/benefits analyses are required when new requirements are proposed · Software acquirers/developers make efforts to continually improve practices over time · Gated reviews and deliverables are integrated into development processes · Requiring software developers to collect/analyze metrics (including earned value) to obtain knowledge of progress and to manage risk While there is recognition that care must be taken to not make these plans too prescriptive, thereby stifling software developer innovation, it is necessary to instill some level of control to ensure future consistent success for DoD software acquisitions.
Evolutionary Environment: An evolutionary software acquisition environment is characterized by incremental improvements to software performance, as opposed to succumbing to programmatic pressures to set unrealistic expectations, in order to improve software processes on a continual basis. An evolutionary environment limits software development to what is possible to manage by adhering to well-understood, well-defined, manageable requirements while simultaneously encouraging continuous process improvement. An evolutionary product development is one of the fundamental elements of a manageable acquisition environment. The concept of continuous improvement is a critical part of the evolutionary environment, and must be supported by both the environment and the organizational culture (including senior management) in order to be successful. Ad hoc processes make it impossible to understand exactly when and how defects in the process occur. DoD Instruction 5000.2, dated 13 May 2003 [DoDI 5000.2, 2003], Section 3.3 formally defines the concept of evolutionary acquisition for DoD acquisition. It further breaks down this definition into two basic processes: Spiral Development and Incremental Development. For Spiral Development, a desired capability is identified, but the end-state requirements are not known at program initiation. Requirements are refined through demonstrations and risk management, with continuous user feedback. Each spiral provides the user with the best possible capability at that point in time. Requirements for future increments depend on feedback from users and technology maturation. For Incremental Development, a desired capability is identified, end-state requirements are known, and the requirement is met over time by developing several increments, each dependent on available mature technology. The DoD Requirements and Acquisition Process Depiction from DoDI 5000.2 is represented in Figure 2, where each of the three software development increments represents a spiral, and each spiral has demonstration, Milestone B and Milestone C requirements. Figure 3 shows a notional comparison between the “traditional” approach to acquisition and the evolutionary acquisition process.
Figure 2. DoDI 5000.2 (12 May 2003) Requirements and Acquisition Process Depiction (Refer to [DoDI 5000.2, 2003])
Note that percentages and increments are notional, not factual Figure 3: Traditional vs. Evolutionary Acquisition Strategies [AFPD63-1, 2003]
The Air Force Agile Acquisition initiative [Sambur, 2003], Air Force Policy Directive AFPD 63-1 [AFPD 63-1, 2003] and Interim Guidance to AFI 63-101 [AFI 63-101, 2003] on a Capability-Based Acquisition System, provide additional emphasis on the evolutionary acquisition process. Particular focus is placed on collaborative processes covering requirements development, seamless verification (i.e., capabilities-based test and evaluation), and technology transition processes. Further discussion on the Air Force Agile Acquisition Initiative can be found at their Acquisition Center of Excellence website, located at http://www.safaq.hq.af.mil/ACE/.
Disciplined Development Processes: Developers must follow disciplined, structured management and development processes in order for the software acquisition process to improve. Each phase of the software acquisition should end in a management review/gate to ensure that the project is on track. To pass these gated reviews, developers must demonstrate that acquirer’s expectations and quality standards are met before proceeding to the next phase. Peer reviews to check each other’s work and remove defects, beginning at the earliest stages of software development, are necessary to prevent costly rework and schedule delays resulting from downstream defect detection. Disciplined software development processes that are characterized by their reliance on configuration management, peer reviews and quality assurance significantly increase the probability of a successful acquisition, and have been proven to assist leading organizations in identifying opportunities for software process improvement. As a result, areas of risk can be identified early and actions can be taken to control them. Figure 4 highlights a disciplined knowledge-based software development process that can serve as a basis for software acquisition process improvement, describing the information needed, the relevant activities and the deliverables for each phase of the process. Figure 4: Highlights of the Knowledge-Based Software Development Process [GAO, 2004]
Requirements need to be managed and controlled before design work begins and all lower-level design elements should be adequately defined before the start of coding. It is critical that the software acquirer and developer work closely together and have open and honest discussions. Requirements management should be viewed as one of the most critical development tasks for ensuring a successful acquisition. Requirements should be sufficiently documented and validated prior to preliminary design. In addition, acquirer/developer stakeholders must use sound systems engineering techniques to establish the software requirements, and aggressively manage and control requirements changes. An Integrated Product Team (IPT) comprised of both acquirers and developers needs to categorize the comprehensive list of requirements as to how critical they are in meeting performance objectives. Ultimately, the list of requirements will need to be negotiated, i.e., which requirements should be delayed or deleted on the basis of resource or schedule goals. Once negotiated, the acquirer and developer must agree on a requirements baseline that details cost, schedule, performance and quality goals that are expected to be achieved. The software development team then develops, allocates and documents lower-level requirements using systems engineering knowledge and based on discussions with the acquirer. Finally, a requirements check is performed on all requirements-related documents. A structured process for establishing a stable design ensures that all requirements are addressed, and that all software components and interfaces are defined. Management and design reviews are important steps in this process. A Preliminary Design Review (PDR) examines design rationale/assumptions to ensure that stated requirements are met. A Critical Design Review (CDR) examines all design features to ensure that acquirer requirements are met. Peer reviews are used to detect errors in the software. The construction of prototypes for the acquirer provides a “hands-on” assessment of progress towards meeting requirements. To enhance software coding and testing quality, requirements should be well-written and achievable. By this phase of the acquisition, designs should be sufficiently detailed. Other processes that are critical to achieving high quality software include peer reviews, documented coding standards, frequent unit testing, access to a reuse library, and the use of program languages that enable code documentation to facilitate future understanding. Test plans should not be developed until after requirements are considered stable, and there should be a minimum of one test to verify/validate each requirement. One of the most widely-known and best-structured software acquisition and development processes is the SEI’s CMMI® series of processes. A very recent addition to this suite of tools is the CMMI Acquisition Module (CMMI-AM®) [CMMI-AM, 2004]. The focus of the CMMI is on “what” should be done, not “how” it should be done, so it allows a great deal of latitude in terms of implementation of the CMMI Key Process Areas. As with the other CMMI documents, a primary emphasis of the Acquisition Module is on Integrated Product and Process Development (IPPD) concepts. These include the effective use of cross-functional or multidisciplinary teams; leadership commitment (particularly of senior management); appropriate allocation/delegation of decision-making authority; definition of organizational structures that reward team performance; the design of downstream processes (transition to Operations and Support – O&S) during the acquisition; a strong focus on customer needs throughout the software life-cycle; timely and effective collaboration between all relevant stakeholders; the continuous and proactive identification and management of risk; and a focus on the measurement and improvement of processes to develop/deliver a software product. A graphical representation of the CMMI-AM® process is shown in Figure 5. Following that figure is Table 2, which includes checklist-type questions that can be used to self-assess coverage of each defined CMMI-AM® process area. The reader is referred to the CMMI-AM® document to get a more in-depth discussion of each of the process areas defined in Figure 5 and Table 2.
Figure 5: Graphical Representation of the CMMI-AM® Process
Table 2: CMMI®-AM Acquisition Process Areas [CMMI-AM, 2004]
The use of checklists such as the one above helps maintain discipline and structure in the software acquisition process, and can cover any phase of a software acquisition life-cycle. Boehm’s presentation “Early Warning Indicators in the Acquisition of Software-Intensive Systems” [Boehm, 2004] includes a limited checklist for assessing detailed software cost and schedule estimates, as well as an acquisition strategy checklist. More detailed and comprehensive checklists and documentation templates are found in standards, handbooks, etc.. For example, IEEE Standard 1062 – 1998 Edition [IEEE, 1998] contains a template for an Acquisition Plan (AP). It also contains separate, extensive checklists that govern twelve different acquisition categories ranging from Organizational Strategy through Software Acceptance (see Table 3). These are the same checklists that are referred to back in Figure 1, which covered the 9 major steps of the IEEE Standard 1062 software acquisition process. The IEEE Standard also makes reference to IEEE /EIA 12207.0-1996 and ISO/IEC 12207 which contain an Evidence Product Checklist for identifying the Procedures, Plans, Records, Documents and Audits/Reviews that should be expected at each step of the software acquisition. Table 4 provides a summary of these work products.
Table 3: IEEE Std 1062, 1998 Edition Software Acquisition Checklists [IEEE, 1998]
Table 4: Evidence Product Checklist for ISO/IEC Standard 12207 Software Life-cycle Processes (Acquisition Phase Only) [ISO 12207, 2002]
Finally, the use of checklists can benefit specific monitoring and oversight activities of software acquisition such as risk management. Gallagher [Gallagher, 1997] defines a software acquisition risk management Key Process Area (KPA) that can be put into a high-level checklist form (see Table 5) in order to assess whether Goals, Activities, Commitments, etc., associated with risk management have been covered in the software acquisition process. Table 5. Software Acquisition Risk Management KPA [Gallagher, 1997]
Collecting/Analyzing Meaningful Metrics: Meaningful metrics are collected and analyzed by both software acquirers and developers in order to track progress, confirm knowledge, manage risk, improve processes and ensure that all stakeholders are well-informed. Meaningful and suitably tailored metrics should cover cost, schedule, software size, requirements, tests performed/completed, number of defects detected/corrected, and quality attributes. Acquirers can apply some of these same metrics to assess whether the developer will be able to deliver software within cost, schedule, performance and quality constraints. The use of earned value analysis tools to track cost/schedule and help mitigate risk is an important driver in determining meaningful metrics, supporting the ability of a program to maintain consistent, predictable practices and acceptable process outcomes. Sharing of metrics between developers and acquirers is an important aspect of a successful software acquisition process. Leading software developers have been described as relentless in their efforts to collect metrics to improve project processes and results [GAO, 2004]. They typically also establish goals and track metrics for company-wide initiatives, such as cost reduction efforts and customer satisfaction. These leaders have continually emphasized the critical nature of measuring processes, collecting metrics and using them to improve performance in their workforce through training. In addition, a good part of their success can be attributed to their use of an Earned Value Management System (EVMS) and a project time-tracking system (to record time spent on “cost of quality” and “cost of poor quality”). The cost of poor quality is a direct reflection of the relative level of effectiveness of an organization’s software acquisition and/or development processes. Typical metrics used by leading software developers are indicated in Table 6.
Table 6: Metrics Used by Leading Software Developers [GAO, 2004]
Some Suggested Software Acquisition “Best” Practices? Although not yet defined in the context of “Gold” practices, there are a number of software acquisition “best” practices that can be considered as an integral part of the three fundamental software acquisition strategies identified earlier. Eslinger and Adams [Adams, 2004a], both from the Aerospace Corporation, have suggested some candidates for consideration. Table 7 summarizes the results from the [Adams, 2004a] reference. Note the implied emphasis on systems engineering knowledge. Table 7: Recommended Software Acquisition Best Practices [Adams, 2004a]
The second reference [Adams, 2004b] defines lessons learned and best practices for the acquisition of COTS-based software systems used in space systems applications. They are generally applicable, however, to all software-intensive systems, regardless of application domain. The suggested best practices are summarized in Table 8. The lessons learned are paraphrased below: · Critical aspects of COTS-based software development/sustainment are out of the control of the customer, developer and user (suppliers are market driven and DoD is not the market; supplier strategies/market position are subject to change) · Full application of system/software engineering is required throughout the COTS-based system/software life-cycle · COTS-based software system development/sustainment requires a close, active and continuous partnership among the customer, developer and user · Every COTS-based software system requires continuous evolution through development/sustainment · Current processes must typically be adapted for COTS-based software system acquisition, development and sustainment · Actual cost/schedule savings associated with COTS-base software systems development/sustainment are typically overstated
Table 8: Recommended COTS-Based Software System Acquisition
Selected Software Acquisition Process Improvement Techniques There are literally dozens of tools and resources that are available to support software acquisition process activities. Many of these are described and linked in the Resources section of this document and relate to tools, methods, etc., that are generally familiar to (and used by) the software acquisition community. These include Earned Value Management Systems, Risk/Configuration Management tools, acquisition document templates, and so on. There are three techniques that will be briefly mentioned here, however, that are not typically associated with or recognized by the software acquisition community or supported by the literature in the context of software acquisition. They may potentially provide some future benefit (albeit with some risk) for improving the software acquisition process for those that wish to research and adapt them as a software acquisition process improvement tool. They are: · Quality Function Deployment (QFD) · Analytic Hierarchy Process (AHP) · Phase Containment Matrix Each technique will be briefly introduced and discussed in the context of its potential role in software acquisition process improvement, but a detailed technical explanation of how each technique works is beyond the scope of this document. The reader is encouraged to visit the sites indicated in the Resources section to gain further insight into these methods.
Quality Function Deployment: Quality Function Deployment (QFD) is a technique that is most closely associated with the Total Quality Management concepts of the last 20 years. QFD is typically structured around a series of interdependent matrices, each of which is commonly referred to as a “House of Quality”. For each stage of the design and production process, engineers identify technical performance measures that will be used to meet itemized customer needs/expectations. They then specify corresponding threshold values to be met in order to achieve overall system performance requirements. The QFD process, therefore, establishes the minimum levels of performance required to satisfy customer needs. [Wollover, 1997] QFD has been successfully demonstrated in a variety of industries as a means to guide process development using technical performance measures to systematically organize and prioritize all independent variables (cost, etc.) and their interrelationships. Traditional QFD consists of six general steps: (1) identify and analyze customer needs and requirements (WHATs), (2) identify technical requirements (HOWs), (3) benchmark the technical requirements (WHATs vs. HOWs), (4) assign rank/priority to customer requirements (WHYs), (5) develop the corresponding House of Quality (6) establish technical requirement metrics that inter/intra-relate to specific customer requirements (HOW MUCHs), and (7) evolve technical performance measures into the next-phase design requirements (define the next House of Quality, with “Technical Requirements” = WHATs and “Design Requirements” = HOWs). Figure 6 is a generic representation of one iteration through this process.
Figure 6: Traditional Quality Function Deployment to Translate Customer Requirements Into What is suggested in this Gold Practice document is a complementary QFD process that, beginning with the same list of customer requirements, maps the software acquisition process to the development of an acquisition strategy/plan, the feasibility of implementing that strategy based on available resources, and the benefits/risks associated with the acquisition based on those resources and a corresponding set of available support tools/techniques. This approach, graphically represented by Figure 7, would potentially define an optimal level of software acquisition support required to meet customer requirements.
Figure 7: Use Quality Function Deployment to Translate Customer Requirements Into a Successful Acquisition
Analytic Hierarchy Process: The Analytic Hierarchy Process (AHP) is a technique to combine information which can be used to prioritize customer needs in QFD, make budget decisions based on a number of explicit or implicit strategic goals, manage conflicting stakeholders, or select from a large number of potential alternatives. The AHP is predicated on the decomposition of multiple-criteria decision-making problems into a mathematically-based criteria hierarchy. At each level in the hierarchy, the relative importance of factors is assessed by comparing them in pairs. Finally, the pair-wise alternatives are compared with respect to the defined criteria [Kontio, 1996]. The technique has been successfully used in several fields, including software engineering and software selection, and has been reported as being effective in a number of case studies and experiments. There are also commercially available tools (such as Expert Choice®) that provide for the entering of judgments, performing all of the necessary calculations, and generating supporting rationale. The output of AHP helps software acquirers, developers and managers make informed decisions. The Analytic Hierarchy Process is a 4-step selection process that is comprised of: 1. Decisions regarding selection criteria 2. Rating of relative criteria importance using mathematical pair-wise comparisons 3. Rating of each potential selection relative to each other selection based on the Step 1 criteria (again using pair-wise comparisons) 4. Combining the two ratings to obtain a normalized relative rating for each potential selection The mathematical theory behind the AHP is beyond the scope of this document. Within the context that it is presented here (software acquisition and development processes), an example of how AHP would be applied is as follows: · Use Quality Function Deployment (QFD) to produce a matrix with required software functionality as one dimension (i.e., customer needs/requirements) and possible COTS software suppliers as the other. · Use Analytic Hierarchy Programming (AHP) to establish a hierarchical rating of the potential suppliers · If necessary, use Linear Programming techniques to optimize the allocation of customer needs/requirements given the relative rating and availability of suppliers
Phase Containment Matrix: A Phase Containment Matrix is a well-known tool for identifying and reducing software defects across the various phases of a software life-cycle. It is constructed by identifying the phases during which a defect is inserted as the column headings, and the phases where the defect was detected as the row headings (see Figure 8).
Figure 8: Phase Containment Matrix for Tracking Software Development Quality
Within each cell, the number of defects that are detected at that phase in the software life-cycle are entered. There is a basic, yet critical, assumption that root-cause defect analysis has been performed in order to accurately determine in which phase of the life-cycle the defect was actually inserted. In-phase defects are often not counted as rework, as these defects are assumed to be a natural part of the process. For example, the rework required to correct the 25 defects that were introduced during the requirements phase, but were also detected and eliminated in the requirements phase, would not be counted, even though there is still potential cost and schedule impact associated with the rework. Ignoring the cost of in-phase rework deludes both the customer and the organization, and deprives the organization of an opportunity to improve its in-phase processes. Analyzing the cost of performing rework on 25 in-phase requirements defects might provide significant justification for improving an acquisition or development process to reduce the number of introduced defects. Out-of-phase defects are generally classified as rework and, just as generally, are going to cost more to fix than in-phase defects. The larger the span (in project phases) between the insertion point and the detection point, the more expensive it is going to be, on a per defect basis, to rework and fix it. This “more expensive” factor can be an order of magnitude of cost between each phase. The concept of a Phase Containment Matrix could, with some adaptation, also be applied to improve software acquisition processes by defining an appropriate set of processes within which software acquisition defects can be detected. Figure 9 provides a simple example of how this might be accomplished.
Figure 9: Process Containment Matrix for Tracking Software Acquisition Quality The interpretation of the results of this phase containment matrix follows the same rationale as the software development life-cycle phase containment matrix. For example, the 20 defects inserted and subsequently detected within the Acquisition Plan/Strategy process represents an opportunity to improve the software acquisition process for a relatively low cost and schedule impact, if root defect causes are identified and corrected. The impact of introducing 10 acquisition planning/strategy defects that are not detected and corrected until the tracking/oversight phase of the software acquisition may be significantly more profound. Again, the introduction and detection of software acquisition defects translates into rework cost and schedule impacts, regardless of whether the defects occur “in-phase” or “out-of-phase”, so a basic paradigm change, coupled with strong senior management and stakeholder commitment, is necessary in order for the technique to be successful.
CHARACTERISTICS OF IMPLEMENTATION:
Enabling Practices: Link to Interrelationships Diagram Enabled Practices: Link to Interrelationships Diagram Impact Areas: Primary: Quality Secondary: All other areas Life-cycle Phase: Primarily in system development and production and deployment phases Scope/Authority: Organizational- or programmatic-level Applicability: Primarily directed towards the acquisition organizations, but has definite impact on the developer Use Indicators: Continued poor performance of acquisition programs (missed milestones, breaching baselines, delivering unacceptable products) Use Inhibitors: Mature organizations that consistently meet proposed managerial/technical targets Appropriate Programs: Relatively large programs; a number of concurrent small programs; projects with marginal staffing Inappropriate Programs: Schedules, outcomes or expectations are easily achieved (e.g., R&D); programs with no permanent organization (e.g., tiger teams) Barriers: Lack of accountability for acquisition excellence; cultural resistance to improvement processes Facilitators: Policy, education and management champions at all levels; availability of documented, demonstrable results
ANTICIPATED BENEFITS OF IMPLEMENTATION:
Successful implementation of software acquisition process improvement can result in: § Realistic, incremental and continuous improvements to software acquisitions – A process of evolutionary software acquisition avoids the temptation to succumb to unrealistic expectations in an attempt to force revolutionary changes in software development. § Adherence to well-understood, well-defined, manageable requirements – An evolutionary software acquisition process ensures that software development is limited to things that are possible to manage. Only requirements that are well-defined and well-understood can be adequately managed. § Following of disciplined, structured software management and development processes that provide predictable results – Disciplined, structured software management and development process phases that end in a management review/gate ensure that the acquisition will remain on track, and that project course corrections can be identified and implemented effectively. § Well-informed, well-disciplined and knowledgeable software acquisition teams – The suitably tailored collection and use of process metrics ensures that the software acquisition is adequately tracked; that project knowledge is confirmed and shared among stakeholders; that acquisition risk is effectively managed; and that software acquirers are firmly integrated into the continuous improvement process. § Increased probability of software acquisition successes and “best value” awards – Heavy reliance on configuration management, risk identification/management, peer reviews and quality assurance, coupled with earned value management techniques, helps ensure a successful and improving acquisition process.
Key Characteristics of the Acquisition Process Improvement Gold Practice
RELATIONSHIPS TO OTHER PRACTICES:
The Figure below represents a high-level process architecture for the subject practice, depicting relationships among this practice and the nature of the influences on the practice (describing how other practices might relate to this practice). These relationship statements are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may “influence” (or be influenced by) the ability to successfully implement other practices. A brief description of these influences is included in the table below.
Process Architecture for the "Acquisition Process Improvement" Gold Practice™
Summary of Relationship Factors
GENERAL
§ OSD Acquisition Web Site
§ OSD Defense Procurement and Acquisition Policy
§ USD(AT&L) Knowledge-Sharing System (AKSS) http://akss.dau.mil/jsp/default.jsp
§ Air Force Acquisition Sites § Secretary of the Air Force/Acquisition (SAF/AQ) § US Air Force Acquisition Center of Excellence http://www.safaq.hq.af.mil/ACE/ § Agile Acquisition Initiative http://www.safaq.hq.af.mil/ACE/agileacq.html § Air Force Capabilities-Based Acquisition System o AFPD 63-1: “Capability-Based Acquisition System” http://www.e-publishing.af.mil/pubfiles/af/63/afpd63-1/afpd63-1.pdf o “Interim Guidance Preceding Air Force Instruction AFI63-101 (Operation of the Capabilities-Based Acquisition System)” http://www.safaq.hq.af.mil/acq_pol/documents/Memo-InterimGuidance63-101.doc § Air Force Acquisition Success Stories http://www.safaq.hq.af.mil/acq_ref/stories.html
§ Dept. of the Navy – Acquisition One Source The DoN Acquisition One Source site is the integration of the ABM Online, Acquisition Reform, Director Acquisition Career Management, and Strategic Business Management web sites. The site is a valuable source for Navy/Marine Corps acquisition related policy, guidance, news, services, tools, and topic information. The site aims to support the broad acquisition workforce (Defense Acquisition Workforce Improvement Act - DAWIA, other government, and industry) with the authoritative information they need.
§ Army Acquisition Sites § Acquisition and Contracting Policy Acquisition Tools http://www.amc.army.mil/amc/rda/rda-ap/aqntools.html § Army Acquisition Support Center Portal The Acquisition Support Center (ASC) is a new field-operating agency under the Assistant Secretary of the Army for Acquisition, Logistics and Technology. It was formed by merging the Army Acquisition Career Management Office with the Army Acquisition Executive Support Agency, as well as career programs CP-14(Contracting) and CP-13/17(LogPro). http://asc.army.mil/portal.cfm
§ DAU Acquisition Research Community of Practice Acquisition Research encompasses all areas of research (both basic and applied) that relate to Acquisition, Logistics and Technology areas. Mission Statement: Provide a forum for use of planned and completed research to influence and shape defense acquisition policy by sharing relevant information and research results among researchers, policy-makers, and other research consumers. http://acc.dau.mil/simplify/ev.php?URL_ID=10161&URL_DO=DO_TOPIC&URL_SECTION=201 (Acquisition Research Community of Practice) http://acc.dau.mil/simplify/ev.php?ID=8122_201&ID2=DO_TOPIC (Software Acquisition)
§ DAU Performance-Based Service Acquisition Resources Expanding the use of Performance-Based Service Acquisitions (PBSA) is a top priority and goal of the Department of Defense. http://acc.dau.mil/simplify/ev.php?URL_ID=15451&URL_DO=DO_TOPIC&URL_SECTION=201
§ SEI Software Acquisition Capability Maturity Model® (SA-CMM®) The Software Acquisition Capability Maturity Model® (SA-CMM®) is a model for benchmarking and improving the software acquisition process. The model follows the same architecture as the Capability Maturity Model for Software (SW-CMM), but with a unique emphasis on acquisition issues and the needs of individuals and groups who are planning and managing software acquisition efforts. http://www.sei.cmu.edu/arm/SA-CMM.html
§ SEI and General Motors CMMI for Acquisition Organizations Project The CMMI for Acquisition Organizations Project was launched cooperatively by the Carnegie Mellon Software Engineering Institute (SEI) and the General Motors Corporation in coordination with the government / industry / SEI CMMI Steering Group. Launched on November 15, 2005, the goal of the project is to create a new business process improvement model for companies looking to source information technology capabilities from third party suppliers. View the press release about this initiative at the following link.
§ Where in Federal Contracting? Guidance – Source Selection This site serves the federal and state acquisition and the federal assistance community, including public and private organizations, by providing quick access to acquisition and assistance information such as contracting laws and pending legislation, current and proposed regulations, guidance, courts and boards of contract appeals, bid protest decisions, contracting newsletters, selected analysis of federal acquisition issues, federal assistance policy, daily listings of grants and cooperative agreements, archived listings of grants and cooperative agreements, and federal assistance sites. http://www.wifcon.com/procurementsource.htm
PAST PERFORMANCE
§ US Army Past Performance Information Management System (PPIMS) https://apps.altess.army.mil/ppims/prod/ppimshp.cfm
§ Contractor Performance Assessment Report System (CPARS) http://www.dscp.dla.mil/subs/contract/cpars.htm
§ NAVSEA Contractor Performance Assessment Report System (CPARS)
§ Federal Past Performance Information Retrieval System (PPIRS) http://www.ppirs.gov/default.htm
Quality Function Deployment (QFD)
§ Quality Function Deployment Institute (QFDI) The QFD Institute is a non-profit organization that researches, develops, and disseminates state-of-the-art methods, tools, and training on QFD and other relevant product development and quality design methodologies.
Analytic Hierarchy Process (AHP)
§ Expert Choice Expert Choice is a provider of advanced decision support software and services. From IT portfolio management to strategic planning to vendor selection, Expert Choice can help your organization make better decisions and improve your bottom line. Expert Choice has entered into a strategic alliance with the Quality Function Deployment Institute (QFDI), which will use Expert Choice’s tools in support of their Quality Function Deployment (QFD) consulting and training practice. § The Quality Portal The link takes you to a brief overview of the Analytic Hierarchy Process, and links to other AHP-related sites, including Expert Choice. http://www.thequalityportal.com/q_ahp.htm
Defense Finance and Accounting Service (DFAS) System Life-cycle Documentation Standards The system life-cycle documentation standards are identified under the life-cycle phase in which each is created. The documentation standards specify the format and content for all life-cycle documents. These standards may be used as templates to help in the completion of documents. Headers and boilerplate text, which must be included in each document, are in regular print in the documentation standards. Instructions or sample information are printed in italics and may be deleted if the standard is used as a template. Phases begin with Pre-Systems Acquisition through Production and Deployment. http://www.dfas.mil/technology/pal/ssps/docstds/lcdocstds.htm http://www.dfas.mil/technology/pal/ssps/docstds/
Acquisition Strategy Decision Guide – Dept. of the Navy This guide was written to assist Program Managers and their teams in the selection and definition of acquisition strategies. It is also intended to be useful in periodically reviewing and revising acquisition strategies when conditions dictate. The guide is built around several “key” concepts and it walks the user through several straightforward step-by-step processes. http://acquisition.navy.mil/navyaos/content/download/1303/6141/file/asdgjan2001.pdf
Contracting for Best Value: A Best Practices Guide to Source Selection – AMC Pamphlet 715-3 This guide provides techniques and practices for obtaining best value products and services through source selection. Consistent with the spirit of acquisition reform, it introduces new and innovative techniques to simplify the source selection process and produce better value. Its purpose is to provide a practical reference tool that will help implement a new way of doing business that promotes flexibility, streamlining, and simplified procedures. http://www.amc.army.mil/amc/rda/rda-ap/ssrc/ssp_toc.htm
Army Source Selection Guide The objective of the Guide was to produce a living document that was web based and lent itself to changes as deemed appropriate. They endeavored to make the Guide simple, straightforward and user-friendly. The document was purposely written as guidance and should not be considered mandatory or regulatory. By incorporating best practices that contracting activities may utilize on an “as-needed” basis, the Guide should prove to be a very valuable tool in conducting source selections. http://www.amc.army.mil/amc/rda/rda-ap/docs/assg-2001.pdf
Expert Choice Expert Choice software is designed to enable decision-makers to develop consensus about what is really important in selecting the best product or service. Expert Choice consultants guide decision-makers to weight criteria using a unique process for developing priorities and building consensus. Expert Choice is often used by government agencies to reduce the amount of time it takes to select the best vendor, then justify the selection against protest using the advanced reporting capabilities of the software. http://www.expertchoice.com/applications/vendorselection.htm
Enterprise Integration (EI) Toolkit The Enterprise Integration (EI) Toolkit contains a Roadmap for you to follow in the development of your component architecture and COTS acquisition and implementation projects. It includes more than 150 templates, checklists, and other tools to help your project be successful.
Guidebooks and Handbooks to Support Acquisition (AT&L Knowledge Sharing System (AKSS))
Software Tools to Support Acquisition (AT&L Knowledge Sharing System (AKSS)) The table presented below is a condensed representation of the list of software tools to support acquisition that is available on the AKSS site. Note that the list of tools is not intended to be complete, as it represents only those tools that are “owned” by a DoD organization. As a result, a number of “commercial” tools are not represented (e.g., COCOMO, SLIM and Cost Xpert in the Price Estimating – Contracting Topic Area). http://akss.dau.mil/jsp/software1.html
Software Acquisition Tools
Contractor Performance Assessment Reports (CPARs) For an actual practice session for entering, reviewing, commenting on and modifying a Contractor Performance Assessment Report (CPAR), click http://cpars.navy.mil/cparsfiles/cpartreq.asp
National Aeronautics and Space Administration (NASA) Past Performance Data Base (PPDB) - NASA has also developed a past performance database to track contractor's performance on NASA contracts. Currently there is no web site with information on NASA's system but for more information, contact NASA Headquarters at (202) 358-1279.
NIH Contractor Performance System
Air Force Single Acquisition Management Plan Guidelines http://www.safaq.hq.af.mil/contracting/toolkit/part07/word/sampguide.doc
SPAWAR Acquisition/Documentation Matrix (Generic) http://enterprise.spawar.navy.mil/apsg/Enclosure5.pdf
Defense Systems Management College (DSMC) Acquisition Strategy Guide The purpose of this Guide is to provide, in a single source, information that PMs should find useful in structuring, developing, and executing an acquisition strategy. A process for developing and executing an acquisition strategy is provided together with criteria for evaluating a proposed strategy. However, this Guide alone does not provide the PM with a definitive acquisition strategy for ones particular program. Well informed, educated, and innovative applications and judgments concerning the particular mission need are necessary to structure a successful acquisition strategy. PMs should continue to seek guidance, data, and assistance from available sources as they prepare and revise their acquisition strategy. http://www.ntsc.navy.mil/Resources/Library/Acqguide/acqstrat.pdf
Tools from the Data and Analysis Center for Software (DACS)
The DACS “Best Value Analysis for Acquisition” tool – DACS has developed a basic tool, using MS Excel®, as a means for comparing bidder’s proposals on technical approach, management approach, past performance and price to determine the combined “best value” proposal based on user-defined evaluation, rating and weighting criteria. There are two versions of this tool. The “Fixed” version allows input of proposal scoring criteria that will be applied to all of the evaluation categories (technical, management, past performance, price). The “Variable” version allows input of tailored scoring criteria for each of these four categories. These tools can be downloaded from:
The DACS “Acquisition Risk Management Scorecard” tool – DACS has developed a MS Excel®-based tool to allow for an assessment of acquisition risk based on either basic, or more advanced, categories of risk. For the basic tool, three qualitative levels of risk are defined (high, medium, low) and user-defined relative weighting scores can be assigned to each. For the advanced tool, the three qualitative areas are subdivided into quantitative ratings, allowing for enhanced risk-scoring discrimination. This tool can be downloaded from: https://goldpractice.thedacs.com/practices/api/Acquisition_Risk_Management_Scorecard.xls
The DACS “Software Acquisition Management Risk Factors” tool – A slightly different risk management scorecard, this MS Excel®-based tool is designed to allow you to score the risk associated with 10 predefined areas of software acquisition risk. The risk factors were taken from the “Software Acquisition Risk Factors” matrix located at the State of Texas Department of Information Resources website. This tool can be downloaded from: https://goldpractice.thedacs.com/practices/api/Software_Acquisition_Management_Risk_Factors.xls
The DACS “Software Acquisition Process Improvement Interview” – This interview/survey was developed by the DACS to support the definition and implementation of a DoD Software Acquisition Process Improvement Program. Section 804 of the Bob Stump National Defense Authorization Act for Fiscal Year 2003 requires the establishment of software process improvement programs by Military Departments and those Defense Agencies that manage Major Defense Acquisition Programs with a substantial software component (i.e., software-intensive systems (SIS)).
Software Acquisition Process Maturity Questionnaire - This package contains a copy of the software acquisition process maturity questionnaire. It is intended for those interested in performing and learning about software acquisition process appraisals. This questionnaire is not an appraisal method itself; rather, it is a tool that may be used in different appraisal methods. http://www.sei.cmu.edu/pub/documents/97.reports/pdf/97sr013.pdf
Tools for Earned Value Management Listed at the DoD Acquisition Web Site This site provides links to material on software tools for earned value management which can be useful for tracking the progress of an acquisition over its life-cycle. http://www.acq.osd.mil/pm/old/tools/tools.htm
Proposal Evaluation Tool (PET) This software tool was developed to answer the question: "How should I allocate my limited resources to maximize the benefit to my organization?" PET is an interactive, Web-based application that uses the Analytical Hierarchy Process and Integer Linear Programming to identify which combination of proposals will maximize the total value to the organization within a specified budget. PET will also support portfolio planning over multiple budget years. The development of this tool was funded by the Office of Naval Research (ONR) Affordability Measurement and Prediction Program (AMPP), to support its efforts in science and technology portfolio prioritization analysis. http://www.tecolote.com/Services/SystemAcquisition.htm
Software Risk Checklist The NASA QE(8000) Risk Management Office produced this 18-page software risk checklist, which is laid out by development phases of a project and is specifically constructed to cover various high-level life-cycle phases, including the system requirements phase, the software planning phase, the software requirements phase, the software design phase, the software implementation phase, and the acceptance testing/release phase. http://osat-ext.grc.nasa.gov/rmo/spa/SoftwareRiskChecklist.doc
Katherine V. Schianasi GAO, Director, Acquisition and Sourcing Management Suellen Eslinger, Aerospace Corporation, Software Acquisition and Process Office Richard Adams, Aerospace Corporation, Software Acquisition and Process Office Lisa Pracchia, NAWCWD Dr. Richard Turner, OUSD(AT&L) Dr. Barry Boehm, Univ. of Southern Cal. Thomas Bernard, ASC/ENAC Brian Gallagher, Software Engineering Institute (SEI)
§ Defense Acquisition Process Tutorial (Defense Acquisition University – DAU) http://dod5000.dau.mil/TUTORIAL/index.htm
§ For an interactive Web Training tutorial on CPARS with "Show Me" features, click http://cpars.navy.mil/cparsfiles/cpcbtdlf.asp
§ Navy/Air Force Practice CPARS Training http://cpars.navy.mil/cparsfiles/cpartreq.asp
§ CPARS Training Materials http://cpars.navy.mil/cpartrng/cpars_training.asp#trnmatl
§ Single Acquisition Management Plan (SAMP) Training (MS PowerPoint presentation as part of SMC Acquisition Reform Training) – Ronald G. Martin HQ SMC/ADX http://ax.losangeles.af.mil/axd/training/samp%20devlp%20v0%20rm.ppt
§ Incentives for Reducing Acquisition Response Time – AFIT SIS 352
The Air Force Institute of Technology presents this short, Internet-based course which provides an overview of incentives available to motivate both government program office personnel as well as contractor personnel to reduce acquisition response times in their day-to-day business. We show that the perception is that government and contractor personnel are not incentivized to exceed program objectives but at best to only meet program objectives. Through examples and short exercises the student realizes that incentives can be effective to reduce acquisition response times. The student is also introduced to Schedule Risk Assessments and their usage as part of source selection. In order to take this course, you must first register with the AFIT Virtual Schoolhouse and be assigned to a section (follow the link below).
§ Introduction to the Software Acquisition CMM
This two-and-a-half-day course introduces those responsible for the acquisition of software-intensive systems, software products, and services to the Capability Maturity Model®: for Software Acquisition (SA-CMM®) and its fundamental concepts as well as the value it can bring to organizations using it. http://www.sei.cmu.edu/products/courses/intro/intro.sa.cmm.html
§ DACS Affordability Training Module “Software Acquisition for Software-Intensive Systems (SIS)”
This is a MS PowerPoint presentation that provides an overview of the processes and procedures associated with the acquisition of software for complex systems. https://goldpractice.thedacs.com/practices/api/Software_Acquisition_for_SIS.ppt
§ Defense Acquisition University (DAU) Continuous Learning Center
This link provides access to several DAU continuous learning modules based on Software Acquisition Management. Just select “Software Acquisition Management” from the Topic drop-down menu to review the available modules. http://clc.dau.mil/kc/no_login/portal.asp?strRedirect=LC_CIA
APPENDICES
Analyzing current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies
Apply process improvement tools (Capability Maturity Models, etc.) to acquisition organizations.
SOURCES (Origins of the Practice):
NONE INDICATED
Section 804. Public Law 107-314, “Bob Stump National Defense Authorization Act for Fiscal Year 2003 – Improvement of Software Acquisition Processes”. (Excerpts)
(a) ESTABLISHMENT OF PROGRAMS. (1) The Secretary of each military department shall establish a program to improve the software acquisition processes of that military department. (2) The head of each Defense Agency that manages a major defense acquisition program with a substantial software component shall establish a program to improve the software acquisition processes of that Defense Agency. (b) PROGRAM REQUIREMENTS. A program to improve software acquisition processes under this section shall, at a minimum, include the following: (1) A documented process for software acquisition planning, requirements development and management, project management and oversight, and risk management. (2) Efforts to develop appropriate metrics for performance measurement and continual process improvement. (3) A process to ensure that key program personnel have an appropriate level of experience or training in software acquisition. (4) A process to ensure that each military department and Defense Agency implements and adheres to established processes and requirements relating to the acquisition of software. (c) DEPARTMENT OF DEFENSE GUIDANCE. The Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, in consultation with the Under Secretary of Defense for Acquisition, Technology, and Logistics, shall: (1) prescribe uniformly applicable guidance for the administration of all of the programs established under subsection (a) and take such actions as are necessary to ensure that the military departments and Defense Agencies comply with the guidance; and (2) assist the Secretaries of the military departments and the heads of the Defense Agencies to carry out such programs effectively by: (A) ensuring that the criteria applicable to the selection of sources provides added emphasis on past performance of potential sources, as well as on the maturity of the software products offered by the potential sources; and (B) identifying, and serving as a clearinghouse for information regarding, best practices in software development and acquisition in both the public and private sectors.
DoD Directive 5000.1, “The Defense Acquisition System”, 12 May 2003
Para. 4.3.1, Pg. 2: Flexibility – There is no one best way to structure an acquisition program to accomplish the objective of the Defense Acquisition System. MDAs and PMs shall tailor program strategies and oversight…to fit the particular conditions of that program…
Para. 4.3.2, Pg. 2: Responsiveness – Evolutionary acquisition strategies are the preferred approach to satisfying operational needs. Spiral development is the preferred process for executing such strategies.
Para. 4.3.3, Pp. 2-3: Innovation – Throughout the Department of Defense, acquisition professionals shall continuously develop and implement initiatives to streamline and improve the Defense Acquisition System. MDAs and PDAs shall examine and, as appropriate, adopt innovative practices (including best commercial practices and electronic business solutions) that reduce cycle time and cost, and encourage teamwork.
Para. E1.2, Pg. 4: Collaboration – The DoD acquisition, capability needs and financial communities, and operational users shall maintain continuous and effective communications with each other by using Integrated Product Teams (IPTs).
Para. E1.14, Pg. 5: Knowledge-Based Acquisition – PMs shall provide knowledge about key aspects of a system at key points in the acquisition process.
Para. E1.16, Pg. 6: Performance-Based Acquisition – …acquisition managers shall consider and use performance-based strategies for acquiring and sustaining products and services whenever feasible.
Para. E1.18, Pg. 6: Products, Services and Technologies – …conduct market research and analysis to determine the availability, suitability, operational supportability, interoperability, safety, and ease of integration of the considered and selected procurement solutions. The DoD Components shall work with users to define capability needs that facilitate…procurement or modification of commercially available products, services and technologies, …, or the development of dual-use technologies.
Para. E1.19, Pg. 6: Professional Workforce – The DoD shall maintain a fully proficient acquisition…workforce that is flexible and highly skilled across a range of management, technical and business disciplines. …the USD(AT&L) shall establish education, training and experience standards for each acquisition position….
Para. E1.21, Pg. 7: Program Stability – The DoD Components shall develop realistic program schedules, long-range investment plans and affordability assessments, and shall strive to ensure stable program funding.
Para. E1.25, Pg. 7: Software Intensive Systems – Acquisition of software intensive systems shall use process improvement and performance measures. Selection of sources shall include consideration of product maturity and past performance.
Para. E1.26, Pg. 7: Streamlined Organizations – The DoD shall use a streamlined management structure in the acquisition system, characterized by short, clearly defined lines of responsibility, authority and accountability.
Para. E1.27, Pg. 7: Systems Engineering – Acquisition programs shall be managed through the application of a systems engineering approach that optimizes total system performance and minimizes total ownership cost. A modular, open-systems approach shall be employed, where feasible.
DoD Instruction 5000.2, “Operation of the Defense Acquisition System”, 12 May 2003
Para. 3.3, Pg. 3: Evolutionary Acquisition – Evolutionary acquisition is the preferred DoD strategy for rapid acquisition of mature technology for the user. An evolutionary approach delivers capability in increments, recognizing, up front, the need for future capability improvements. The success of the strategy depends on consistent and continuous definition of requirements, and the maturation of technologies that lead to disciplined development and production of systems that provide increasing capability towards a materiel concept.
Air Force Policy Directive AFPD63-1, “Capability-Based Acquisition System”
Para 7.2, Pg. 6: Acquisition Strategy – Evolutionary acquisition (EA) is the preferred acquisition strategy… Spiral development is the process to execute the EA strategy.
CASE STUDIES FROM THE LITERATURE:
The case studies discussed in this section and summarized in the table below represent an assessment by the US Government Accountability Office of five major DoD software-intensive weapon system acquisitions, with mixed results. The report states that “when DoD managers had a smaller, more evolutionary product with manageable requirements, used disciplined development processes with gated reviews, and collected/used metrics to manage software development progress…they delivered their product with less cost increase and less schedule delay”. Conversely, “when DoD managers had expectations of developing revolutionary capabilities and did not use structured management reviews or collect /use metrics for software development…they experienced significant cost growth and schedule delays”. The change percentages for each scenario/program are illustrated in the table. Program Outcomes Linked to Management Controls [GAO, 2004]
NOTE: For the F/A-22, SBIRS and Comanche programs, the GAO assessments address conditions found before the restructuring of these programs.
Successful DoD Acquisitions [GAO, 2004] The Tactical Tomahawk and F/A-18 C/D programs were developed in an evolutionary environment, engaged in extensive work on requirements, controlled requirements changes, collected and used detailed metrics to track development progress, and had less cost and schedule increase than the other programs reviewed in the report.
The Navy’s Tactical Tomahawk missile will provide ships and submarines with enhanced capability to attack targets on land. New features include improved anti-jamming global positioning system, in-flight retargeting, and the ability to transmit battle damage imagery. Tomahawk program developers had disciplined development processes and used extensive peer reviews to discover defects and provided the acquirer with insight at each stage of development. They were responsible for collecting/reporting data on a monthly basis, relying on metrics—cost, schedule, effort, size, requirements, testing, and defects that are similar to those used by leading commercial firms. The program office managed the acquisition based on the trends found in these metrics.
The F/A-18 C/D is a Navy attack fighter aircraft that has been deployed for a number of years. Periodically, the Navy upgrades the flight software to incorporate new features, add the capability to fire new munitions, and correct deficiencies discovered since the last upgrade. Working in an evolutionary environment, F/A-18 C/D program officials recognized that the success of the software upgrade to incorporate additional performance into the flight operations software depended on extensive requirements analysis before program start and firm control as requirements changed throughout development. This analysis ensured that the effort needed to meet requirements was well understood at the beginning of development, thus limiting the amount of redesign. Proposals for new requirements or changes to requirements after the program began were analyzed for cost, schedule, and performance impact. As with the Tomahawk program, F/A-18 developers adhered to disciplined development processes, used extensive peer reviews to discover defects, and collected meaningful metrics to track progress.
“Unsuccessful” DoD Acquisitions [GAO, 2004] The F/A-22, SBIRS, and Comanche are complex programs that attempted to achieve quantum leaps in performance requiring extensive use of software rather than follow an evolutionary approach to software development. They all initially lacked controls over requirements, software processes, and metrics, thereby causing major program upheavals. They encountered significant requirements changes, schedule slips, and cost increases because software defects were not discovered until later stages of the programs. Each of these programs has been restructured to incorporate requirements management controls, more-defined software development processes and additional metrics.
The Air Force’s F/A-22, originally planned to be an air dominance aircraft, will also have air-to-ground attack capability. It is expected to have advanced features, such as stealth characteristics, to make it less detectable to adversaries and capable of high speeds for long ranges. The F/A-22’s avionics are designed to greatly improve pilots’ awareness of the situation surrounding them. Early in the development process for the F/A-22, it was reported that the program’s planned strategy for software development and acquisition was generally sound. The Air Force’s plans to collect software costs and other software metrics to measure progress as examples of this sound strategy were cited as the reasons. At that time, the program’s plans to be event- rather than schedule-driven were endorsed. However, as early as 1994, many features of this sound strategy were not being followed. Delayed software deliveries contributed to cost increases and schedule delays. Requirements and design changes accounted for 37 percent of the critical problem reports leading to avionics shutdowns in the F/A-22, according to program office reports. Program officials and contractor personnel agreed that requirements volatility had been a problem; however, they were unable to provide any specific measure of requirements changes because they had not tracked the overall growth in software requirements since the first 3 years of the program. According to Lockheed Martin officials, the avionics system software is made up of 84 computer software configuration items, 6 each of which account for a specific avionics function, such as the interaction between the pilot and the aircraft. In discussions with contractor and program personnel, it was stated that disciplined processes in requirements control, design, testing, and configuration management were not uniformly followed because of cost and schedule pressures. The F/A-22 software strategy also called for the collection of software metrics to measure costs. Program and contractor officials were unable to provide metrics for sufficient management visibility over the overall progress of the software. The contractor stated that the Air Force did not compile metrics from lower levels into major segments such as avionics.
The Air Force’s SBIRS satellites are being developed to replace DOD’s older missile-warning satellites. In addition to missile warning and missile defense missions, the satellites will perform technical intelligence and battlespace characterization missions. Since the program was initiated in 1996, SBIRS has faced cost, scheduling, and technology problems. It was reported that SBIRS has experienced serious software design problems. Officials from Lockheed Martin, the prime contractor, stated that the program had uncontrolled requirements growth, as well as overly optimistic expectations about reusing software from a previous program. Program and contractor officials agreed that deficient systems engineering and the scarcity of personnel in software engineering disciplines contributed to ineffective control and to not understanding how much of the previous software could be reused. These officials also stated that neither the program office nor the contractor had a change management control process in place to analyze change requests. A thorough analysis late in the program revealed that very little of the software could be reused. Furthermore, because of a deficiency in resources devoted to systems engineering, the total requirements for the system were not adequately defined. A report from an independent review team stated that more robust systems engineering could have precluded some of the problems. The report concluded that problems with the first SBIRS increment were primarily due to problems with software development and poor program execution. Peer reviews and engineering review boards were in place to monitor development, but, for reasons ranging from schedule pressures to reduced staffing, these decision bodies were ineffective. SBIRS contractor officials stated that they collected data on additions to requirements and on the number of lines of code, but because there were no restrictions on accepting new requirements and no control limits to the size of code, the metrics were not used to manage the project on a daily basis.
The Army’s Comanche is a multi-mission helicopter intended to perform tactical armed reconnaissance. It is designed to operate in adverse weather across a wide spectrum of threat environments and provide improved speed, agility, reliability, maintainability, and low observability over existing helicopters. Since the program’s first cost estimate, originally approved in 1985, the research and development cost for Comanche has almost quadrupled, and the time to obtain an initial capability has increased from 9 to over 21 years. Several studies have identified software development as a problem area and highlighted requirements volatility and inadequate requirements analysis as having a large impact on the program. The lack of a disciplined process for Comanche’s software acquisition was also cited as a reason for program shortfalls; however, the exact percentage of cost growth attributed to software is not known because the program office lacked adequate visibility into the software development process and, therefore, has little historical data on software. Comanche officials stated that initially they did not require a uniform set of metrics from the contractor. They said they received earned value information from the contractor, but it combined software and hardware development data.
All three programs have been restructured and have instituted changes to bring more knowledge into the programs. For example, F/A-22 program officials report that their contractors have teamed with divisions within their companies that have more disciplined processes and they are reporting fewer problems with the avionics software. SBIRS program officials stated that they have instituted more controls over requirements changes, requiring analysis and approval at higher levels. Comanche officials reported that the program office has quarterly software reviews to focus attention on software development progress with the contractor and has adopted an incremental, block development strategy for software development. Program officials stated that they have asked for more detailed metrics by which to manage the programs.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The DACS on the Web
RSS Feeds
Wikipedia
Facebook
Linkedin
Vimeo
|
|
Contact Us
Technical Questions?Submit Form PolicyJoin The DACS!Register Now Benefits |
|
DACS ResourcesJournal of Software Technology Services / Consultation Bibliographic Database Cost Estimation Tool Kit Calendar of Events Technical Reports Online Learning DoD Acronyms Webinars Store |
|
About The DACS |
Quanterion Solutions Incorporated, Copyright © 2011, Privacy Policy
For Best Browsing Experience: IE8 / Firefox 9.0+ / Chrome 10+, 1024x768+, 96 DPI
For Best Browsing Experience: IE8 / Firefox 9.0+ / Chrome 10+, 1024x768+, 96 DPI
|
You need to be logged in to give feedback
|
