SOFTWARE ACQUISITION GOLD PRACTICE ^™

ACQUISITION PROCESS IMPROVEMENT

FOCUS AREA:    PROCESS

Definition and Summary: The analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies, including specific process-oriented tools, methods, models, etc. (such as the SEI SA-CMM ^® and CMMI-AM^®) to aid in process improvement.

 

“ Organizations that have a strong, consistent evolutionary environment and practices for setting product requirements, maintaining a disciplined development process and using metrics to oversee development progress achieve favorable cost, schedule and quality outcomes .”

This conclusion [GAO, 2004] should serve as the rallying point for any and all acquisition organizations who are struggling to improve the results from their software acquisition processes.

“Decisions to begin programs and make significant investments are made [by DoD] without matching requirements to available resources and without demanding sufficient knowledge at key points.”

Taken from the same report, this statement provides the realization that there is still a long way to go and that, unfortunately, some may not have yet started on the journey.

 

Successful implementation of software acquisition process improvement can result in:

§         Realistic, incremental and continuous improvements to software acquisitions

§         Adherence to well-understood, well-defined, manageable requirements

§         Following of disciplined, structured software management and development processes that provide predictable results

§         Well-informed, well-disciplined and knowledgeable software acquisition teams

§         Increased probability of software acquisition successes and “best value” awards

DESCRIPTION OF THE PRACTICE:

 

SUMMARY DESCRIPTION

Acquisition process improvement has been defined as the analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies [Eslinger, 1999].  The definition provided in Turner’s dissertation [Turner, 2002] incorporates the concept of specific process-oriented tools (such as the SEI CMM in 2002, and expanded upon by the SA-CMM^® in 2003 and CMMI-AM^® in 2004) to aid in the acquisition improvement process.  The Navy [Pracchia, 2004] uses, perhaps, the most comprehensive definition of software acquisition process improvement, defining it as:

•          A documented process for software acquisition planning, requirements development/management, project management/oversight and risk management

•          Efforts to develop appropriate metrics for performance measurement and continual process improvement

•          A process to ensure that key program personnel have an appropriate level of experience/training in software acquisition

•          A process to ensure that each military department and defense agency select, implement and adhere to established processes and requirements relating to software acquisition

One of the better descriptions of the characteristics of a software acquisition process that is successfully being used by leading software acquirers/developers is contained in the “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” report from the Government Accountability Office (GAO – formerly the General Accounting Office) [GAO, 2004].  Three fundamental management strategies are recommended within this report:

·       Evolutionary Environment

·          Disciplined Development Processes

·          Collecting/Analyzing Meaningful Metrics

These strategies limit software development efforts to what can be managed and result in decisions based on sufficient systems engineering knowledge to adequately assess risks.  In addition, business decisions are consciously made to invest time/resources in achieving high software process maturity levels.  Strong upper-level management support is a strong enabler for ensuring that these combined strategies are successful.

 

DETAILED DESCRIPTION

“Organizations that have a strong, consistent evolutionary environment and practices for setting product requirements, maintaining a disciplined development process and using metrics to oversee development progress achieve favorable cost, schedule and quality outcomes.”

This conclusion, taken from a 2004 report entitled “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” [GAO, 2004], should serve as the rallying point for any and all acquisition organizations who are struggling to improve the results from their software acquisition processes.

“Decisions to begin programs and make significant investments are made [by DoD] without matching requirements to available resources and without demanding sufficient knowledge at key points.”

Taken from the same report, this statement provides the realization that there is still a long way to go and that, unfortunately, some may not have yet started on the journey.

The “Acquisition Process Improvement” Gold Practice begins with some observations of what constitutes a software acquisition process, then looks at some of the fundamental management strategies that have been defined as successful in the software acquisition process and assesses their characteristics and potential impact on the process.  In the context of this Gold Practice document, the following definitions are used:

Acquisition

The conceptualization, initiation, design, development, test, contracting, production, deployment, Logistics Support (LS), modification, and disposal of software systems, supplies or services to satisfy customer needs.

Acquisition environment

Internal and external factors that impact on, and help shape, every software acquisition program.  These factors include political forces, policies, regulations, reactions to unanticipated requirements and emergencies.

Acquisition life-cycle

The life of a software acquisition program consists of phases, each preceded by a milestone or other decision point, during which a system goes through Research, Development, Test and Evaluation (RDT&E) and production.

 

Acquisition program

A directed, funded effort over the software acquisition life-cycle that provides a new, improved, or continuing materiel, weapon or information system or service capability in response to an approved need.

Acquisition process

The steps involved, and practices employed, in the planning, contracting, implementation, acceptance and follow-on activities of a software acquisition.  The acquirer will bear most of the responsibility for the results from the planning, contracting, and follow-on phases of the acquisition, while the developer is most accountable for the results coming out of the implementation and acceptance phases of the acquisition.

Development process

The process of working out and extending the theoretical, practical, and useful applications of a basic design, idea, or scientific discovery, characterized by the design, coding, modification or improvement of software.

 

The Software Acquisition Process

It is necessary to first understand what a software acquisition process is before steps can be taken to improve it.  IEEE Std 1062, 1998 Edition, entitled “Software Acquisition Process Steps” [IEEE, 1998] describes a generic nine-step process for software acquisition, summarized in Figure 1. 

 

Figure 1:  IEEE Std 1062, 1998 Edition Software Acquisition Process Steps [IEEE, 1998]

 

There are three steps that comprise the planning phase of a software acquisition.  During this phase, the acquisition strategy should be planned based on a review of the acquirer’s objectives.  A formal Acquisition Strategy document is developed to guide the acquisition.  Implementing the acquisition strategy into the organization’s process is the next step.  Appropriate contracting practices need to be included in the process.  The final step of the planning process encompasses the determination of software requirements.  In addition to defining the software being acquired, this step includes the preparation of the quality and maintenance plans for accepting the supplier’s/developer’s software.  The release of a Request for Proposal (RFP) is the completion milestone for the planning phase of the software acquisition life-cycle.

The contracting phase of the software acquisition is also comprised of three steps.  The first step is the identification of potential suppliers.  Suppliers should be selected who will provide documentation for their software, demonstrate its performance, and submit formal proposals.  Failure to perform any of these actions may serve as a basis for rejecting a potential supplier.  Past performance data from previous contracts should be reviewed for all potential suppliers.  The next step is to actually prepare the contract requirements.  The quality of work (acceptable performance and acceptance criteria) should be described, and contract provisions that tie payments to deliverables should be prepared.  Legal counsel should also be sought to review and approve contractual language/requirements.  Finally, proposals will be evaluated and a qualified supplier(s) will be selected.  If appropriate, an alternate supplier should be negotiated with, primarily as a risk reduction measure.  Supplier selection completes the contracting phase of the software acquisition life-cycle, which started with the release of the RFP and ends with the contract signing.

Once the contract is signed, software development can begin and the product implementation phase of the software acquisition life-cycle is initiated.  During this phase, the acquirer must manage supplier performance.  The supplier’s progress is monitored to ensure all milestones are met.  The acquirer also must approve all appropriate work products.  Note that, during this phase, the acquirer has the responsibility to provide all acquirer deliverables to the supplier when required.  The product implementation phase of the software acquisition is completed when the software product is delivered to the acquirer (pre-approval).

The next phase of the software acquisition life-cycle is product acceptance.  Adequate testing needs to be performed, and a process for certifying that all discrepancies have been corrected and all acceptance criteria have been satisfied must be established.  Acceptance of the software product by the acquirer based on mutually agreed-to pre-defined criteria signals the end of the product acceptance phase.

The follow-on phase of software acquisition covers the period between software acceptance and software retirement, when it is no longer in use.  A follow-up of the software acquisition contract should typically be performed to evaluate contracting practices, record lessons learned, and evaluate user satisfaction with the software.  During this final phase of the software acquisition life-cycle, the acquirer should record and retain supplier performance data to use for the acquisition of future software products.

More recently, the Software Engineering Institute (SEI) has released the Software Acquisition Capability Maturity Model (SA-CMM^®) [SA-CMM^®, 2002] to represent the important phases and elements of the software acquisition processes.  Table 1 highlights the correlation between the five major IEEE Std 1062 software acquisition process steps discussed above [IEEE, 1998] and the Level 2 Key Process Areas (KPAs) of the SA-CMM^®.  The Level 3 through 5 KPAs are equally important for specific acquisition phases, or for all acquisition phases.  Knowledge of the basic concepts behind the SEI CMM modules is assumed.  The relationship of the SA-CMM^® to the more recently released CMMI-AM^® process [CMMI-AM^®, 2004], which imbeds the concepts of Integrated Product and Process Development (IPPD) into the software acquisition process, thereby emphasizing the concept of acquisition process improvement, will be discussed later.  Note that the IEEE Std 1062, SA-CMM^® and the CMMI-AM^® processes all focus on what should be done, rather than how it should be done, representing one of the biggest challenges being faced by software acquirers in achieving software acquisition process improvement.

Table 1:  SA-CMM^® Key Process Areas

 

What is Software Acquisition Process Improvement?

Acquisition process improvement has been defined [Eslinger, 1999] as the analysis of current software acquisition processes for deficiencies and implementing new/modified processes to correct those deficiencies.  The definition provided in Turner’s dissertation [Turner, 2002] incorporates the concept of specific process-oriented tools (such as the SEI CMM) to aid in the acquisition improvement process.  The Navy [Pracchia, 2004] uses, perhaps, the most comprehensive definition of software acquisition process improvement, defining it as:

 

•          A documented process for software acquisition planning, requirements development/management, project management/oversight and risk management

•          Efforts to develop appropriate metrics for performance measurement and continual process improvement

•          A process to ensure that key program personnel have an appropriate level of experience/training in software acquisition

•          A process to ensure that each military department and defense agency select, implement and adhere to established processes and requirements relating to software acquisition

 

How is DoD Software Acquisition Process Improvement Being Addressed?

In a presentation on Software Acquisition Process Improvement, Borden [Borden, 2004] addresses nine target areas for improvement which pretty much cover the entire acquisition/development life-cycle:

·          Acquisition planning

·          Requirements development/management

·          Configuration management

·          Risk management

·          Project management/oversight

·          Test & evaluation

·          Integrated team management

·          Solicitation/source selection

·          Performance measurement

Dr. Barry Boehm [Boehm, 2004] has described a number of general software acquisition goals (defined as critical success factors) that should be addressed to support software acquisition process improvement:

·          System/software objectives and constraints are adequately defined/validated

·          System/software acquisition strategies are appropriate/compatible

·          Success-critical stakeholders have committed adequate software capability/resources to perform their software-related tasks

·          Software product/process plans are feasible/compatible

·          Feasibility Rationale provides convincing evidence that software progress is satisfactory

There are a number of other published reports that contain more specific recommendations on improvements that can be made to the DoD software acquisition process.  These improvements include (1) documenting agreements between the software developer and acquirer that contain baseline requirements based on systems-engineering knowledge, (2) gated reviews during the software development process, (3) obtaining meaningful metrics from the developer for managing the program, (4) attaining knowledge about the technology that is planned early in the development process, (5) ensuring an appropriate balance between requirements and available resources, (6) ensuring that the software design is mature before it is released, and (7) having all processes under control prior to release.  In general, the more that acquisition managers know about software development processes and metrics, the better equipped they are to improve the processes for software acquisition.

The Navy is also in the process of developing a formal Software Acquisition Process Improvement Plan (SAPIP) [Pracchia, 2004].  The burden is placed on ASD/C3I and OUSD(AT&L) (analogous to “senior management”) to (1) provide applicable improvement program administration and compliance guidance and ensure that secretaries of the departments and selected agencies comply with that guidance and (2) assist departments/agencies with their respective improvement programs by ensuring the use of applicable source-selection criteria and access to information regarding software acquisition/development best practices in both the public and private sectors.

One of the better descriptions of a software acquisition process that is successfully being used by leading software acquirers/developers is contained in the “Stronger Management Practices are Needed to Improve DoD’s Software-Intensive Weapons Acquisitions” report from the GAO [GAO, 2004].  Three fundamental management strategies are recommended within this report:

·       Evolutionary Environment

·          Disciplined Development Processes

·          Collecting/Analyzing Meaningful Metrics

These strategies limit software development efforts to what can be managed, and they result in informed decisions based on sufficient systems engineering knowledge such that risk can be adequately assessed.  In addition, business decisions are consciously made to invest sufficient time and resources in order to achieve high software process maturity levels (high quality, on-time, within budget).  Committed upper-level management support is an effective enabler for ensuring that these combined strategies are successful.

The DoD has, in many software acquisitions, achieved poor results due to inconsistent practices in these areas, as decisions to begin programs and make significant investments are made without ensuring that adequate resources are available to meet requirements, and without demanding critical knowledge at key decision points.  While the DoD has made attempts to improve software acquisition processes by establishing a framework for an evolutionary environment in its requirements generation and acquisition policies, the GAO report concludes that the DoD policies do not contain the controls needed to ensure that individual programs will adhere to disciplined requirements engineering and management processes, nor do they include the necessary metrics to do so.  The report also observed that the Services’ and Milestone Decision Authority (MDA) planning documentation did not include practices proven successful in leading commercial firms.  Suggestions for strengthening these plans included specific criteria to ensure that:

·          Requirements baselines based on systems engineering are documented/agreed to by both the acquirer/developer before a program is initiated

·          Cost/benefits analyses are required when new requirements are proposed

·          Software acquirers/developers make efforts to continually improve practices over time

·          Gated reviews and deliverables are integrated into development processes

·          Requiring software developers to collect/analyze metrics (including earned value) to obtain knowledge of progress and to manage risk

While there is recognition that care must be taken to not make these plans too prescriptive, thereby stifling software developer innovation, it is necessary to instill some level of control to ensure future consistent success for DoD software acquisitions.

 

Evolutionary Environment:

An evolutionary software acquisition environment is characterized by incremental improvements to software performance, as opposed to succumbing to programmatic pressures to set unrealistic expectations, in order to improve software processes on a continual basis.  An evolutionary environment limits software development to what is possible to manage by adhering to well-understood, well-defined, manageable requirements while simultaneously encouraging continuous process improvement.  An evolutionary product development is one of the fundamental elements of a manageable acquisition environment.  The concept of continuous improvement is a critical part of the evolutionary environment, and must be supported by both the environment and the organizational culture (including senior management) in order to be successful.  Ad hoc processes make it impossible to understand exactly when and how defects in the process occur.

DoD Instruction 5000.2, dated 13 May 2003 [DoDI 5000.2, 2003], Section 3.3 formally defines the concept of evolutionary acquisition for DoD acquisition.  It further breaks down this definition into two basic processes: Spiral Development and Incremental Development.  For Spiral Development, a desired capability is identified, but the end-state requirements are not known at program initiation.  Requirements are refined through demonstrations and risk management, with continuous user feedback.  Each spiral provides the user with the best possible capability at that point in time.  Requirements for future increments depend on feedback from users and technology maturation.  For Incremental Development, a desired capability is identified, end-state requirements are known, and the requirement is met over time by developing several increments, each dependent on available mature technology.  The DoD Requirements and Acquisition Process Depiction from DoDI 5000.2 is represented in Figure 2, where each of the three software development increments represents a spiral, and each spiral has demonstration, Milestone B and Milestone C requirements.  Figure 3 shows a notional comparison between the “traditional” approach to acquisition and the evolutionary acquisition process.

Figure 2.  DoDI 5000.2 (12 May 2003) Requirements and Acquisition Process Depiction (Refer to [DoDI 5000.2, 2003])

 

 

_{Note that percentages and increments are notional, not factual}

Figure 3:  Traditional vs. Evolutionary Acquisition Strategies [AFPD63-1, 2003]

 

The Air Force Agile Acquisition initiative [Sambur, 2003], Air Force Policy Directive AFPD 63-1 [AFPD 63-1, 2003] and Interim Guidance to AFI 63-101 [AFI 63-101, 2003] on a Capability-Based Acquisition System, provide additional emphasis on the evolutionary acquisition process.  Particular focus is placed on collaborative processes covering requirements development, seamless verification (i.e., capabilities-based test and evaluation), and technology transition processes.  Further discussion on the Air Force Agile Acquisition Initiative can be found at their Acquisition Center of Excellence website, located at http://www.safaq.hq.af.mil/ACE/.

 

Disciplined Development Processes:

Developers must follow disciplined, structured management and development processes in order for the software acquisition process to improve.  Each phase of the software acquisition should end in a management review/gate to ensure that the project is on track.  To pass these gated reviews, developers must demonstrate that acquirer’s expectations and quality standards are met before proceeding to the next phase.  Peer reviews to check each other’s work and remove defects, beginning at the earliest stages of software development, are necessary to prevent costly rework and schedule delays resulting from downstream defect detection.  Disciplined software development processes that are characterized by their reliance on configuration management, peer reviews and quality assurance significantly increase the probability of a successful acquisition, and have been proven to assist leading organizations in identifying opportunities for software process improvement.  As a result, areas of risk can be identified early and actions can be taken to control them.

Figure 4 highlights a disciplined knowledge-based software development process that can serve as a basis for software acquisition process improvement, describing the information needed, the relevant activities and the deliverables for each phase of the process.

 

Figure 4:  Highlights of the Knowledge-Based Software Development Process [GAO, 2004]

 

Requirements need to be managed and controlled before design work begins and all lower-level design elements should be adequately defined before the start of coding.  It is critical that the software acquirer and developer work closely together and have open and honest discussions.  Requirements management should be viewed as one of the most critical development tasks for ensuring a successful acquisition.  Requirements should be sufficiently documented and validated prior to preliminary design.  In addition, acquirer/developer stakeholders must use sound systems engineering techniques to establish the software requirements, and aggressively manage and control requirements changes.  An Integrated Product Team (IPT) comprised of both acquirers and developers needs to categorize the comprehensive list of requirements as to how critical they are in meeting performance objectives.  Ultimately, the list of requirements will need to be negotiated, i.e., which requirements should be delayed or deleted on the basis of resource or schedule goals.  Once negotiated, the acquirer and developer must agree on a requirements baseline that details cost, schedule, performance and quality goals that are expected to be achieved.  The software development team then develops, allocates and documents lower-level requirements using systems engineering knowledge and based on discussions with the acquirer.  Finally, a requirements check is performed on all requirements-related documents.

A structured process for establishing a stable design ensures that all requirements are addressed, and that all software components and interfaces are defined.  Management and design reviews are important steps in this process.  A Preliminary Design Review (PDR) examines design rationale/assumptions to ensure that stated requirements are met.  A Critical Design Review (CDR) examines all design features to ensure that acquirer requirements are met.  Peer reviews are used to detect errors in the software.  The construction of prototypes for the acquirer provides a “hands-on” assessment of progress towards meeting requirements.

To enhance software coding and testing quality, requirements should be well-written and achievable.  By this phase of the acquisition, designs should be sufficiently detailed.  Other processes that are critical to achieving high quality software include peer reviews, documented coding standards, frequent unit testing, access to a reuse library, and the use of program languages that enable code documentation to facilitate future understanding.  Test plans should not be developed until after requirements are considered stable, and there should be a minimum of one test to verify/validate each requirement.

One of the most widely-known and best-structured software acquisition and development processes is the SEI’s CMMI^® series of processes.  A very recent addition to this suite of tools is the CMMI Acquisition Module (CMMI-AM^®) [CMMI-AM, 2004].  The focus of the CMMI is on “what” should be done, not “how” it should be done, so it allows a great deal of latitude in terms of implementation of the CMMI Key Process Areas.  As with the other CMMI documents, a primary emphasis of the Acquisition Module is on Integrated Product and Process Development (IPPD) concepts.  These include the effective use of cross-functional or multidisciplinary teams; leadership commitment (particularly of senior management); appropriate allocation/delegation of decision-making authority; definition of organizational structures that reward team performance; the design of downstream processes (transition to Operations and Support – O&S) during the acquisition; a strong focus on customer needs throughout the software life-cycle; timely and effective collaboration between all relevant stakeholders; the continuous and proactive identification and management of risk; and a focus on the measurement and improvement of processes to develop/deliver a software product.

A graphical representation of the CMMI-AM^® process is shown in Figure 5.  Following that figure is Table 2, which includes checklist-type questions that can be used to self-assess coverage of each defined CMMI-AM^® process area.  The reader is referred to the CMMI-AM^® document to get a more in-depth discussion of each of the process areas defined in Figure 5 and Table 2.

Figure 5:  Graphical Representation of the CMMI-AM^® Process

 

Table 2:  CMMI^®-AM Acquisition Process Areas [CMMI-AM, 2004]

Process Area	Questions
Organizational Environment for Integration	§   Is infrastructure provided to establish fully functional teams from among all stakeholders? §   Does the acquisition project have two focuses on integration – technical and organizational? §   Are technical capabilities/requirements examined/understood as they relate to overall project goals/objectives? §   Do organizational project entities operate with a single vision and cooperative purpose? §   Do project members understand the interrelationships of all project aspects and how they relate to specific goals/objectives?
Integrated Acquisition Project Management	§   Are project management processes consistent with/tailored from the organization’s standard processes? §   If a standard set of processes does not exist, has the project defined its own processes to meet project needs? §   Is there an integrated management process that incorporates/involves all stakeholders? §   Is the project management process defined in an overall Project Management Plan, or equivalent document? §   Are formal interfaces used among stakeholders, in the form of Memorandums of Understanding (MOUs), Memorandums of Agreement (MOAs), contractual commitments, associate contractor agreements, and similar documents, depending on the nature of the interfaces and involved stakeholders? §   Does the project’s defined process include all life-cycle processes (including IPPD) that are applied by the project? §   Do the project’s defined processes include (1) selecting the team structure, (2) allocating personnel resources, (3) implementing cross-integrated team communication, and (4) conducting issue-resolution processes? §   Does development of a comprehensive project plan or the nature of the project’s defined process require an iterative effort due to a complex, multilayered, integrated team structure?
Acquisition Project Planning	§   Does planning start by setting the acquisition project strategy, then planning the acquisition process in increasing levels of detail? §   Does the acquisition project review potential suppliers’ planning processes for adequacy and compliance? §   Are potential suppliers’ plans reviewed for consistency with system acquisition plans? §   If the acquisition project involves replacement of an existing system, has operation/disposal of the existing system been considered as part of the acquisition planning for the new system? §   Are transition activities included in the acquisition project plan? §   For integrated teams: §   Does acquisition project data include that developed/used solely by a particular team, as well as data applicable across integrated team boundaries? §   Does acquisition project resource planning consider staffing of the integrated teams? §   Is stakeholder involvement planned down to the integrated team level? §   Is special attention paid to resource commitments? §   Do integrated team plans get buy-in from team members, interfacing teams, the acquisition project, and the acquisition process owners whose standard processes are selected for tailoring?
Solicitation & Contract Monitoring	§   Does the project comply with applicable federal, departmental and service acquisition regulations and policies? §   Does the solicitation address issues appropriate to the system domain or acquisition environment (e.g., supplier process evaluations, operational safety suitability/effectiveness, safety, certifications, architectural evaluations and interoperability)? §   Are representatives responsible for functional disciplines within the acquisition project or stakeholder organizations consulted for appropriate inclusion of those disciplines into the solicitation and contract monitoring process? §   When integrated teams are formed, is team membership negotiated with suppliers and incorporated into an agreement? §   Do teaming agreements identify integrated decision-making, reporting requirements (business and technical) and trade studies requiring supplier involvement? §   Are supplier efforts orchestrated to support the acquisition project’s IPPD efforts?
Requirements Management	§   Does requirements management include the direct management of acquirer-controlled requirements and oversight of supplier requirements management? §   Are requirements managed/maintained such that changes are not implemented without assessing the impact on the project? §   Does the requirements management process define “approved” requirements providers and an approved path by which requirements are provided to the supplier? §   Are suppliers prevented from receiving requirements changes from unauthorized sources that are outside the flow of the acquirer’s established requirements management process? §   Are commitments to requirements by acquisition project participants demonstrated by having coordinated/approved documents that define requirements? §   Is each change to a controlled requirement assessed for impact on acquisition project performance, cost and schedule baselines, and project risk? §   Are existing cost, schedule and performance baselines changed, as required, to accommodate a requirements change?
Requirements Development	§   Are customer requirements grouped/coordinated into a set of requirements that will define the scope/direction of the acquisition? §   Do customer requirements and additional acquirer requirements become the basis for the processes used by the supplier’s organization? §   Do requirements flow from the stakeholders/customer to the system level, to multiple subsystem levels and, eventually, to software component levels? §   Is the responsibility for requirements development/flowdown split between the acquirer and developer, and who is responsible for which levels? §   Is there an iterative process of requirements allocation, high-level design and requirements definition for the next lower level? §   Does the acquirer exercise responsibility for defining/baselining the requirements levels under their control? §   Does the acquirer exercise responsibility for monitoring the supplier’s definition of lower-level requirements, and  reviewing/ensuring that required work products are being developed.? §   Does the acquirer exercise responsibility for monitoring the lower-level requirements development process to ensure that requirements produced at each level result in a system that satisfies customer/operational requirements? §   In addition to functional/performance requirements, are interface requirements also covered? §   Do requirements also include non-functional requirements such as data rights, delivery dates, milestone exit criteria, and attributes such as evolvability, maintainability and reusability? §   Is the same requirements process used for acquiring services instead of products?
Integrated Teaming	§   Does integrated teaming consider the overall scope of/requirement for participation of stakeholders from users; acquisition executives; acquisition organizations; developers (primes, subcontractors, suppliers, vendors); test organizations; and other support organizations? §   Should the team adopt a common process for team operation or rely on each team member to use his/her own organization’s processes? §   Are various team member processes compatible at the interface points of the processes? §   Do life-cycle support considerations drive the selection of processes across the team members (tools, support data commonality/compatibility)? §   Are integrated information infrastructures needed to facilitate/coordinate within/external to the team, especially for geographically dispersed members/stakeholders?
Validation	§   Is validation performed early and continuously throughout the acquisition life-cycle? §   Are validation processes used to demonstrate that acquisition process work products fulfill the acquisition strategy, and that the processes will successfully acquire products/services? §   Are validation processes used to ensure that received products/services fulfill their intended use? §   Is the test community treated as a major stakeholder in the validation process, beginning with up-front planning through final system validation? §   Has the acquisition project defined the degree to which validation is required, both early in the acquisition project definition and when products/systems are received? §   Do acquisition project plans identify adequate resources to execute validation activities?
Verification	§   Is verification performed early and continuously throughout the acquisition life-cycle? §   Does the acquisition project ensure that selected work products meet project requirements? §   Does the acquisition project ensure that the evolving acquired products satisfy contractual requirements?
Transition to Operations & Support	§   Does acquisition planning for transition include establishing support strategies through organic support infrastructures, contractor logistics support, or other sources? §   Are the roles/responsibilities of the acquirer, supporter and user defined in the life-cycle support of the system? §   Has the acquirer determined whether it will execute the transition function directly, or as a result of the acquisition itself? §   Are responsibilities for capability enhancements during the support phase defined, taking into account the magnitude/complexity of the envisioned change? §   Is the organization responsible for support (i.e.,” level 1 maintenance”) and enhancements (i.e., “level 2 maintenance”) explicitly identified? §   Has the acquisition project assigned responsibility for implementation of process improvement practices? §   Is the acquisition project working with operational units to plan for product transition into operational use and eventual disposal of the product (or technical/functional obsolescence of the software)?
Project Monitoring & Control	§   Are monitoring/control functions implemented in the acquisition project as acquisition planning is performed and the acquisition strategy is defined? §   Does the acquisition project have internal processes, plans and work products that should be monitored for progress/satisfactory completion? §   Do the monitored work products include specifications, plans, Request for Proposal components, etc.? §   Do monitored items include staffing levels/qualifications, system performance objectives/thresholds, infrastructure readiness (tools, networks, etc.) and other project planning activities/products? §   Is acquisition project risk actively identified and mitigated? §   Is corrective action applied when execution does not match acquisition project planning (e.g., internal staffing, project plan completion dates, draft/final solicitation & contract award milestone slips)? §   Are corrective actions required to resolve variances from acquisition project plans tracked to closure? §   After supplier selection/contract award, is monitoring and control still applied to the internal acquisition project §   After supplier selection/contract award, does monitoring and control also cover supplier performance against the supplier’s project plan?
Acquisition Process & Product Quality Assurance	§   Have the acquisition project products and processes (e.g., solicitation packages, etc.) been evaluated? §   Has the solicitation package been developed per the standard/format agreed to by the project team? §   Does the solicitation package conform to all applicable policies and laws? §   Does the acquisition project’s risk management process conform to that called out for in the risk management plan?
Risk Management	§   Is the acquisition strategy influenced by risk identification and estimation of risk probability/impact? §   Does the acquirer assess/manage overall acquisition project risks over the duration of the project? §   Does the acquirer assess/manage risks associated with supplier performance? §   For acquisitions using integrated teams, are risks associated with loss of inter- or intra-team coordination considered?
Configuration Management	§   Are acquisition work products (e.g., solicitation packages) placed under internal CM control? §   Are interim/final primary/subordinate supplier work products monitored to ensure that project goals are met? §   Are provisions made for conducting CM of supplier products/documentation? §   Are methods established/maintained to ensure that acquisition project data is complete/consistent? §   Has the acquisition project determined which work products should be CM-controlled?
Decision Analysis & Resolution	§   Is there a repeatable, criteria-based decision-making process used for making critical decisions that define/guide the acquisition process itself? §   Is there a repeatable, criteria-based decision-making process used for making critical decisions with the selected supplier? §   Does the process for decision making document the acquisition project decision rationale? §   Can criteria for critical decisions be revisited when changes/technology insertion decisions impacting essential requirements are considered?
Measurement & Analysis	§   Does the acquisition project have the information it needs for determining status of its activities throughout the acquisition life-cycle, the supplier’s activities per contractual requirements and the status of the evolving products acquired? §   In acquisition projects requiring multiple products or teaming relationships, is additional information needed/available to ensure programmatic, technical and operational interoperability objectives are identified/measured/achieved?

The use of checklists such as the one above helps maintain discipline and structure in the software acquisition process, and can cover any phase of a software acquisition life-cycle.  Boehm’s presentation “Early Warning Indicators in the Acquisition of Software-Intensive Systems” [Boehm, 2004] includes a limited checklist for assessing detailed software cost and schedule estimates, as well as an acquisition strategy checklist.  More detailed and comprehensive checklists and documentation templates are found in standards, handbooks, etc..  For example, IEEE Standard 1062 – 1998 Edition [IEEE, 1998] contains a template for an Acquisition Plan (AP).  It also contains separate, extensive checklists that govern twelve different acquisition categories ranging from Organizational Strategy through Software Acceptance (see Table 3).  These are the same checklists that are referred to back in Figure 1, which covered the 9 major steps of the IEEE Standard 1062 software acquisition process.  The IEEE Standard also makes reference to IEEE /EIA 12207.0-1996 and ISO/IEC 12207 which contain an Evidence Product Checklist for identifying the Procedures, Plans, Records, Documents and Audits/Reviews that should be expected at each step of the software acquisition.  Table 4 provides a summary of these work products.

 

Table 3:  IEEE Std 1062, 1998 Edition Software Acquisition Checklists [IEEE, 1998]

 

 

Table 4:  Evidence Product Checklist for ISO/IEC Standard 12207 Software Life-cycle Processes (Acquisition Phase Only) [ISO 12207, 2002]

Clause No./ Name	Procedures	Plans	Records	Documents	Audits/ Reviews
5.1.1 Initiation	·       Acquisition Plan ·       Concept Document ·       Purchasing Requirements Document ·       Purchasing Specification Document ·       Software & System Acceptance Document ·       Software & System Acceptance Plan ·       Software & System Acceptance ·       Supplier Selection Plan ·       User Requirements Document	·       Acquisition ·       Software and System Acceptance ·       Supplier Selection	·       Acquisition ·       Requirements Used to Control Risk ·       Residual Risk	·       Concept ·       Purchasing Requirements ·       Purchasing Specification ·       Software & System Acceptance ·       User Requirements	·       Acquisition Plan ·       Concept Document ·       Purchasing Requirements Document ·       Purchasing Specification Document ·       Risk Management File Audit ·       Software & System Acceptance Plan ·       Software & System Acceptance Document ·       Supplier Selection Plan ·       User Requirements Document
5.1.2 RFP Preparation	·       Contract Milestone Document ·       Request for Proposal (RFP)			·       Contract Milestone ·       Request for Proposal (RFP)	·       Contract Milestone Document ·       Request for Proposal (RFP)
5.1.3 Contract Preparation & Update	·       Contract ·       Supplier Selection		·       Contract  Change	·       Contract	·       Contract
5.1.4 Supplier Monitoring					·       Joint Supplier Audits & Reviews
5.1.5 Acceptance & Completion	·       Customer-Supplied Software & Data Products		·       Customer-Supplied Software & Data Products

Finally, the use of checklists can benefit specific monitoring and oversight activities of software acquisition such as risk management.  Gallagher [Gallagher, 1997] defines a software acquisition risk management Key Process Area (KPA) that can be put into a high-level checklist form (see Table 5) in order to assess whether Goals, Activities, Commitments, etc., associated with risk management have been covered in the software acquisition process.

Table 5.  Software Acquisition Risk Management KPA [Gallagher, 1997]

Category	Goal	Description	Check
Goals	1	SA risk management is an integral part of the project’s defined SA process	______
	2	The project identifies/deals with risk in a positive manner, such that identification is recognized/rewarded, and results in effective risk handling	______
Activities	1	SA risk management activities are integrated into SA planning	______
	2	The Software Acquisition Risk Management Plan is developed in accordance with the project’s defined SA process	______
	3	The project team performs its SA risk management activities in accordance with documented plans	______
	4	Risk management is conducted as an integral part of the solicitation, project performance management, and contract performance management processes	______
	5	SA risk handling actions are tracked and controlled until the risks are mitigated	______
Commitments	1	The acquisition organization has a written policy for the management of SA risk	______
	2	Responsibility for SA risk management activities is designated	______
Abilities	1	A group that is responsible for coordinating SA risk management activities exists	______
	2	Adequate resources are provided for SA risk management activities	______
	3	Individuals performing SA risk management activities have experience or receive required training	______
Measurement	1	Measurements are made/used to determine status of the acquisition risk management activities and resultant products	______
Verification	1	Acquisition risk management activities are reviewed by acquisition organization management on a periodic basis	______
	2	Acquisition risk management activities are reviewed by the project manager on both a periodic and event-driven basis	______

Collecting/Analyzing Meaningful Metrics:

Meaningful metrics are collected and analyzed by both software acquirers and developers in order to track progress, confirm knowledge, manage risk, improve processes and ensure that all stakeholders are well-informed.  Meaningful and suitably tailored metrics should cover cost, schedule, software size, requirements, tests performed/completed, number of defects detected/corrected, and quality attributes.  Acquirers can apply some of these same metrics to assess whether the developer will be able to deliver software within cost, schedule, performance and quality constraints.  The use of earned value analysis tools to track cost/schedule and help mitigate risk is an important driver in determining meaningful metrics, supporting the ability of a program to maintain consistent, predictable practices and acceptable process outcomes.  Sharing of metrics between developers and acquirers is an important aspect of a successful software acquisition process.

Leading software developers have been described as relentless in their efforts to collect metrics to improve project processes and results [GAO, 2004].  They typically also establish goals and track metrics for company-wide initiatives, such as cost reduction efforts and customer satisfaction.  These leaders have continually emphasized the critical nature of measuring processes, collecting metrics and using them to improve performance in their workforce through training.  In addition, a good part of their success can be attributed to their use of an Earned Value Management System (EVMS) and a project time-tracking system (to record time spent on “cost of quality” and “cost of poor quality”).  The cost of poor quality is a direct reflection of the relative level of effectiveness of an organization’s software acquisition and/or development processes.  Typical metrics used by leading software developers are indicated in Table 6.

 

 

Table 6:  Metrics Used by Leading Software Developers [GAO, 2004]

Major Metric	Examples	Usefulness
Cost	·   Cost per phase ·   Effort per phase ·   Planned vs. actual cost ·   Cost performance index	·   Products of an earned value management system, indicating progress towards completing software development against the plan ·   Poor cost performance index indicates problems meeting cost goals ·   May need to consider project scope reduction to meet release dates, or canceling the program
Schedule	·   Planned vs. actual delivery dates ·   Schedule estimation accuracy ·   Percentage of project on time ·   Schedule performance index	·   Products of an earned value management system, indicating achieved schedule progress against the plan ·   Used over the development life-cycle to gauge progress toward developing key products or meeting critical milestones ·   Close attention to schedule deviations identifies the ability to meet project goals, and if/when additional resources are needed
Size	·   Amount of new, modified and reused code ·   Size estimation accuracy	·   Used by management to compare amount of code produced vs. estimated ·   Changes to size estimates indicate potential cost/schedule problems
Requirements	·   Total requirements/features committed for delivery ·   Percentage of requirements completed ·   Number of requirements changes by phase	·   Used to assess progress towards meeting acquirer’s performance demands ·   Large number of changes, or late changes, can impact cost/schedule commitments ·   Large number of changes, or late changes, can result in software with higher defect levels ·
Tests	·   Number if tests planned, completed, passed ·   Percent of plan tests completed ·   Percent of planned tests passed	·   Determine extent to which planned software tests have been successfully accomplished ·   Deviation from planned number of tests may indicate inadequate testing and subsequent quality problems, resulting in costly rework in later project phases
Defects	·   Number of defects per phase ·   Phase defect originated vs. phase found ·   Cost to fix defect ·   Severity of defect ·   Total unresolved defects	·   Track software problems to (1) the phase where they were found, (2) the phase where they should have been found and (3) determine the cost to fix ·   Defects found after the phase in which they were created indicate performance problems that may increase cost/schedule due to rework of software and correction of development processes Too few defects found may indicate inadequate test coverage during the test phase, or insufficient formal technical review of documents in the design phase
Quality	·   Cost of quality efforts ·   Cost of poor quality (rework) ·   Number of quality goals missed/achieved ·   Customer satisfaction survey results	·   Provides information on potential reliability of delivered software ·   Indicates the amount of money/time invested in development to assure a given quality level ·   Defects found/fixed in the phase that they occur indicates good process quality ·   Defects found/fixed in downstream phases are costly/time-consuming, and indicate weaknesses in the development process that will require corrective action

 

 

Some Suggested Software Acquisition “Best” Practices?

Although not yet defined in the context of “Gold” practices, there are a number of software acquisition “best” practices that can be considered as an integral part of the three fundamental software acquisition strategies identified earlier.  Eslinger and Adams [Adams, 2004a], both from the Aerospace Corporation, have suggested some candidates for consideration.  Table 7 summarizes the results from the [Adams, 2004a] reference.  Note the implied emphasis on systems engineering knowledge.

Table 7:  Recommended Software Acquisition Best Practices [Adams, 2004a]

Practice	Comments
Best Practices for Defining the Program Life-cycle
Use a Software-Friendly Acquisition Model	·  Evolutionary acquisition is more suited to large, complex software-intensive systems
Choose Software-Friendly Points in the Life-cycle for Contract Actions	·  Avoid contract actions in the middle of software development spirals ·  Develop a firm basis for software costing before Milestone B
Tailor the Acquisition Model	·  System Design Review level of maturity before Milestone B ·  Selection of a single contractor at appropriate point in the software development life-cycle ·  With or without a production phase
Best Practices for Developing the Initial Government Architecture Concepts
Perform Software-Inclusive Architecture Trade Studies	·  With system architecture trades ·  Identify/address critical HW/SW architecture issues ·  Include major legacy components and COTS software
Include Software in Evaluation of Architecture Concepts	·  Evaluate software evolution and growth capability ·  Include software in life-cycle cost analysis (COTS software refresh, legacy and new software re-engineering/maintenance
Select a Set of Integrated HW/SW Architecture Concepts	·  Able to grow with each successive evolution with little expected rework ·  Able to integrate each successive evolution with previous evolutions (and legacy system, as applicable
Best Practices for Developing the Initial Government Cost/Schedule Baseline
Determine Realistic SW Size Estimates for Each Evolution	·  Use Gov’t HW/SW architecture concept ·  Include all SW functionality and infrastructure needed ·  Use historical data from similar past programs & early concept study data
Determine Realistic SW Effort & Cost Estimates for Each Evolution	·  Include COTS, reuse and newly-developed software ·  Include tasks not reflected in cost models (e.g., integration of SW components costed separately, COTS)
Determine Realistic SW Schedule Estimates for Each Evolution	·  Include all software effort in schedule ·  Never compress software schedule >20% off nominal
Best Practices for Defining Executable Program Evolutions
Consider SW Implications When Defining Evolution Capabilities	·  Analyze feasibility of developing the required software for each evolution ·  Analyze feasibility of integrating the software in each evolution with all previous evolutions (and legacy systems), as applicable ·  Analyze impacts of COTS software refresh and legacy software upgrades
Consider SW Implications When Defining Evolution Schedules	·  Analyze feasibility of overlapping software development schedules for closely-spaced evolutions ·  Avoid plans that require developing subsequent evolutions on unknown software baselines ·  Analyze feasibility of COTS refresh and legacy software upgrade schedules
Best Practices for Developing the Global Acquisition Strategy
Develop Plans for Computer System Technology Insertion	·  Include COTS HW and SW refresh in each successive evolution ·  Understand new computer HW and SW technologies needed for each evolution and study their readiness
Develop Plans for Software Support	·  Plan for managing multiple baselines (operations and development) ·  Plan for integrating software maintenance actions on operational evolutions into evolutions under development
Develop Plans for Evaluation of Contractor Software Capability	·  Perform a Government evaluation of contractor team software capability ·  Prior to or part of selection of a single development contractor
Best Practices for Establishing the Requirements Baseline
Include Software in Government System Performance Requirements	·  Specialty engineering, particularly reliability, maintainability and availability ·  Key Performance Parameters (KPP) ·  Open system architecture ·  Design for evolution and growth
Contract for Delivery of SW-Intensive Requirements Specifications	·  Require System and Segment Specifications as CDRL items ·  Use System/Subsystem Specification DID (DI-IPSC-81431a)
Best Practices for Developing the System Architectural Design
Contract for Software Architecture Trade Studies	·  Integrate with system architecture trades ·  Include major software legacy components and COTS software
Contract for Delivery of System Architecture	·  Require system architecture as a CDRL item ·  Require an integrated HW/SW architecture, defined by multiple architecture views ·  Include newly-developed, reuse and COTS software
Best Practices for Reducing Software Development Risk
Contract for Software Product Risk Reduction	·  Studies/prototyping of high risk areas for software, e.g., mission processing algorithms and mission planning concepts ·  Simulation development ·  Increase readiness level of computer HW and SW technologies
Contract for Software Process Risk Reduction	·  Require delivery of Software Development Plan (DID DI-IPSC-81427a) ·  Require compliance with robust software development standard ·  Enable contractor team to prepare for software capability evaluation
Best Practices for Managing the Phase A/B Contract
Ensure Contractor(s) Define Software-Inclusive Requirements Specifications	·  Software systems engineers (Government and contractors) must participate with Government and contractor systems engineers
Ensure Contractor(s) Define HW/SW Architecture	·  Software systems engineers (Government and contractors) must participate with Government and contractor systems engineers
Participate with Contractor in Software Risk Reduction	·  Government software acquisition personnel with technical expertise in software product and process engineering must participate
Best Practices for Updating the Global Acquisition Strategy
Update the SW-Inclusive Program Baseline	·  Software-inclusive system requirements ·  Integrated HW/SW architecture ·  Realistic software size, effort, cost and schedule estimates for each evolution
Update Definition of SW-Friendly Evolutions	·  Evolution capabilities, schedules and integration strategies ·  COTS software refresh and legacy software upgrades
Update Software-Specific Plans	·  Software support strategy ·  Contractor team software capability evaluations ·  Software technology insertion ·  Software transition to operations
Best Practices Spanning the Acquisition Life-cycle
Software Acquisition Risk Management	·  Continuous, across the entire acquisition life-cycle; across all evolutions; within each ongoing evolution ·  Program-level risk management and contractor development risk management are necessary, but not sufficient
Software Systems Acquisition	·  Integrate software acquisition with the system acquisition process, from capability needs identification through system retirement, especially during early acquisition life-cycle phases

 

The second reference [Adams, 2004b] defines lessons learned and best practices for the acquisition of COTS-based software systems used in space systems applications.  They are generally applicable, however, to all software-intensive systems, regardless of application domain.  The suggested best practices are summarized in Table 8.  The lessons learned are paraphrased below:

·          Critical aspects of COTS-based software development/sustainment are out of the control of the customer, developer and user (suppliers are market driven and DoD is not the market; supplier strategies/market position are subject to change)

·          Full application of system/software engineering is required throughout the COTS-based system/software life-cycle

·          COTS-based software system development/sustainment requires a close, active and continuous partnership  among the customer, developer and user

·          Every COTS-based software system requires continuous evolution through development/sustainment

·          Current processes must typically be adapted for COTS-based software system acquisition, development and sustainment

·          Actual cost/schedule savings associated with COTS-base software systems development/sustainment are typically overstated

 

Table 8:  Recommended COTS-Based Software System Acquisition
Best Practices [Adams, 2004b]

Practice	Comments
Best Practices for Establishing a Program Baseline
Perform Software Architecture Trade Studies	·  Perform with system architecture trades ·  Include major COTS and legacy components ·  Supports Government software architecture baseline selection ·  Include user in all trades
Determine Realistic, Independent Baseline Software Estimates	·  Size, effort, cost and schedule ·  COTS, reuse and newly-developed ·  Include tasks not reflected in cost models ·  Include COTS refresh through both development and sustainment
Include Software in Systems Performance Requirements	·  Prioritized requirements ·  COTS software support requirements ·  Specialty engineering (reliability, maintainability & availability (RMA), security, safety) ·  Key Performance Parameters (KPP) ·  Open system architecture
Best Practices for Obtaining Contractual Insight
Require Key Software Technical and Management Deliverables	·  Highest risk reduction potential is in plans; requirements and architecture; test plans, procedures and reports; metrics reports; delivery, installation and O&M documentation ·  Use electronic delivery
Require Timely Electronic Access to All Software Products	·  COTS evaluation trade studies ·  Intermediate and final products (requirements, architecture and design; implementation (including code); integration and verification testing)
Require Software-Level Technical and Management Reviews	·  In addition to system reviews ·  Include COTS software experts in reviews
Best Practices for Obtaining Contractual Commitment
Mandate Compliance with Robust Commercial Standards	·  E.g., EIA/IEEE J-STD-016 (commercialized version of MIL-STD-498) ·  Tailor standard for COTS-based software system development
Require Contractor Commitment to Software Development Plan (SDP)	·  Require SDP to include processes for COTS software ·  Require Integrated Management Plan (IMP) to have adequate system engineering and sustainment for COTS ·  Include commitment to SDP in IMP
Best Practices for Providing Tools for Contract Management
Incentivize Software Quality (Not Just Cost/Schedule)	·  Use award and incentive fee plans ·  Reward adherence to defined software processes and software process improvement ·  Reward timely/acceptable response to Government comments ·  Reward low rework rates ·  Reward meeting performance requirements  (e.g., RMA) post delivery/launch ·  Reward architecture development that supports COTS-based software system evolution
Mandate Periodic Team Software Capability Appraisals	·  Relate results and improvement actions directly to award fees ·  Explicitly include COTS processes in appraisals
Best Practices for Selecting a Capable Software Contractor Team
Evaluate Software Capability/Processes of Offeror Teams	·  Individual team member evaluation is insufficient ·  Evaluate software capability as a separate subfactor under the Mission Capability factor ·  Weight according to software risk
Evaluate Teams’ Proposed Software and Related Processes	·  Corporate and past project process evaluation is insufficient ·  Include COTS software, systems engineering and logistics processes
Evaluate Realism of Cost and Schedule Bids	·  Be suspicious of productivity extremes, COTS, reuse, low lines of code, and short integration times ·  Ensure that all COTS software tasks are included in the cost and schedule bid ·  Ensure bid contains sufficient cost and schedule margin
Evaluate Software and Hardware Architecture with System Design
Best Practices for Performing Technical Product Review
Focus Technical Review Resources on Areas of Highest Risk	·  IPTs, Technical Interchange Meetings (TIMs), working groups, per reviews, etc. ·  Software-level technical reviews ·  High risk/critical software products (including COTS software) ·  Key software technical deliverables
Include Users/Operators in All Technical Review Activities	·  Ensure users/operators understand the evolving design, including COTS software capabilities and impacts on O&M
Monitor Software Integration and Verification Adequacy	·  Begin at the build level ·  Focus on areas of highest risk ·  Focus on early performance analysis results and meeting KPPs ·  Ensure that COTS software performance is measured ·  Ensure requirements allocated to COTS software are verified
Best Practices for Performing Software Process Review
Review Effectiveness of Team’s Defined Software and Related Processes	·  Identify process deficiencies (especially across team boundaries and with COTS products) ·  Assist with process improvement ·  Individual Level 2 and 3 CMMI/CMM compliance may not be sufficient
Perform Periodic Team Software Capability Appraisals	·  During contract performance ·  Support for significant program or award fee milestones ·  Explicitly include COTS processes
Review Team’s Adherence to Defined Software and Related Processes	·  Identify adherence deficiencies ·  Assist in deficiency correction
Best Practices for Managing the Contract
Use Incentive/Award Fees Aggressively	·  Motivate good software and related practices ·  Focus on quality an d architecture
Apply Proactive Quantitative Management	·  Ensure a comprehensive software/systems metrics program balanced across information categories (include leading quality indicators, e.g., rework) ·  Perform cross-metric analysis ·  Earned value alone is insufficient
Ensure Adherence to Software-Inclusive Requirements	·  Especially RMA, safety and security ·  Especially COTS software supportability
Perform Periodic Independent Assessments	·  Support for significant program or award fee milestones ·  Act aggressively on findings
Best Practices That Span the Life-cycle
Software Acquisition Risk Management	·  Continuous software acquisition risk management across all acquisition organization levels ·  Program-level risk management and contractor development risk management are necessary, but not sufficient ·  Establish management reserves consistent with software risks (including/especially COTS software risks)
Software Systems Acquisition	·  Integrate software acquisition with the system acquisition process (from mission needs identification through system retirement, especially during pre-contract activities)

 

 

Selected Software Acquisition Process Improvement Techniques

There are literally dozens of tools and resources that are available to support software acquisition process activities.  Many of these are described and linked in the Resources section of this document and relate to tools, methods, etc., that are generally familiar to (and used by) the software acquisition community.  These include Earned Value Management Systems, Risk/Configuration Management tools, acquisition document templates, and so on.  There are three techniques that will be briefly mentioned here, however, that are not typically associated with or recognized by the software acquisition community or supported by the literature in the context of software acquisition.  They may potentially provide some future benefit (albeit with some risk) for improving the software acquisition process for those that wish to research and adapt them as a software acquisition process improvement tool.  They are:

·          Quality Function Deployment (QFD)

·          Analytic Hierarchy Process (AHP)

·          Phase Containment Matrix

Each technique will be briefly introduced and discussed in the context of its potential role in software acquisition process improvement, but a detailed technical explanation of how each technique works is beyond the scope of this document.  The reader is encouraged to visit the sites indicated in the Resources section to gain further insight into these methods.

 

Quality Function Deployment:

Quality Function Deployment (QFD) is a technique that is most closely associated with the Total Quality Management concepts of the last 20 years.  QFD is typically structured around a series of interdependent matrices, each of which is commonly referred to as a “House of Quality”.  For each stage of the design and production process, engineers identify technical performance measures that will be used to meet itemized customer needs/expectations.  They then specify corresponding threshold values to be met in order to achieve overall system performance requirements.  The QFD process, therefore, establishes the minimum levels of performance required to satisfy customer needs.  [Wollover, 1997]

QFD has been successfully demonstrated in a variety of industries as a means to guide process development using technical performance measures to systematically organize and prioritize all independent variables (cost, etc.) and their interrelationships.  Traditional QFD consists of six general steps: (1) identify and analyze customer needs and requirements (WHATs), (2) identify technical requirements (HOWs), (3) benchmark the technical requirements (WHATs vs. HOWs), (4) assign rank/priority to customer requirements (WHYs), (5) develop the corresponding House of Quality (6) establish technical requirement metrics that inter/intra-relate to specific customer requirements (HOW MUCHs), and (7) evolve technical performance measures into the next-phase design requirements (define the next House of Quality, with “Technical Requirements” = WHATs and “Design Requirements” = HOWs).  Figure 6 is a generic representation of one iteration through this process.

Figure 6:  Traditional Quality Function Deployment to Translate Customer Requirements Into
Robust Technical Requirements

What is suggested in this Gold Practice document is a complementary QFD process that, beginning with the same list of customer requirements, maps the software acquisition process to the development of an acquisition strategy/plan, the feasibility of implementing that strategy based on available resources, and the benefits/risks associated with the acquisition based on those resources and a corresponding set of available support tools/techniques.  This approach, graphically represented by Figure 7, would potentially define an optimal level of software acquisition support required to meet customer requirements.

Figure 7:  Use Quality Function Deployment to Translate Customer Requirements Into a Successful Acquisition

Analytic Hierarchy Process:

The Analytic Hierarchy Process (AHP) is a technique to combine information which can be used to prioritize customer needs in QFD, make budget decisions based on a number of explicit or implicit strategic goals, manage conflicting stakeholders, or select from a large number of potential alternatives. The AHP is predicated on the decomposition of multiple-criteria decision-making problems into a mathematically-based criteria hierarchy.  At each level in the hierarchy, the relative importance of factors is assessed by comparing them in pairs.  Finally, the pair-wise alternatives are compared with respect to the defined criteria [Kontio, 1996].  The technique has been successfully used in several fields, including software engineering and software selection, and has been reported as being effective in a number of case studies and experiments.  There are also commercially available tools (such as Expert Choice^®) that provide for the entering of judgments, performing all of the necessary calculations, and generating supporting rationale.  The output of AHP helps software acquirers, developers and managers make informed decisions.  

The Analytic Hierarchy Process is a 4-step selection process that is comprised of:

1.   Decisions regarding selection criteria

2.   Rating of relative criteria importance using mathematical pair-wise comparisons

3.   Rating of each potential selection relative to each other selection based on the Step 1 criteria (again using pair-wise comparisons)

4.   Combining the two ratings to obtain a normalized relative rating for each potential selection

The mathematical theory behind the AHP is beyond the scope of this document.  Within the context that it is presented here (software acquisition and development processes), an example of how AHP would be applied is as follows:

·          Use Quality Function Deployment (QFD) to produce a matrix with required software functionality as one dimension (i.e., customer needs/requirements) and possible COTS software suppliers as the other.

·          Use Analytic Hierarchy Programming (AHP) to establish a hierarchical rating of the potential suppliers

·          If necessary, use Linear Programming techniques to optimize the allocation of customer needs/requirements given the relative rating and availability of suppliers

Phase Containment Matrix:

A Phase Containment Matrix is a well-known tool for identifying and reducing software defects across the various phases of a software life-cycle.  It is constructed by identifying the phases during which a defect is inserted as the column headings, and the phases where the defect was detected as the row headings (see Figure 8). 

Figure 8:  Phase Containment Matrix for Tracking Software Development Quality

 

Within each cell, the number of defects that are detected at that phase in the software life-cycle are entered.  There is a basic, yet critical, assumption that root-cause defect analysis has been performed in order to accurately determine in which phase of the life-cycle the defect was actually inserted.

In-phase defects are often not counted as rework, as these defects are assumed to be a natural part of the process.  For example, the rework required to correct the 25 defects that were introduced during the requirements phase, but were also detected and eliminated in the requirements phase, would not be counted, even though there is still potential cost and schedule impact associated with the rework.  Ignoring the cost of in-phase rework deludes both the customer and the organization, and deprives the organization of an opportunity to improve its in-phase processes.  Analyzing the cost of performing rework on 25 in-phase requirements defects might provide significant justification for improving an acquisition or development  process to reduce the number of introduced defects.

Out-of-phase defects are generally classified as rework and, just as generally, are going to cost more to fix than in-phase defects.  The larger the span (in project phases) between the insertion point and the detection point, the more expensive it is going to be, on a per defect basis, to rework and fix it.  This “more expensive” factor can be an order of magnitude of cost between each phase.

The concept of a Phase Containment Matrix could, with some adaptation, also be applied to improve software acquisition processes by defining an appropriate set of processes within which software acquisition defects can be detected.  Figure 9 provides a simple example of how this might be accomplished.

Figure 9:  Process Containment Matrix for Tracking Software Acquisition Quality

The interpretation of the results of this phase containment matrix follows the same rationale as the software development life-cycle phase containment matrix.  For example, the 20 defects inserted and subsequently detected within the Acquisition Plan/Strategy process represents an opportunity to improve the software acquisition process for a relatively low cost and schedule impact, if root defect causes are identified and corrected.  The impact of introducing 10 acquisition planning/strategy defects that are not detected and corrected until the tracking/oversight phase of the software acquisition may be significantly more profound.  Again, the introduction and detection of software acquisition defects translates into rework cost and schedule impacts, regardless of whether the defects occur “in-phase” or “out-of-phase”, so a basic paradigm change, coupled with strong senior management and stakeholder commitment, is necessary in order for the technique to be successful.

 

 

CHARACTERISTICS OF IMPLEMENTATION:

 

SUMMARY CHARACTERISTICS

Enabling Practices:            Link to Interrelationships Diagram Enabled Practices:             Link to Interrelationships Diagram Impact Areas:                    Primary: Quality  Secondary: All other areas Life-cycle Phase:               Primarily in system development and production and deployment phases Scope/Authority:                Organizational- or programmatic-level Applicability:                     Primarily directed towards the acquisition organizations, but has definite impact on the developer Use Indicators:                   Continued poor performance of acquisition programs (missed milestones, breaching baselines, delivering unacceptable products) Use Inhibitors:                    Mature organizations that consistently meet proposed managerial/technical targets Appropriate Programs:      Relatively large programs; a number of concurrent small programs; projects with marginal staffing Inappropriate Programs:   Schedules, outcomes or expectations are easily achieved (e.g., R&D); programs with no permanent organization (e.g., tiger teams) Barriers:                             Lack of accountability for acquisition excellence; cultural resistance to improvement processes Facilitators:                        Policy, education and management champions at all levels; availability of documented, demonstrable results

 

 

ANTICIPATED BENEFITS OF IMPLEMENTATION:

 

Successful implementation of software acquisition process improvement can result in:

§         Realistic, incremental and continuous improvements to software acquisitions – A process of evolutionary software acquisition avoids the temptation to succumb to unrealistic expectations in an attempt to force revolutionary changes in software development.

§         Adherence to well-understood, well-defined, manageable requirements – An evolutionary software acquisition process ensures that software development is limited to things that are possible to manage.  Only requirements that are well-defined and well-understood can be adequately managed.

§         Following of disciplined, structured software management and development processes that provide predictable results – Disciplined, structured software management and development process phases that end in a management review/gate ensure that the acquisition will remain on track, and that project course corrections can be identified and implemented effectively.

§         Well-informed, well-disciplined and knowledgeable software acquisition teams – The suitably tailored collection and use of process metrics ensures that the software acquisition is adequately tracked; that project knowledge is confirmed and shared among stakeholders; that acquisition risk is effectively managed; and that software acquirers are firmly integrated into the continuous improvement process.

§         Increased probability of software acquisition successes and “best value” awards – Heavy reliance on configuration management, risk identification/management, peer reviews and quality assurance, coupled with earned value management techniques, helps ensure a successful and improving acquisition process.

 

DETAILED CHARACTERISTICS

 

Key Characteristics of the Acquisition Process Improvement Gold Practice

Characteristic	Comments
Manageable Development Environment is Defined and Implemented	·      Business decisions must be made to invest suitable time/resources in achieving high levels of software process maturity ·      Honest relationships between the acquirer and developer must be established ·      There must be a mutual understanding of software requirements between the acquirer and developer in order to optimize setting/managing requirements ·      There must be a match between available resources and requirements, supported by systems engineering analysis and trade-off analyses that consider the performance, cost and schedule impacts of major changes to software requirements ·      Technical and administrative “knowledge” must be obtained early in the software development process ·      Sufficient requirements knowledge must be demanded by PMs and other stakeholders at key process points ·      In order to ensure that software requirements are appropriately set/managed, it is necessary to document that software requirements are achievable based on systems engineering knowledge
Disciplined Development Processes, Supported by Gated Reviews	·      Processes that meet program needs must be established and adhered to ·      Acquirers should develop a list of tailored systems engineering deliverables that include software based on the results of required engineering activities at the appropriate stages/phases of system development ·      A sufficient amount of software development time should be devoted to requirements-setting activities ·      Well-written and achievable requirements must be developed and managed/controlled before actual software design begins ·      Lower-level software design elements should be defined before any coding begins ·      The number and timing of requirements changes should be aggressively managed ·      Test plans that are developed should be based on a stable set of requirements ·      Ensure that the software design is mature before approving release/production ·      All production processes must be under control before production begins ·      Provide for gated reviews of systems engineering baselines on an event-driven basis
Established and Leveraged Metrics are Defined and Utilized	·      The collection and reporting of metrics related to software acquisition and development should be required internally, and from contractors, in order to ensure adequate oversight knowledge of software-intensive acquisitions ·      There should be relentless pursuit of tailored metrics that are derived from effective processes ·      Require software contractors to collect/report cost, schedule, size, requirements, test, defect and quality metrics on a per-month basis, if appropriate, and before program milestones ·      Ensure that contractors are using an earned value system that reports work at a suitably detailed level ·      Software acquisition should be enforced with suitable controls and performance incentives in DoD acquisition policy, software acquisition improvement plans and software development contracts
Use of Appropriate Tools and Techniques to Support Acquisition Activities	·      Formal processes such as SA-CMM® and CMMI-AM® provide a framework for implementing a structured software acquisition improvement process ·      Tools and techniques to be considered include document templates, documentation standards, checklists, QFD, AHP, Phase Containment Matrices, earned value tracking, knowledge-sharing systems, lessons learned repositories, past performance databases, centralized acquisition resources, software tools that support risk management, configuration management, source selection, decision-making, etc.
Use of Integrated Process Teams (IPTs) That Involve All Stakeholders (including End Users)	·      Integrated Process Teams conduct process analyses and identify bottlenecks and non-value-added process structure and flow, with a constant focus on measurement/improvement of software acquisition processes. ·      IPTs keep the focus on the customer’s needs and requirements ·      Effective IPTs contribute to collaborative requirements development, an important attribute of the Air Force “Agile Acquisition” initiative ·      Teams should be cross-functional and multi-disciplinary, and there must be leadership/senior management commitment to support the team (with appropriate allocation/delegation of decision-making) ·      Organizational structures should be defined such that team performance is suitably rewarded

 

 

RELATIONSHIPS TO OTHER PRACTICES:

 

The Figure below represents a high-level process architecture for the subject practice, depicting relationships among this practice and the nature of the influences on the practice (describing how other practices might relate to this practice).  These relationship statements are based on definitions of specific “best practices” found in the literature and the notion that the successful implementation of practices may “influence” (or be influenced by) the ability to successfully implement other practices.  A brief description of these influences is included in the table below.

Process Architecture for the "Acquisition Process Improvement" Gold Practice™

 

Summary of Relationship Factors

INPUTS TO THE PRACTICE
Establish an evolutionary acquisition environment	A concerted effort must be made to establish a software acquisition environment that lowers project risk and increases the probability of successful software development outcomes.  Performance-Based Specifications provide a basis for directly relating requirements to end-user needs, and should be reviewed at each iteration of the acquisition process.  Commercial Specifications and Standards/Open Systems support the premise of making incremental improvements to the acquisition process, as opposed to setting unrealistic expectations.  As a gauge to determine what expectations may be feasible to achieve, Use Past Performance to assess whether acquisition objectives are feasible, and whether potential suppliers are capable of meeting those objectives.
Use a disciplined development process	Software acquirers and developers alike must commit to disciplined processes in order to make an acquisition successful.  The use of Integrated Process Teams (IPTs) and Integrated Product and Process Development (IPPD) ensure that all stakeholders are knowledgeable and informed.  A disciplined acquisition process should Plan for Technology Insertion.  Processes covering Configuration Management, Formal Risk Management and Defect Tracking Against Quality Targets are needed to maintain the discipline of the acquisition process and the quality of the resulting software product(s).  Each phase of the acquisition should end in management-supported Demonstration-Based Reviews to ensure that the project remains on track, and that the acquirer’s and end user’s expectations and quality standards are achieved.
Collect and analyze meaningful metrics	Meaningful metrics are those that track progress, confirm knowledge, manage risk, improve processes, and ensure that all stakeholders are well-informed.  Using a Goal-Question-Metric Approach can help to establish meaningful metrics that relate directly to the goals and requirements of the acquisition.  Project cost and schedule progress and results are enhanced by key management tools that can Track Earned Value.  Metrics-Based Scheduling and Management objectives provide a basis for assessing whether a project should proceed as planned, be subjected to reallocation of resources or reassessment of priorities, or be cancelled altogether.  Effective use of metrics, therefore, allows for informed decision-making at the most cost-effective milestones of the acquisition.
Sufficiently document, validate and manage requirements	A key element in the success of a software acquisition is the focus of management attention on the requirements-setting phase of the software lifecycle.  Missing, vague or poorly managed and controlled requirements changes result in poor software development outcomes.  Therefore, an effective Requirements Trade-Off/Negotiations process that includes all stakeholders should be implemented.  The use of Integrated Product and Process Development supports a common knowledge base to facilitate the requirements documentation, development and validation process.  Finally, a process must be established, implemented and supported in order to effectively Manage Requirements on the project.
Enhance the probability of acquiring high-quality software	Continuous process improvement is a fundamental concept governing the quality of any work product being produced.  This is no less true for software acquisition processes.  The ability to Assess Reuse Risks and Costs ensures that reuse products of questionable quality do not blindly make their way into the project.  For software developers, being able/allowed to Compile and Smoke Test Frequently ensures that software defects are detected and removed as early as possible, which is a critical determining factor for achieving high-quality software.  High quality software should also result, by definition, if the software acquisition environment is evolutionary, is based on disciplined  development processes, and makes effective use of meaningful metrics to support program decisions.
Employ systems engineering knowledge	It is critical to ensure that software requirements are achievable based on knowledge gained from systems engineering prior to beginning software development.  Trade-off analyses should be performed, supported by systems engineering analyses, that consider the performance, cost and schedule impacts to the program if there are major changes to software requirements.  Knowledge of system-level hardware and software interfaces is necessary in order to Ensure Interoperability.  A list of systems engineering deliverables (including software), tailored to the characteristics of the program, should be developed by acquirers.
OUTPUTS FROM THE PRACTICE
Acquire high quality software, on-time and within budget	The improvement of software acquisition processes will also improve the probability of Best Value Awards, as acquirers and developers team more closely with end-users to gain systems knowledge, employ disciplined acquisition/development processes, promote risk management and control activities, and use meaningful metrics to assess technical and administrative progress and quality.
Provide lessons learned to support future acquisition process improvement	It is not acceptable for a software acquisition to rest on its laurels once the program is completed.  Lessons learned must be identified, analyzed and captured so that future software acquisitions will be able to Use Past Performance to build upon program success (or to avoid the same failures).  Centralized lessons-learned databases exist and should be aggressively used to disseminate and leverage the knowledge gained from previous acquisitions.

 

RESOURCES:

 

§         Websites

GENERAL

§         OSD Acquisition Web Site

http://www.acq.osd.mil/

 

§         OSD Defense Procurement and Acquisition Policy

http://www.acq.osd.mil/dpap/

 

§         USD(AT&L) Knowledge-Sharing System (AKSS)

http://akss.dau.mil/jsp/default.jsp

 

§         Air Force Acquisition Sites

§         Secretary of the Air Force/Acquisition (SAF/AQ)

http://www.safaq.hq.af.mil/

§         US Air Force Acquisition Center of Excellence

http://www.safaq.hq.af.mil/ACE/

§         Agile Acquisition Initiative

http://www.safaq.hq.af.mil/ACE/agileacq.html

§         Air Force Capabilities-Based Acquisition System

o        AFPD 63-1: “Capability-Based Acquisition System”

http://www.e-publishing.af.mil/pubfiles/af/63/afpd63-1/afpd63-1.pdf

o        “Interim Guidance Preceding Air Force Instruction AFI63-101 (Operation of the Capabilities-Based Acquisition System)”

http://www.safaq.hq.af.mil/acq_pol/documents/Memo-InterimGuidance63-101.doc

§         Air Force Acquisition Success Stories

http://www.safaq.hq.af.mil/acq_ref/stories.html

§         Dept. of the Navy – Acquisition One Source

The DoN Acquisition One Source site is the integration of the ABM Online, Acquisition Reform, Director Acquisition Career Management, and Strategic Business Management web sites.  The site is a valuable source for Navy/Marine Corps acquisition related policy, guidance, news, services, tools, and topic information. The site aims to support the broad acquisition workforce (Defense Acquisition Workforce Improvement Act - DAWIA, other government, and industry) with the authoritative information they need.

http://acquisition.navy.mil/

 

§         Army Acquisition Sites

§         Acquisition and Contracting Policy Acquisition Tools

http://www.amc.army.mil/amc/rda/rda-ap/aqntools.html

§         Army Acquisition Support Center Portal

The Acquisition Support Center (ASC) is a new field-operating agency under the Assistant Secretary of the Army for Acquisition, Logistics and Technology. It was formed by merging the Army Acquisition Career Management Office with the Army Acquisition Executive Support Agency, as well as career programs CP-14(Contracting) and CP-13/17(LogPro). 

http://asc.army.mil/portal.cfm

 

§         DAU Acquisition Research Community of Practice

Acquisition Research encompasses all areas of research (both basic and applied) that relate to Acquisition, Logistics and Technology areas.  Mission Statement:  Provide a forum for use of planned and completed research to influence and shape defense acquisition policy by sharing relevant information and research results among researchers, policy-makers, and other research consumers.

http://acc.dau.mil/simplify/ev.php?URL_ID=10161&URL_DO=DO_TOPIC&URL_SECTION=201  (Acquisition Research Community of Practice)

http://acc.dau.mil/simplify/ev.php?ID=8122_201&ID2=DO_TOPIC  (Software Acquisition)

 

§         DAU Performance-Based Service Acquisition Resources

Expanding the use of Performance-Based Service Acquisitions (PBSA) is a top priority and goal of the Department of Defense.

http://acc.dau.mil/simplify/ev.php?URL_ID=15451&URL_DO=DO_TOPIC&URL_SECTION=201

 

§         SEI Software Acquisition Capability Maturity Model^® (SA-CMM^®)

The Software Acquisition Capability Maturity Model^® (SA-CMM^®) is a model for benchmarking and improving the software acquisition process. The model follows the same architecture as the Capability Maturity Model for Software (SW-CMM), but with a unique emphasis on acquisition issues and the needs of individuals and groups who are planning and managing software acquisition efforts.

http://www.sei.cmu.edu/arm/SA-CMM.html

 

§         SEI and General Motors CMMI for Acquisition Organizations Project

The CMMI for Acquisition Organizations Project was launched cooperatively by the Carnegie Mellon Software Engineering Institute (SEI) and the General Motors Corporation in coordination with the government / industry / SEI CMMI Steering Group. Launched on November 15, 2005, the goal of the project is to create a new business process improvement model for companies looking to source information technology capabilities from third party suppliers. View the press release about this initiative at the following link.
http://www.sei.cmu.edu/about/press/SEI_GM.html

 

§         Where in Federal Contracting?  Guidance – Source Selection

This site serves the federal and state acquisition and the federal assistance community, including public and private organizations, by providing quick access to acquisition and assistance information such as contracting laws and pending legislation, current and proposed regulations, guidance, courts and boards of contract appeals, bid protest decisions, contracting newsletters, selected analysis of federal acquisition issues, federal assistance policy, daily listings of grants and cooperative agreements, archived listings of grants and cooperative agreements, and federal assistance sites.

http://www.wifcon.com/procurementsource.htm

 

PAST PERFORMANCE

 

§         US Army Past Performance Information Management System (PPIMS)

https://apps.altess.army.mil/ppims/prod/ppimshp.cfm

 

§         Contractor Performance Assessment Report System (CPARS)

http://www.dscp.dla.mil/subs/contract/cpars.htm

§         NAVSEA Contractor Performance Assessment Report System (CPARS)

http://www.cpars.navy.mil/

§         Federal Past Performance Information Retrieval System (PPIRS)

http://www.ppirs.gov/default.htm

 

Quality Function Deployment (QFD)

 

§         Quality Function Deployment Institute (QFDI)

The QFD Institute is a non-profit organization that researches, develops, and disseminates state-of-the-art methods, tools, and training on QFD and other relevant product development and quality design methodologies.

http://www.qfdi.org/

 

Analytic Hierarchy Process (AHP)

 

§         Expert Choice

Expert Choice is a provider of advanced decision support software and services.  From IT portfolio management to strategic planning to vendor selection, Expert Choice can help your organization make better decisions and improve your bottom line.  Expert Choice has entered into a strategic alliance with the Quality Function Deployment Institute (QFDI), which will use Expert Choice’s tools in support of their Quality Function Deployment (QFD) consulting and training practice.

http://www.expertchoice.com

§         The Quality Portal

The link takes you to a brief overview of the Analytic Hierarchy Process, and links to other AHP-related sites, including Expert Choice.

http://www.thequalityportal.com/q_ahp.htm

 

§         Tools and Methods

Defense Finance and Accounting Service (DFAS) System Life-cycle Documentation Standards

The system life-cycle documentation standards are identified under the life-cycle phase in which each is created.  The documentation standards specify the format and content for all life-cycle documents.  These standards may be used as templates to help in the completion of documents.  Headers and boilerplate text, which must be included in each document, are in regular print in the documentation standards.  Instructions or sample information are printed in italics and may be deleted if the standard is used as a template.  Phases begin with Pre-Systems Acquisition through Production and Deployment.

http://www.dfas.mil/technology/pal/ssps/docstds/lcdocstds.htm

http://www.dfas.mil/technology/pal/ssps/docstds/

 

Acquisition Strategy Decision Guide – Dept. of the Navy

This guide was written to assist Program Managers and their teams in the selection and definition of acquisition strategies.  It is also intended to be useful in periodically reviewing and revising acquisition strategies when conditions dictate.  The guide is built around several “key” concepts and it walks the user through several straightforward step-by-step processes.

http://acquisition.navy.mil/navyaos/content/download/1303/6141/file/asdgjan2001.pdf

 

Contracting for Best Value: A Best Practices Guide to Source Selection – AMC Pamphlet 715-3

This guide provides techniques and practices for obtaining best value products and services through source selection.  Consistent with the spirit of acquisition reform, it introduces new and innovative techniques to simplify the source selection process and produce better value.  Its purpose is to provide a practical reference tool that will help implement a new way of doing business that promotes flexibility, streamlining, and simplified procedures.

http://www.amc.army.mil/amc/rda/rda-ap/ssrc/ssp_toc.htm

 

Army Source Selection Guide

The objective of the Guide was to produce a living document that was web based and lent itself to changes as deemed appropriate.  They endeavored to make the Guide simple, straightforward and user-friendly.  The document was purposely written as guidance and should not be considered mandatory or regulatory.  By incorporating best practices that contracting activities may utilize on an “as-needed” basis, the Guide should prove to be a very valuable tool in conducting source selections.

http://www.amc.army.mil/amc/rda/rda-ap/docs/assg-2001.pdf

 

Expert Choice

Expert Choice software is designed to enable decision-makers to develop consensus about what is really important in selecting the best product or service.  Expert Choice consultants guide decision-makers to weight criteria using a unique process for developing priorities and building consensus.  Expert Choice is often used by government agencies to reduce the amount of time it takes to select the best vendor, then justify the selection against protest using the advanced reporting capabilities of the software.

http://www.expertchoice.com/applications/vendorselection.htm

 

Enterprise Integration (EI) Toolkit

The Enterprise Integration (EI) Toolkit contains a Roadmap for you to follow in the development of your component architecture and COTS acquisition and implementation projects.  It includes more than 150 templates, checklists, and other tools to help your project be successful.

http://www.eitoolkit.com/

 

Guidebooks and Handbooks to Support Acquisition (AT&L Knowledge Sharing System (AKSS))

http://akss.dau.mil/jsp/AkssPage.jsp?fName=../jsp/Guide_Hand_Book_links.jsp&title=Guidebooks%20and%20Handbooks

 

Software Tools to Support Acquisition (AT&L Knowledge Sharing System (AKSS))

The table presented below is a condensed representation of the list of software tools to support acquisition that is available on the AKSS site.  Note that the list of tools is not intended to be complete, as it represents only those tools that are “owned” by a DoD organization.  As a result, a number of “commercial” tools are not represented (e.g., COCOMO, SLIM and Cost Xpert in the Price Estimating – Contracting Topic Area).

http://akss.dau.mil/jsp/software1.html

 

Software Acquisition Tools

Topic Area	Name	Owner - Organization
Auditing	E-Z Quant Application Software	DCAA - Technical Audit Services Division
CDRL	Integrated CDRL Tracking and Routing System (ICARS 5.0)	Air Force - AFMC
Contracting	Acquisition Professional (AcqPro) Document Generator Award Fee (AWF 3.0) - commercial product ConWrite Pre-Award Information Exchange System (PIXS 5.0) USAF Game Board	Army - TACOM Air Force - SMC Air Force - AFMC Air Force – ASC/SUG Air Force – HQ AFMC/DR
Cost Analysis	COMPASS LITE Cost Analysis Strategy Assessment (CASA) Web-Life-cycle Cost Analyzer (Web-LCCA)	Army – LOGSA/AMC Army – Logistics Support Activity Air Force – ASC/COTS – Logicon - TASC
Cost Estimating	Munitions Design Trade/Operation & Support Cost Model SEER - Software Estimation Model (SEER-SEM)	Air Force – AAC/FMC   Air Force – ASC/FMCE
Cost Management	Correlation Calculator for Cost-Risk Analysis (C-RISK 3.0) Formal Risk Analysis (FRISK 4.00)	Air Force – SMC/FMC COTS – Aerospace Corp. Air Force – SMC/FMC COTS – Aerospace Corp.
Decision Support	Resource Analysis Decision Support System (RADSS 2000)	Air Force - TASC
Finance	Merged Obligation and Liquidation Tracking System (MOLTS 8.0) Viewfinder/ds and Louis II Web-Life-cycle Cost Analyzer (Web-LCCA)	Air Force – COTS – Logicon – TASC   Air Force – SSG/FNAF Air Force – COTS – Logicon – TASC
IT Management	Enterprise Integration Toolkit Wide Anti-Virus Software Enterprise Licenses	OSD – ADUSD(L&MR) DISA – DITCO-NCR
Price Estimating – Contracting	PRICE Estimating Suite (PRICE H/HL/M/S)	Air Force – ASC/FMCE
Program Management	Acquisition Professional (AcqPro) Document Generator Conformer (Conformer 8.2)   Consolidated Acquisition Reporting System (CARS) Enterprise Integration Toolkit ModRite (ModRite 4.0)	Army – TACOM/SFAW-GCSS-W-AB-CC/TASC Air Force – ASC/COTS – Logicon – TASC OSD – OUSD(AT&L)   OSD – ADUSD(L&MR) Air Force – HQ AFSOC/XPQA
Project Management	Process Analysis and Project Integrated Environment	Air Force – SMC/Det 11
Requirements	Action Officer Guide (AOG v2.5 and 3.2)(*.mil only) Data Item Descriptions Advisor Xtie-RT Requirements Tracer (3.1.02)	Air Force – AFMC   Air Force – AFMC DoD – Teledyne Brown
Risk Assessment	Consolidation Risk Assessment Methodology (CORAM) Cost/Risk Identification & Management System (CRIMS) Risk Matrix (Risk Matrix 2.20)	Air Force – Eglin AFB   Air Force – SAF/AQXR   Air Force – ESC/BP
Source Selection	FedSelect Proposal Eval Source Selection Support Tool Source Selection Application EZSource USAF Game Board	OSD – DoD/MDA   Air Force – ASC/SYG Air Force – HQ AFMC/DR
Total Ownership Cost	AFTOC Management Information System	Air force – SAF/AFCAA
Training	Training Management System (TMS)	DSCA - DISAM

 

 

Contractor Performance Assessment Reports (CPARs)

For an actual practice session for entering, reviewing, commenting on and modifying a Contractor Performance Assessment Report (CPAR), click http://cpars.navy.mil/cparsfiles/cpartreq.asp

 

National Aeronautics and Space Administration (NASA) Past Performance Data Base (PPDB) - NASA has also developed a past performance database to track contractor's performance on NASA contracts. Currently there is no web site with information on NASA's system but for more information, contact NASA Headquarters at (202) 358-1279.

 

NIH Contractor Performance System

http://cps.od.nih.gov/

 

Air Force Single Acquisition Management Plan Guidelines

http://www.safaq.hq.af.mil/contracting/toolkit/part07/word/sampguide.doc

 

SPAWAR Acquisition/Documentation Matrix (Generic)

http://enterprise.spawar.navy.mil/apsg/Enclosure5.pdf

 

Defense Systems Management College (DSMC) Acquisition Strategy Guide

The purpose of this Guide is to provide, in a single source, information that PMs should find useful in structuring, developing, and executing an acquisition strategy.  A process for developing and executing an acquisition strategy is provided together with criteria for evaluating a proposed strategy.  However, this Guide alone does not provide the PM with a definitive acquisition strategy for ones particular program.  Well informed, educated, and innovative applications and judgments concerning the particular mission need are necessary to structure a successful acquisition strategy.  PMs should continue to seek guidance, data, and assistance from available sources as they prepare and revise their acquisition strategy.

http://www.ntsc.navy.mil/Resources/Library/Acqguide/acqstrat.pdf

 

Tools from the Data and Analysis Center for Software (DACS)

 

The DACS “Best Value Analysis for Acquisition” tool – DACS has developed a basic tool, using MS Excel^®, as a means for comparing bidder’s proposals on technical approach, management approach, past performance and price to determine the combined “best value” proposal based on user-defined evaluation, rating and weighting criteria.  There are two versions of this tool.  The “Fixed” version allows input of proposal scoring criteria that will be applied to all of the evaluation categories (technical, management, past performance, price).  The “Variable” version allows input of tailored scoring criteria for each of these four categories.  These tools can be downloaded from:

http://www.goldpractices.com/practices/api/Best_Value_Analysis_for_Acquisition–Fixed_Rating_Criteria.xls

http://www.goldpractices.com/practices/api/Best_Value_Analysis_for_Acquisition–Variable_Rating_Criteria.xls

 

The DACS “Acquisition Risk Management Scorecard” tool – DACS has developed a MS Excel^®-based tool to allow for an assessment of acquisition risk based on either basic, or more advanced, categories of risk.  For the basic tool, three qualitative levels of risk are defined (high, medium, low) and user-defined relative weighting scores can be assigned to each.  For the advanced tool, the three qualitative areas are subdivided into quantitative ratings, allowing for enhanced risk-scoring discrimination.  This tool can be downloaded from:

http://www.goldpractices.com/practices/api/Acquisition_Risk_Management_Scorecard.xls

 

The DACS “Software Acquisition Management Risk Factors” tool – A slightly different risk management scorecard, this MS Excel^®-based tool is designed to allow you to score the risk associated with 10 predefined areas of software acquisition risk.  The risk factors were taken from the “Software Acquisition Risk Factors” matrix located at the State of Texas Department of Information Resources website.  This tool can be downloaded from:

http://www.goldpractices.com/practices/api/Software_Acquisition_Management_Risk_Factors.xls

 

The DACS “Software Acquisition Process Improvement Interview” – This interview/survey was developed by the DACS to support the definition and implementation of a DoD Software Acquisition Process Improvement Program.  Section 804 of the Bob Stump National Defense Authorization Act for Fiscal Year 2003 requires the establishment of software process improvement programs by Military Departments and those Defense Agencies that manage Major Defense Acquisition Programs with a substantial software component (i.e., software-intensive systems (SIS)).

http://www.goldpractices.com/practices/api/Software_Acquisition_Process_Improvement_Interview.doc

 

Software Acquisition Process Maturity Questionnaire - This package contains a copy of the software acquisition process maturity questionnaire.  It is intended for those interested in performing and learning about software acquisition process appraisals.  This questionnaire is not an appraisal method itself; rather, it is a tool that may be used in different appraisal methods.

http://www.sei.cmu.edu/pub/documents/97.reports/pdf/97sr013.pdf

 

Tools for Earned Value Management Listed at the DoD Acquisition Web Site

This site provides links to material on software tools for earned value management which can be useful for tracking the progress of an acquisition over its life-cycle.

http://www.acq.osd.mil/pm/old/tools/tools.htm

 

Proposal Evaluation Tool (PET)

This software tool was developed to answer the question: "How should I allocate my limited resources to maximize the benefit to my organization?"  PET is an interactive, Web-based application that uses the Analytical Hierarchy Process and Integer Linear Programming to identify which combination of proposals will maximize the total value to the organization within a specified budget.  PET will also support portfolio planning over multiple budget years.  The development of this tool was funded by the Office of Naval Research (ONR) Affordability Measurement and Prediction Program (AMPP), to support its efforts in science and technology portfolio prioritization analysis.

http://www.tecolote.com/Services/SystemAcquisition.htm

 

Software Risk Checklist

The NASA QE(8000) Risk Management Office produced this 18-page software risk checklist, which is laid out by development phases of a project and is specifically constructed to cover various high-level life-cycle phases, including the system requirements phase, the software planning phase, the software requirements phase, the software design phase, the software implementation phase, and the acceptance testing/release phase.

http://osat-ext.grc.nasa.gov/rmo/spa/SoftwareRiskChecklist.doc

 

§         Experts/Contact Points

Katherine V. Schianasi

GAO, Director, Acquisition and Sourcing Management

schinasik@gao.gov

Suellen Eslinger, Aerospace Corporation, Software Acquisition and Process Office

(Suellen.Eslinger@aero.org)

Richard Adams, Aerospace Corporation, Software Acquisition and Process Office

(Richard.J.Adams@aero.org)

Lisa Pracchia, NAWCWD

(Lisa.Pracchia@navy.mil)

Dr. Richard Turner, OUSD(AT&L)

(Rich.Turner.CTR@osd.mil)

Dr. Barry Boehm, Univ. of Southern Cal.

(boehm@usc.edu)

Thomas Bernard, ASC/ENAC

(Thomas.bernard@wpafb.af.mil)

Brian Gallagher, Software Engineering Institute (SEI)

(bg@sei.cmu.edu)

 

§         Training Opportunities:

 

§         Defense Acquisition Process Tutorial (Defense Acquisition University – DAU)

http://dod5000.dau.mil/TUTORIAL/index.htm

 

§         For an interactive Web Training tutorial on CPARS with "Show Me" features, click

http://cpars.navy.mil/cparsfiles/cpcbtdlf.asp

 

§         Navy/Air Force Practice CPARS Training 

http://cpars.navy.mil/cparsfiles/cpartreq.asp

 

§         CPARS Training Materials 

http://cpars.navy.mil/cpartrng/cpars_training.asp#trnmatl

 

§         Single Acquisition Management Plan (SAMP) Training (MS PowerPoint presentation as part of SMC Acquisition Reform Training) – Ronald G. Martin HQ SMC/ADX

http://ax.losangeles.af.mil/axd/training/samp%20devlp%20v0%20rm.ppt

 

§         Incentives for Reducing Acquisition Response Time – AFIT SIS 352

 

The Air Force Institute of Technology presents this short, Internet-based course which provides an overview of incentives available to motivate both government program office personnel as well as contractor personnel to reduce acquisition response times in their day-to-day business.

We show that the perception is that government and contractor personnel are not incentivized to exceed program objectives but at best to only meet program objectives. Through examples and short exercises the student realizes that incentives can be effective to reduce acquisition response times. The student is also introduced to Schedule Risk Assessments and their usage as part of source selection.  In order to take this course, you must first register with the AFIT Virtual Schoolhouse and be assigned to a section (follow the link below).

http://clc.dau.mil/kc/ilc/course_launch_generic.asp?crs_ident=C00175&return_loc=course%5Finfo%5
Fenroll%5Ffrm%2Easp%3FblnReturn%3DTrue%26table%3Dcrs%26function%3Dcourse%5Finfo%5Fenr
oll%26crs%5Fident%3DC00175%26keywords%3D%26topic%3DAll&strReturnLocString=Cancel

 

§         Introduction to the Software Acquisition CMM

 

This two-and-a-half-day course introduces those responsible for the acquisition of software-intensive systems, software products, and services to the Capability Maturity Model^®: for Software Acquisition (SA-CMM^®) and its fundamental concepts as well as the value it can bring to organizations using it.

http://www.sei.cmu.edu/products/courses/intro/intro.sa.cmm.html

 

§         DACS Affordability Training Module “Software Acquisition for Software-Intensive Systems (SIS)”

 

This is a MS PowerPoint presentation that provides an overview of the processes and procedures associated with the acquisition of software for complex systems.

http://www.goldpractices.com/practices/api/Software_Acquisition_for_SIS.ppt

 

§         Defense Acquisition University (DAU) Continuous Learning Center

 

This link provides access to several DAU continuous learning modules based on Software Acquisition Management.  Just select “Software Acquisition Management” from the Topic drop-down menu to review the available modules.

http://clc.dau.mil/kc/no_login/portal.asp?strRedirect=LC_CIA

 

§