Working Group Report

System Development Performance Measurement

October 2011

An issue often cited in studies and reports is the ineffective use of measures and predictive leading indicators to proactively plan and manage the successful acquisition and execution of defense programs. This is reflected as one of the top NDIA systems engineering issues needing to be addressed:

In September 2010, the NDIA Systems Engineering Division and Practical Software and Systems Measurement (PSM) sponsored a working group to consider these issues and provide recommendations on a set of information needs, leading indicators, and measures for use by both acquirers and suppliers to obtain better insight into program status and risks to aid ongoing communication and to provide input to decision-making at key program milestones and decision points. Working Group Report
 

Systems Engineering Leading Indicators Guide

Version 2.0²

February 5, 2010 Supersedes Initial Release, June 2007

Editors: Garry Roedler, Lockheed Martin Corporation /
Donna H. Rhodes, Massachusetts Institute of Technology /
Howard Schimmoller, Lockheed Martin Corporation /
Cheryl Jones, US Army

This Systems Engineering Leading Indicators Guide is the result of a project initiated by the MIT Lean Advancement Initiative (LAI), in cooperation with the International Council on Systems Engineering (INCOSE), Practical Software and Systems Measurement (PSM), and MIT Systems Engineering Advancement Research Initiative (SEAri). Additional collaborating organizations involved in Version 2.0 include Naval Air Systems Command (NAVAIR), US Department of Defense Systems Engineering Research Center (SERC), and National Defense Industrial Association (NDIA) Systems Engineering Division (SED). Many leading measurement and systems engineering experts from government, industry, and academia volunteered their time to work on this initiative. The Systems Engineering Leading Indicators Guide is issued by INCOSE as document number INCOSE‐TP‐2005‐001‐03, and is available on its website. It is also available on the PSM website and the MIT LAI and MIT SEAri websites. [Full Release Announcement]
 

Practical Measurement Framework for Software Assurance and Information Security

Version 1.0

By Nadya Bartol, Booz Allen Hamilton, October 2008

The Practical Measurement Framework for Software Assurance and Information Security provides an approach for measuring the effectiveness of achieving Software Assurance (SwA) goals and objectives at an organizational, program or project level.  It addresses how to assess the degree of assurance provided by software, using quantitative and qualitative methodologies and techniques.  This framework incorporates existing measurement methodologies and is intended to help organizations and projects integrate SwA measurement into their existing programs.  Organizations can use common measurement framework to implement SwA and security measurement at the desired organizational level, tailor it to the organizational stakeholders, and integrate into existing measurement and risk management activities.  The document provides example stakeholder goals and measures, as well as a generic measurement process, to help organizations get started with SwA measurement.

If you wish to comment on the document, please use the comment form and send comments to nadya_bartol@bah.com.
 

Acquisition Measurement

Version 1.0

By Rita Creel, Joe Dean, and Cheryl Jones, July 2008

This paper provides a foundation for the discussion and advancement of acquisition measurement. Its primary focus is on improving the way in which acquisition projects and organizations manage and conduct their own activities. A secondary focus is management and oversight of the supplier. The paper provides guidance, a WBS, and Information Need - Category - Measure (ICM) table.
 

Safety Measurement White Paper V3.0

23 January 2006

This White Paper is the result of work by the Safety & Security Technical Working Group of PSM.  The objective of this work has been to propose additions to the existing PSM guidance materials appropriate for organizations and projects developing safety-critical products.
 

Security Measurement White Paper V3.0

13 January 2006

This White Paper reports on research on the application of measurement principles to the security properties of software-intensive systems.  The work has the objective of integrating security measurement with the general measurement principles as developed by the PSM project and in accordance with the related standard ISO/IEC 15939:2002.  The application of measurement principles to security is a relatively new field and presents several challenges, explored in this paper.
 

Technical Measurement Guide V1.0

By Cheryl Jones and Garry Roedler, 27 December 2005

This guidance is the result of a joint project that was conducted between the Practical Software and Systems Measurement (PSM) project office, the International Council On Systems Engineering (INCOSE), and various companies.  This guide provides information on implementing technical measurement on a project, including Measures of Effectiveness (MOEs), Key Performance Parameters (KPPs), Measures of Performance (MOPs), and/or Technical Performance Measures (TPMs).
 

Measurement Guidance for Process Improvement V1.0

By Joyce Statz, 4 September 2005

This paper consolidates the output of several technical working group meetings addressing process improvement measurement.  This material may be used with process improvement efforts of many different types and sizes: individual improvement projects, programs of projects, and small process improvement team activities.  The paper identifies potential measures in the areas of (1) Anticipated value of the process improvement project, to justify doing the project, (2) Readiness of the organization for process improvement, and (3) Progress of the process improvement project including both progress to plan and progress in attaining the anticipated value or results.
Click here to view the Measurement Information Specifications.
 

System of Systems Measurement White Paper (Draft)

14 July 2003

SoS implementations often take longer, cost more, and present greater challenges than anticipated. Although project-centric information needs are relevant, the SoS domain is sufficiently different to require new measurement constructs and measures to be developed, validated, and put into practice. Click here to view this White Paper.
 

Applying PSM to Enterprise Measurement

By David Card and Robert MacIver, March 2003

This report describes the application of the concepts of Practical Software Measurement (PSM) to satisfy the information needs of enterprise management. In particular, it focuses on the relationship between PSM, an established approach for project measurement, and the Balanced Scorecard, an established approach for enterprise measurement. The key to integrating the two approaches lies in reconciling the two views of information needs.
 

Measures for DoD Software Product Lines

By Dave Zubrow, April 2001

This paper is one of a series commissioned by the Department of the Undersecretary of Defense, Science and Technology (DUSD) (S&T) that characterizes the status of measurement associated with a particular aspect of software engineering. The specific focus of this paper is measures for software product lines.
To view this paper, click here to log in and select the Measurement Technology Papers product.
 

Measuring System Interoperability

By Mark Kasunic, March 2001

This paper is one of a series commissioned by DoD that characterizes the status of measurement associated with a particular aspect of software engineering. The specific focus of this paper is on measures for interoperability.
To view this paper, click here to log in and select the Measurement Technology Papers product.
 

Measures in Support of Evolutionary Acquisition

By Jim Marple, Brad Clark, Cheryl Jones, David Zubrow, March 2001

The intent of this paper is to identify what measures are necessary to aid in acquisition agency and a contractor when it's decided that a program should follow an evolutionary acquisition strategy. Evolutionary acquisition is a strategy that develops and deploys a core capability with the intent to field additional capabilities as stakeholder needs, expectations, constraints, and interfaces are better understood.
To view this paper, click here to log in and select the Measurement Technology Papers product.
 

Object-Oriented Measurement

By David N. Card, Khaled El Emam, Betsy Scalzo January 2001

This technical report summarizes the state of the practice for measurement of object-oriented (OO) software development projects. These reports are intended to help those software intensive projects that are employing new technologies and methods to establish effective measurement systems for project management purposes.
To view this paper, click here to log in and select the Measurement Technology Papers product.

PSM: Measuring for Process Management and Improvement (MPM)

April 1997

This SEI-developed PSM:MPM guidebook describes the application of well-established principles and methods of evaluating and controlling process performance. PSM:MPM focuses on the issues that help organizations achieve technical goals and improve long-term profitability.  Download MPM.zip (1112KB)