Technical Reviews and Audits
Lead Authors: Ken Garlington Contributing Authors: David Endler, Garry Roedler, Mike Yokell
Technical reviews and audits are a mechanism by which sufficiently independent and knowledgeable stakeholders analyze the current state of a system, work product, or set of work products using pre-established criteria. The results of these analyses are used to support programmatic and technical decisions.
Concepts
A technical review is a series of systems engineering activities conducted at logical transition points in a system life cycle, by which the progress of a project is assessed relative to its technical requirements using a mutually agreed-upon set of criteria (ISO/IEC/IEEE 24748-8:2019). An audit is an independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria (ISO/IEC/IEEE 15288:2023). Different organizations, functional disciplines, etc. can use either term or other terms for these related concepts.
Technical reviews and audits support multiple systems engineering processes, including:
- Project assessment and control. Technical reviews and audits help provide visibility into the project’s technical progress and risks, and support decision gates by establishing readiness to proceed to the next milestone or life cycle stage.
- Configuration management. Technical reviews and audits support establishing, revising, and verifying configuration baselines.
- Validation. Through the involvement of appropriate stakeholders (including end users), technical reviews and audits enable early validation of requirements, design, and other work products.
Technical reviews and audits can be applied to a system, system element, or other established technical boundary. In addition to their use at transition points in a system life cycle, they can also be used to authorize risky activities, such as testing with safety or security concerns.
Approach
A typical approach to a technical review or audit includes preparing, conducting, and completing it.
Prepare
Technical reviews and audits assess the progress of a project typically include:
- Business case or mission.
- Budget.
- Technical maturity, including associated risks.
These must be balanced to ensure that business and technical baselines are acceptable and will lead to satisfactory verification and validation. The decision as to whether or not to proceed is sometimes associated with great uncertainty. For this reason, it is advisable to properly document the assumptions made and the risks identified.
Technical planning is used to define and coordinate the elements of each technical review or audit, including:
- technical review or audit name,
- purpose and expected outcomes,
- scope (project, system, system element, etc.)
- relevant properties of the system, etc. to be assessed,
- timing and frequency, and
- specific activities and tasks.
To ensure objective and comprehensive results that support the expected outcomes, specific entry and exit criteria should be defined and coordinated, and the completion of the entry criteria confirmed, prior to conducting a technical review and audit (see also Life Cycle Stages and Entry/Exit Criteria). In addition, any necessary independence and areas of expertise required should be considered in the selection and coordination of subject matter experts (SMEs) as participants, including representatives from specialty engineering.
For some key stakeholders, particularly for senior technical and programmatic personnel responsible for a portfolio of projects, a technical review or audit can be a primary method of participating in a project. Planning should consider the needs of these and other stakeholders with respect to specific scheduling, coordination, presentation of project context, etc. Roles and responsibilities for each participant should be coordinated prior to conducting the technical review or audit. Most technical reviews and audits require a set of interrelated information items to be assessed for consistency, completeness, etc. The specific configuration of the information items in the set should be identified and made available to participants as needed prior to conducting the technical review or audit.
According to Boehm and Lane, there are three different types of technical reviews:
| Review type | Examples | Considerations |
|---|---|---|
| Schedule-based | Explicit dates in agreement |
|
| Event-based | End of development phase (e.g. design) based on delivery of artifacts as defined in agreement |
|
| Evidence-based | Achieving a specific technical risk level based on supplied evidence |
|
For all of these, there can be misunderstandings or adverse drivers (e.g. schedule, cost) that cause the technical review to be held with insufficient preparation, information, etc.
Conduct
Participants support the purpose and expected outcomes of the technical review or audit through the analysis of relevant information using pre-established criteria. The information may be available as documents, models, simulations, prototypes, or other appropriate formats. Usually, different participants will perform their analysis from different perspectives (safety, security, reliability, etc.) consistent with their expertise. Participants may perform their work independently or as a group. Typically, all participants will meet to discuss analysis results and identify actions based on the results.
In many cases, it can be particularly challenging to identify the right people who are able to objectively assess the outcome of a review. If these people are “too close” to the project, they may be biased (e.g., miss out on the next bonus). If these people are “too far away” from the project, critical issues may not be discovered. Thus, it is most effective to build a review team with a combination of people close enough to provide important insights and some people who are far enough removed to have independent viewpoints.
Complete
Once the exit criteria have been met for the technical review or audit, the results are communicated to the participants (and other stakeholders unable to participate), and any identified actions are implemented as needed to support the desired project outcomes.
Timing and Frequency
Technical reviews and audits can be applied to both a system of interest and recursively to constituent system elements as needed. They can occur sequentially, iteratively, incrementally or concurrently. In any case, they should not be conducted until the pre-defined entry criteria have been met.
Representative Technical Reviews and Audits
Technical reviews and audits are defined based on the needs of the organization, domain, life cycle, etc. Although industry and organizational standards in this area can be useful, they should be carefully tailored for the needs of each project. Example applications are described in:
- United States Department of Defense Systems Engineering Guidebook (Feb 2022),
- AAP-48, North Atlantic Treaty Organization System Life Cycle Processes (May 2022),
- NPR 7123.1, National Aeronautics and Space Administration Systems Engineering Processes and Requirements, Jul 2023, and
- Incremental Commitment Spiral Model
For additional information on applying technical reviews and audits, see ISO/IEC/IEEE 24748-8.
References
Works Cited
Boehm, B. and Lane, J., “Improved Acquisition Process Through Incremental Commitments,” Tutorial at the 13th Annual NDIA Systems Engineering Conference, The National Defense Industrial Association (NDIA), 2010.
US DoD. 2022. Systems Engineering Guidebook. Washington, DC: US Department of Defense. Available at: https://www.dau.edu/sites/default/files/2024-02/Systems%20Engineering%20Guidebook%2C%20February%202022.pdf
ISO/IEC/IEEE 15288. 2023. Systems and software engineering — System life cycle processes. Geneva, Switzerland: International Organization for Standardization (ISO), International Electrotechnical Commission, and Institute of Electrical and Electronics Engineers. Available at https://www.iso.org/standard/81702.html.
ISO/IEC/IEEE 24748-8. 2019. Systems and software engineering — Life cycle management, Part 8: Technical reviews and audits on defense programs. Geneva, Switzerland: International Organization for Standardization (ISO), International Electrotechnical Commission, and Institute of Electrical and Electronics Engineers. Available at https://www.iso.org/standard/75405.html.
NATO. 2022. "NATO System Life Cycle Processes." Brussels, Belgium: North Atlantic Treaty Organization (NATO). NATO-AAP-48. Available at: https://standards.globalspec.com/std/14514800/aap-48
Primary References
None.
Additional References
None.