Session: 16-01-01: Government Agency Student Poster Competition

Paper Number: 151065

151065 - Hidden Beliefs in Engineering Verification Strategies 

In the systems engineering process, the purpose of a verification strategy is to ensure that a system has been built right. The verification strategy contains the requirements to be verified, method of verification, parameters to be verified, and measures of performance for parameters to be verified against. Execution of verification activities produces verification evidence, used as information in performing subsequent verification activities. This research aims to demonstrate the existence of hidden beliefs in verification strategies and specifically asserts that the verification matrix does not explicitly and adequately capture this influencing factor of hidden belief requisite to performing the verification activity. It aims to establish the existence of hidden beliefs through observing differences in the identification of supporting information artifacts in compliance judgements made.

In performing a verification activity, the engineer is essentially codifying their belief on the requirement being met. The engineer interprets information available in combination with existing knowledge through the lens of belief to determine whether the requirement is verified and what evidence supports this judgement. A verification strategy can be conceived as a belief network, and analogous to a belief network, information acts as nodes and their associations as edges. The nodes that provide information (such as tests or analyses) are directly accessible. Other nodes, such as system characteristics, are predicted based on the information nodes, and remain hidden. These nodes of hidden belief are not explicitly codified in verification strategies. Lack of considering hidden belief may lead to gaps in both verification strategy design and implementation, and an increased risk of system failure. Incorporating belief is imperative for communication amongst system stakeholders, and for change management. Additionally, as reliance on AI as part of a digital engineering ecosystem increases, exclusion of explicit consideration of belief causes algorithms to likely be trained on an incomplete set of data.

An empirical study was conducted by sampling two different populations: 1) junior and senior year aerospace engineering students, and 2) seasoned aerospace engineering practitioners. Participants were randomly segregated into control and experimental groups and asked to perform verification activities typical to a commercial aircraft by filling in a verification matrix and documenting their rationale in a report. Two sets of verification evidence, Set A and Set B, were used. Each group performed the verification activity twice, each time with a different set of evidence. The control group received Set A followed by Set B. The sequence was reversed for the experimental group. This order of presentation variation was intended to test the null hypothesis that both groups would make identical selections of verification artifacts that fulfill the verification success criteria. Rejection of this hypothesis would prove the existence of hidden belief, and its manifestation captured through variations in artifact selection.

The results showed the lack of consideration of important information in making compliance decisions, dissimilarity in artifact selection by both groups, differing assumptions arising from hidden belief explaining disparity in artifacts selected, and disbelief and low confidence in information provided (verification method and calculations). The conclusion was that hidden beliefs exist, and without explicit codification in the matrix, risk contributing to verification errors. This research aims to contribute to formalizing theory for design and implementation of verification strategies. The future scope includes quantification of factors influencing hidden belief and provision of mathematical strategies for  completeness in elicitation.  An extension into digital engineering with experimentation on existing AI, and design of updated intelligence models is planned for.

Presenting Author: Joanna Joseph University of Arizona

Presenting Author Biography: Joanna Joseph is an engineer currently pursuing her PhD in Systems and Industrial Engineering at the University of Arizona. She holds degrees in Systems Engineering and Mechanical Engineering and has prior experience working in the energy and environmental industry.

Her research is focused on building on the theoretical foundations of systems engineering, in the context of verification and human cognition. Her work is inter-disciplinary and lies in the intersection between systems science and engineering, psychology, cognitive science, and philosophy. She believes in the value of adding to the theoretical foundations of systems engineering, while creating frameworks for adoption by practitioners.

In addition to systems engineering research, she is a certified CIRTL Practitioner and is involved in academia through active teaching experience as well as certified pedagogical and education research.

Authors:

Joanna Joseph University of Arizona
Alejandro Salado University of Arizona