Session: 04-06-01: AI for Heterogeneous Materials Design, Discovery, and Manufacturing I

Paper Number: 167168

Challenges and Opportunities in Credibility Assessment of Computational Models: A Case Study in Rapid Discovery and Testing of Metallic Alloys 

Advances in high-resolution, high-throughput measurement techniques, coupled with powerful developments in artificial intelligence (AI), have enabled breakthroughs in rapid discovery and testing of novel materials. For example, recent research shows that high-performance metallic alloys can be discovered and tested for advanced mechanical properties such as creep and fatigue within a few days, using AI-based prediction of long-term macroscopic properties from early-stage microscopic behavior. In contrast, the state-of-the-art methods directly measuring those properties can take weeks, if not months. These possibilities open up new research questions pertaining to the validity, predictivity, and practicality (e.g., fitness for purpose and use) of the underlying models. These questions are especially important in applications that deal with human safety and life, such as engineering design and certification of aerospace and medical products.

We present a risk-based credibility assessment framework to support engineering and regulatory decision-making in the use of computational (including AI) models. The framework consists of a number of steps and best practices to define quantities of interest (QoIs), establish context of use (CoU), assess risk, develop a plan to establish credibility within the CoU, execute the plan, document the results and deviations from the plan, determine adequacy of the models based on such results, and iterating over these steps until the required adequacy is achieved. We will use the models underlying rapid discovery and testing of novel metallic alloys in the DARPA METALS program as a use-case study to highlight opportunities, gaps, and guidelines for future directions that are generalizable to a broader set of engineering applications.

The paper will present an overview of the ASME Validation, Verification, and Uncertainty Quantification (VVUQ) standard series, definitions, and historical context on the scope and evolution of standards leading to industrial uptake, e.g., recent interest from the pharmaceutical industry in VV40. We also review the state-of-the-art in VVUQ of computational models such as sensitivity analysis and sampling-based methods, applied at different levels of granularity ranging from high-fidelity to surrogate and reduced-order models, including statistical AI models (e.g., machine learning). We propose a multi-step process for risk-based credibility assessment of such models, inspired by the U.S. Food and Drug Administration (FDA) Draft Guidance on “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products” with the observation that the underlying principles can serve as a groundwork for engineering applications such as advanced materials for aerospace and energy sectors. As a case study, we consider the AI models used in DARPA metals to 1) predict advanced mechanical (e.g., fatigue and creep) properties of metallic alloys from high-resolution digital image correlation (HR-DIC) measurements under a scanning electron microscope (SEM); 2) evaluate realizability of materials with new properties to constrain simultaneous design of mechanical parts and discovery of new alloys (including compositionally-graded alloys); and 3) mapping compositions to properties and vice versa to enable design of experiments (DoE) for material development and validation of realizability. We explore potential extensions and applications of the risk-based credibility assessment in advanced manufacturing, qualification and certification, and structural health monitoring. We conclude the paper with practical challenges, gaps in the proposed framework in addressing them, and future directions.

Presenting Author: Morad Behandish SRI International

Presenting Author Biography: Morad Behandish leads the Computational Design (CD) group as a Senior Manager and Associate Director in the AI Center of SRI International. He founded CD in 2020 at the Xerox Palo Alto Research Center (PARC), which was acquired by SRI in 2023. Morad’s background is in geometric and physical modeling, computing, and AI to democratize design and digital manufacturing. Since 2018, he has led multiple DoD and DARPA-funded projects, with total funding in excess of $10M, to de-risk radically new concepts. He has also overseen commercial R&D programs for technology transfer to the industry in the "Next-Gen Production" focus area, one of the three strategic initiatives of PARC under its open innovation model. Morad has published 50+ scientific articles, filed 60+ inventions (40+ patented or pending), and received 15+ awards for research, service, leadership, and entrepreneurship, including 3 Best Paper Awards and an ASME/CIE Best Doctoral Thesis Award. He has served the scientific community in various capacities such as an Editor of the journal of Computer-Aided Design (CAD), Program Chair for the symposia on Solid and Physical Modeling (SPM), EC member of the Solid Modeling Association (SMA), and other roles in ASME.

Authors:

Morad Behandish SRI International
Jean-Charles Stinville University of Illinois Urbana-Champaign
Adrian J. Lew Stanford University
Vijay Srinivasan National Institute of Standards and Technology (NIST)