Session: 14-06-01: Machine Learning and Deep Learning in Safety, Reliability, and Maintenance I

Paper Number: 145309

145309 - Dynamic Fault Tree Games and Reinforcement Learning for System Reliability 

Modern complex systems, such as power grids, transportation networks, and industrial processes, are characterized by their intricate interconnections and dependencies, making them susceptible to various faults and failures. Ensuring the reliability and robustness of such systems is paramount for maintaining safety, productivity, and efficiency in critical domains. Traditional fault analysis techniques, while effective to some extent, often struggle to keep up with the increasing system complexity of modern systems. Fault Tree Analysis is an important technique for assessing and mitigating risks inherent in complex systems. It provides a graphical representation of the logical interconnections among system components and the potential events culminating in system failure. While static fault trees delineate fixed relationships, Dynamic Fault Trees (DFTs) can accommodate dynamic systems where conditions change over time, such as repairs, maintenance, or system reconfiguration. This study investigates the reliability of a system employing Dynamic Fault Tree and Adversarial Multiagent Reinforcement Learning (Adv-MARL). The methodology used includes utilizing a specified DFT to construct a system model and formulate a zero-sum game with two players: one aiming to disable DFT events to induce top event failure, and the other repairing failed events to prevent top event from failing. Both players are trained using Reinforcement Learning (RL), where rewards are provided based on their actions, and Neural Networks (NN) enable them to learn and optimize expected rewards for each subsequent action. Two learning algorithms, Proximal Policy Optimization (PPO) and Double Deep Q-Network (DDQN), are implemented to guide learning, facilitating a comparison of agent performances under each algorithm. Achieving optimal performance often necessitates a significant volume of agent-environment interactions, posing challenges for large-scale systems. To address this, Transfer Learning (TL) is introduced. TL aims to use the agents trained with Adv-MARL and improve its working in each cycle. Thus, avoiding training a new agent from scratch.  Q-learning is a well-known RL algorithm in which the Q-value function is estimated using the Bellman equation iteratively until it converges to the optimal value. In deep Q-learning, Convolutional Neural Network (CNN) are used to estimate the Q-value. The initial layers of the CNN learn the low-level representation of data from the task which can be generalized for the succeeding learning task. Through Transfer Learning, parameters of these initial layers are transferred and fine-tuned, facilitating knowledge transfer between environments and enhancing learning efficiency. Preliminary results show that the players learn faster with PPO and TL shows improved performance while learning and takes less time to adjust to the new game iteration.

Presenting Author: Joachim Grimstad Universität Stuttgart

Presenting Author Biography: Joachim Grimstad is currently a PhD Student at the University of Stuttgart, Institute of Automation and Software Engineering. He has a B.Eng. – Subsea Technology, Maintenance and Operations from the Western Norway University of Applied Sciences and a M.Sc.Eng – RAMS (Reliability, Availability, Maintainability, Safety) from the Norwegian University of Science and Technology.

Authors:

Mayank Jha Universität Stuttgart
Amal Chulliyat Jose Universität Stuttgart
Joachim Grimstad Universität Stuttgart
Andrey Morozov Universität Stuttgart