Session: 07-17-01: Machine Learning and Artificial Intelligence in Dynamics, Vibrations and Control

Paper Number: 144480

144480 - Deep Q-Learning Based Optimal Energy Management of a Plug-in Hybrid Electric Vehicle 

With rapid development of new and efficient technologies in next generation vehicles, energy management of Plug in Hybrid Electric Vehicles (PHEV) plays a critical role both from environment and economic considerations. PHEV blends the benefits of conventional combustion engine and electric motor to improve energy consumption. The Energy Management Strategy (EMS) of PHEV needs to manage the motor power and engine power to reduce energy consumption and to balance the state of charge (SoC) of the battery.

EMS strategies for HEV are typically divided into two categories: 1) Rule based, and 2) Optimization based. Rule-based EMS uses expert knowledge to improve the energy efficiency of the whole vehicle focusing mainly on aspects like engine starting, idle operation, regen braking, and adjusting the engine operating point to improve engine efficiency. Rule-based EMS’s are easy to implement and simple to understand, therefore they have a wide range of applications for practical purposes. However, there application is limited to specific driving cycles which making it difficult to achieve the optimal fuel economy.

Optimization based on the other hand utilizes system model and an objective function to compute the optimal solution with given constraints. In general, these optimization-based strategies can determine the optimal power split between engine and electric system. However, they require system model and high computation power.

Here we propose a value-based Reinforcement learning approach called Deep Q learning based energy management strategy to determine the optimal action to maximize the fuel economy without the need to have a vehicle model. The goal of reinforcement learning is to find a control strategy that achieve maximum cumulative rewards by mapping the states to actions to achieve optimal energy consumption by means of exploration and feedback mechanisms, enabling the application of experience to unseen operating conditions.

A deep Q (DQN) reinforcement learning algorithm is utilized to train using a PHEV vehicle model on a UDSS cycle and HWFET driving cycle. DQN is neural network takes a given state 𝑠 as inputs and outputs action,

𝑎 power split between engine and motor. The agent interacts with the PHEV environment during training and stores the transactions 𝑠_𝑡, 𝑎_𝑡,𝑟_𝑡, 𝑠_𝑡+1, then random mini-batches are sampled to calculate the state- action value 𝑄(𝑠, 𝑎; 𝜃𝑖) and target network is used to generate target Q value. These two are then used to compute loss function and update neural network.

Fuel economy and convergence characteristics of DQN are discussed and compared with rule-based EMS and the performance is analyzed on a HWFET cycle.

Presenting Author: Sohel Anwar Purdue University in Indianapolis

Presenting Author Biography: Dr. Sohel Anwar is a Full Professor in the Department of Mechanical and Energy Engineering (MEE) at Purdue School of Engineering and Tech, IUPUI, Indianapolis, IN, USA. He is also the director of Mechatronics and Autonomous Research Lab (MARL). He has over 28 years of combined academic and industry R & D experience in the general area of mechatronics, automation, and controls. His research program is focused on Hybrid and Electric vehicle powertrain control and optimization, Li-Ion battery diagnostics / prognostics / fast charging control, Autonomous vehicle simulation and control, novel sensor development and fusion, and smart novel medical devices. Professor Anwar has published more than 160 peer-reviewed journal, conference papers, book chapters, and eBook. He is also an inventor or co-inventor on 15 US patents (with 1 currently pending). He has supervised the research work of more than 60 current/former graduate students (12 PhD, 49 MS). Dr. Anwar's research projects has been supported by the grants from National Science Foundation (NSF), National Institute of Health (NIH), US Army, Department of Energy (DoE), State of Indiana, Cummins Inc., and many other Industry partners. Dr. Anwar earned his PhD in Mechanical Engineering with specialization in Controls from the University of Arizona in 1995.

Authors:

Vikas Narang Purdue University in Indianapolis
Kartavya Neema Applied Intuition
Sohel Anwar Purdue University in Indianapolis