Browsing by Author "Liu, Xuze"

Now showing 1 - 8 of 8
  • Thumbnail Image
    ItemOpen Access
    Application of advanced tree search and proximal policy optimization on formula-E race strategy development
    (Elsevier, 2022-02-25) Liu, Xuze; Fotouhi, Abbas; Auger, Daniel J.
    Energy and thermal management is a crucial element in Formula-E race strategy development. Most published literature focuses on the optimal management strategy for a single lap and results in sub-optimal solutions to the larger multi-lap problem. In this study, two Monte Carlo Tree Search (MCTS) enhancement techniques are proposed for multi-lap Formula-E racing strategy development. It is shown that using the bivariate Gaussian distribution enhancement, race finishing time improves by at least 0.25% and its variance reduces by more than 26%. Compared to the published conventional MCTS technique used in multi-lap problems, this proposed technique is proved to bring a remarkable enhancement with no additional computational time cost. By further enhancing the MCTS using proximal policy optimization, the final product is capable of generating more than 0.5% quicker race time solutions and improving the consistency by over 90% which makes it a very suitable method particularly when enough training time is guaranteed
  • Thumbnail Image
    ItemOpen Access
    Energy-optimal overtaking manoeuvres of Formula-E cars
    (Taylor and Francis, 2022-07-11) Liu, Xuze; Fotouhi, Abbas; Auger, Daniel J.
    Overtaking in motorsport races is a crucial action in order to gain a higher position during a race. In Formula-E, it not only requires precisely performed overtaking manoeuvres but also requires appropriate energy management which is also a crucial element in race strategy due to the limited total amount of energy on board. This study proposes the use of optimal control techniques for feasibility and energy management analysis for such a complex problem. The analysis involves two cars on a track and the goal is to find the optimal overtaking strategy under various scenarios. The results demonstrate that overtaking can be infeasible despite that there is a potential speed advantage over the target car. An overtaking manoeuvre can be executed efficiently by using higher propulsion power. When an overtaking is feasible, the energy-optimal overtaking position varies according to different initial conditions.
  • Thumbnail Image
    ItemOpen Access
    Formula-E multi-car race strategy development—a novel approach using reinforcement learning
    (IEEE, 2024-08) Liu, Xuze; Fotouhi, Abbas; Auger, Daniel
    Electric motorsport such as Formula E is becoming more and more popular in recent years. Race strategy in such races can be very complex involving resource management, e.g. energy and thermal management, but more importantly multi-car interactions which could be both collaborative and competitive. Reinforcement Learning has been implemented in the literature for such electric racing strategy development but only accounts for one single car. In this paper, we proposed a new architecture iRaXL to implement reinforcement learning for such complex strategy development featuring hybrid action space, multi-car interactions, and non-zero-sum gaming. The iRaXL proves to be able to develop different strategies for individual competitors and also team-based objectives. In a bigger scope, this framework can be used to solve more generic problems with hybrid features such as zero/non-zero-sum games, discretized/continuous action space, and competition/collaboration interactions.
  • Thumbnail Image
    ItemOpen Access
    Formula-E race strategy development using artificial neural networks and Monte Carlo Tree Search
    (Springer Verlag, 2020-03-30) Liu, Xuze; Fotouhi, Abbas
    Energy management has been one of the most important parts in electric race strategies since the Fédération Internationale de l'Automobile (FIA) Formula-E championships was launched in 2014. Since that time, a number of unfavorable race finishes have been witnessed due to poor energy management. Previous researches have been focused on managing the power flow between different energy sources or different energy consumers based on a fixed cycle. However, there is no published work in the literature about energy management of a full electric racing car on repeated course but with changeable settings and driving styles. Different from traditional energy management problems, the electric race strategy is more of a multi-stage decision making problem which has a very large scale. Meanwhile, this is a time-critical task in motorsport where fast prediction tools are needed and decisions have to be made in seconds to benefit the final outcome of the race. In this study, the use of Artificial Neural Networks (ANN) and tree search techniques are investigated as an approach to solve such a large-scale problem. ANN prediction models are developed to replace the traditional lap time simulation as a much faster performance prediction tool. Implementation of Monte Carlo Tree Search (MCTS) based on the proposed ANN fast prediction models has provided decent capability to generate decision-making solution for both pre-race planning and in-race reaction to unexpected scenarios.
  • Thumbnail Image
    ItemOpen Access
    Formula-E race strategy development using distributed policy gradient reinforcement learning
    (Elsevier, 2021-01-20) Liu, Xuze; Fotouhi, Abbas; Auger, Daniel J.
    Energy and thermal management is a crucial element in Formula-E race strategy development. In this study, the race-level strategy development is formulated into a Markov decision process (MDP) problem featuring a hybrid-type action space. Deep Deterministic Policy Gradient (DDPG) reinforcement learning is implemented under distributed architecture Ape-X and integrated with the prioritized experience replay and reward shaping techniques to optimize a hybrid-type set of actions of both continuous and discrete components. Soft boundary violation penalties in reward shaping, significantly improves the performance of DDPG and makes it capable of generating faster race finishing solutions. The new proposed method has shown superior performance in comparison to the Monte Carlo Tree Search (MCTS) with policy gradient reinforcement learning, which solves this problem in a fully discrete action space as presented in the literature. The advantages are faster race finishing time and better handling of ambient temperature rise.
  • Thumbnail Image
    ItemOpen Access
    Nash double Q-based multi-agent deep reinforcement learning for interactive merging strategy in mixed traffic
    (Elsevier, 2023-09-19) Li, Lin; Zhao, Wanzhong; Wang, Chunyan; Fotouhi, Abbas; Liu, Xuze
    The interaction between ramp and mainline vehicles plays a crucial role in merging areas, especially in the mixed-traffic environment. The driving behaviours of human drivers are uncertain and diverse, and the uncertainty makes it more complex for connected automated vehicles (CAV) to plan trajectories and merge into the mainline. To overcome this problem, a interactive merging strategy based on multi-agent deep reinforcement learning (MADRL) is designed, enabling the ramp vehicle (CAV) to consider the dynamic reaction of mainline vehicles. There are two agents in our interactive strategy, one of which is to predict and analyse the behaviour of mainline vehicles (human-driven vehicles, HDV, or non-connected vehicles). The other is created for exploring optimal merging actions of ramp vehicles. Firstly, game theory is used to model the competitive behaviours between ramp and mainline vehicles, and the Nash equilibrium of joint actions guides the ramp vehicle to learn best response to the mainline vehicle. Secondly, the Nash double Q algorithm is developed to ensure the outputs of Q networks are trained to efficiently converge to the Nash equilibrium point. The trained Q networks are then used for online control. Finally, our strategy is compared with single RL and existing MADRL algorithms in real on-ramp scenarios. Simulations show our strategy to be successful in coordinating both vehicles via analysis of human drivers, resulting in improved driving performance in terms of global safety, efficiency, and comfort.
  • Thumbnail Image
    ItemOpen Access
    Optimal control of race car with aerodynamic slipstreaming effect
    (IEEE, 2024-05-13) Liu, Xuze; Fotouhi, Abbas; Cecotti, Marco; Auger, Daniel
    This article presents a new method to describe the aerodynamics slipstreaming effect on the downstream car. This new approach can be implemented in lap time simulations (LTSs) and used to study the optimal trajectory of a downstream car operating in the wake of an upstream car. Two different scenarios are investigated using this method. In the energy-saving scenario for electric racing cars, the result shows the optimal strategy varies depending on the upstream car’s pace and the initial gap between the two cars. Chasing to stay in the wake is less effective when the initial gap is relatively big. In the overtaking scenario on an oval track, it is shown that the wake of the upstream car benefits the downstream car’s acceleration but, meanwhile, reduces the lateral performance limit of the downstream car due to downforce loss. In order to maintain a competitive performance, it is essential for the downstream car to choose an alternative racing line to drive outside the wake when braking and passing through a corner.
  • Thumbnail Image
    ItemOpen Access
    Optimal energy management for formula-E cars with regulatory limits and thermal constraints
    (Elsevier, 2020-09-13) Liu, Xuze; Fotouhi, Abbas; Auger, Daniel J.
    In this paper, novel solutions are proposed for energy and thermal management in Formula-E cars using optimal control theory. Optimal control techniques are used to optimize net energy consumption (accounting for loss-reductions from energy recovery from regenerative braking) to achieve minimal lap time which is a crucial element in developing a competitive race strategy in Formula E races. A thermal battery model is used to impose thermal constraints on the optimal energy management strategy in order to realistically capture working constraints during a race. The effects of energy and thermal constraints on the proposed strategy are then demonstrated and two different pedal lifting techniques were introduced. Both the current second generation and a concept third generation type of formula-E cars are studied and compared. While third generation is significantly more efficient with 10% to 30% less energy consumption, it potentially faces more critical thermal issues with more than 60% more heat generation.