Browsing by Author "Hu, Minghua"
Now showing 1 - 6 of 6
Results Per Page
Sort Options
Item Open Access A 4D-trajectory planning method based on hybrid optimization strategy for demand and capacity balancing(IEEE, 2021-11-15) Chen, Yutong; Xu, Yan; Hu, Minghua; Huang, Fei; Nie, QiTo effectively solve the Demand and Capacity Balancing (DCB) in future Trajectory-Based Operation (TBO) scenarios, this article first proposes a pre-tactical-and-tactical integrated Four-Dimensional Trajectory (4DT) planning framework. The framework decomposes large-scale 4DT planning into two stages, namely, the General 4DT (G4DT) planning in the pre-tactical stage and the Special 4DT (S4DT) planning in the tactical stage. A Hybrid Optimization Strategy (HOS) based planning method is designed for G4DT planning. In this method, the sequential decision architecture based on time window, heuristic strategy (greedy strategy) and optimization algorithm are combined to realize the fast trajectory planning of large-scale flights. In the optimization model based on continuous time, the nonlinear model is transformed into a linear model by constructing the flight conflict correlation matrix, which greatly improves the solving speed of the model. Real flight schedule data for French and Spanish airspace were used to verify the effectiveness and efficiency of the HOS method. This method is compared with Computer-Assisted Slot Allocation (CASA). The results show that the proposed method can effectively reduce the flight delay time and improve the flight on-time rate. Due to its fast operation speed, the proposed method has great potential to dynamically update the planning results according to the real-time air space operation status in actual operation.Item Open Access Demand and capacity balancing technology based on multi-agent reinforcement learning(IEEE, 2021-11-15) Chen, Yutong; Xu, Yan; Hu, Minghua; Yang, LeiTo effectively solve Demand and Capacity Balancing (DCB) in large-scale and high-density scenarios through the Ground Delay Program (GDP) in the pre-tactical stage, a sequential decision-making framework based on a time window is proposed. On this basis, the problem is transformed into Markov Decision Process (MDP) based on local observation, and then Multi-Agent Reinforcement Learning (MARL) method is adopted. Each flight is regarded as an independent agent to decide whether to implement GDP according to its local state observation. By designing the reward function in multiple combinations, a Mixed Competition and Cooperation (MCC) mode considering fairness is formed among agents. To improve the efficiency of MARL, we use the double Q-Learning Network (DQN), experience replay technology, adaptive ϵ-greedy strategy and Decentralized Training with Decentralized Execution (DTDE) framework. The experimental results show that the training process of the MARL method is convergent, efficient and stable. Compared with the Computer-Assisted Slot Allocation (CASA) method used in the actual operation, the number of flight delays and the average delay time is reduced by 33.7% and 36.7% respectively.Item Open Access General multi-agent reinforcement learning integrating adaptive manoeuvre strategy for real-time multi-aircraft conflict resolution(Elsevier, 2023-04-12) Chen, Yutong; Hu, Minghua; Yang, Lei; Xu, Yan; Xie, HuaReinforcement learning (RL) techniques are under investigation for resolving conflict in air traffic management (ATM), exploiting their computational capabilities and ability to cope with flight uncertainty. However, the limitations of generalisation make it difficult for existing RL-based conflict resolution (CR) methods to be effective in practice. This paper proposes a general multi-agent reinforcement learning (MARL) method that integrates an adaptive manoeuvre strategy to enhance both the solution’s efficiency and the model’s generalisation in multi-aircraft conflict resolution (MACR). A partial observation approach based on the imminent threat detection sectors is used to gather critical environmental information, enabling the model to be applied in arbitrary scenarios. Agents are trained to provide the correct flight intention (such as increasing speed and yawing to the left), while an adaptive manoeuvre strategy generates the specific manoeuvre (speed and heading parameters) based on the flight intention. To address flight uncertainty and performance challenges caused by the intrinsic non-stationarity in MARL, a warning area for each aircraft is introduced. We employ a state-of-the-art Deep Q-learning Network (DQN) method, Rainbow DQN, to improve the efficiency of the RL algorithm. The multi-agent system is trained and deployed in a distributed manner to adapt to real-world scenarios. A sensitivity analysis of uncertainty levels and warning area sizes is conducted to explore their impact on the proposed method. Simulation experiments confirm the effectiveness of the training and generalisation of the proposed method.Item Open Access General multi-agent reinforcement learning integrating heuristic-based delay priority strategy for demand and capacity balancing(Elsevier, 2023-06-22) Chen, Yutong; Xu, Yan; Hu, MinghuaReinforcement learning (RL) techniques have been studied for solving the demand and capacity balancing (DCB) problem in air traffic management to exploit their full computational potential. Due to the lack of generalisation and the seemingly reduced optimisation performance affected by the training scenarios, it is challenging for existing RL-based DCB methods to be effectively applied in practice. This paper proposes a general multi-agent reinforcement learning (MARL) method that integrates a heuristic-based delay priority strategy to improve the efficiency of the solution and the generalisation of the model. The delay priority strategy is used to reduce the potential learning task and thus training difficulty. This study explores what features of the delay priority strategy are better suited to the MARL method. A long short-term memory (LSTM) network is integrated into a deep q-learning network (DQN) to ensure the model compatible with arbitrary DCB instances and to facilitate agents to identify key sectors. This study is conducted as a part of a large-scale European DCB research project, where real data from French and Spanish airspace are used for experimentation. Results suggest that the proposed method has advantages in generalisation, optimisation performance and computational performance over state-of-the-art RL-based DCB methods.Item Open Access General real-time three-dimensional multi-aircraft conflict resolution method using multi-agent reinforcement learning(Elsevier, 2023-10-10) Chen, Yutong; Xu, Yan; Yang, Lei; Hu, MinghuaReinforcement learning (RL) techniques have been studied for solving the conflict resolution (CR) problem in air traffic management, leveraging their potential for computation and ability to handle uncertainty. However, challenges remain that impede the application of RL methods to CR in practice, including three-dimensional manoeuvres, generalisation, trajectory recovery, and success rate. This paper proposes a general multi-agent reinforcement learning approach for real-time three-dimensional multi-aircraft conflict resolution, in which agents share a neural network and are deployed on each aircraft to form a distributed decision-making system. To address the challenges, several technologies are introduced, including a partial observation model based on imminent threats for generalisation, a safety separation relaxation model for multiple flight levels for three-dimensional manoeuvres, an adaptive manoeuvre strategy for trajectory recovery, and a conflict buffer model for success rate. The Rainbow Deep Q-learning Network (DQN) is used to enhance the efficiency of the RL process. A simulation environment that considers flight uncertainty (resulting from mechanical and navigation errors and wind) is constructed to train and evaluate the proposed approach. The experimental results demonstrate that the proposed method can resolve conflicts in scenarios with much higher traffic density than in today’s real-world situations.Item Open Access Locally generalised multi-agent reinforcement learning for demand and capacity balancing with customised neural networks(Elsevier, 2023-04-18) Chen, Yutong; Hu, Minghua; Xu, Yan; Yang, LeiReinforcement Learning (RL) techniques are being studied to solve the Demand and Capacity Balancing (DCB) problems to fully exploit their computational performance. A locally generalised Multi-Agent Reinforcement Learning (MARL) for real-world DCB problems is proposed. The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management (ATFM) region to quickly obtain a satisfactory solution. In this method, agents of all flights in a scenario form a multi-agent decision-making system based on partial observation. The trained agent with the customised neural network can be deployed directly on the corresponding flight, allowing it to solve the DCB problem jointly. A cooperation coefficient is introduced in the reward function, which is used to adjust the agent’s cooperation preference in a multi-agent system, thereby controlling the distribution of flight delay time allocation. A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated. Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method. From a statistical point of view, it is proven that the proposed method is generalised within the scope of the flights and sectors of interest, and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods. The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.