Development of vision guided real-time trajectory planning system for autonomous ground refuelling operations using hybrid dataset

Accurate and rapid object localisation and pose estimation are playing key roles during some of the real-time robotic operations such as object grasping and object manipulating. To do so, high-level robotic vision solutions need to be adopted. Computer vision approaches require a large amount of data to be able to create a perception pipeline robustly. Preparing such dataset to train the deep neural network could be challenging as the collection and manual annotation of huge amounts of data can take long hours and the development of the dataset needs to cover different conditions in weather and lighting. To ease this process, generating a synthetic dataset could be used. Due to the limitations of the synthetic dataset which will be described further down, instead of using a sole synthetic dataset, a hybrid dataset can be developed with the real dataset to overcome the limitations of both datasets. Even though the main objective of this study is to fulfil an autonomous nozzle insertion process for the ground refuelling operation of civil aircraft, the proposed approach is generic and can be adapted to any 3D visual robotic manipulation operation. This study is also offered to be the first visual trajectory planning control mechanism depending on the hybrid dataset to this date.