Object detection for ground-based non-cooperative surveillance in urban air mobility utilizing lidar-camera fusion

Public safety and security are critical components in the Concept of Operations (ConOps) for Urban Air Mobility (UAM). The potential flight conflicts posed to air and ground objects need to be assessed, especially near critical regions and infrastructures, e.g. vertiports. In this sense, all targets, whether cooperative or non-cooperative air and ground targets, should be detected and tracked for conflict and risk assessment. To achieve this goal, ground-based non-cooperative sensors like cameras and lidar are utilized for situational awareness in this paper. In addition, a multi-modal dataset that contains both air and ground objects is constructed in different illumination and foggy weather scenarios. Finally, a lidar-camera fusion framework with multi-resolution voxelization and depth map learning is proposed for data-driven object detection. Experiments on the constructed dataset show the failure of existing lidar-based backbones in learning extremely sparse points, as a comparison, the fusion framework is outstanding in distinguishing air and ground objects, meanwhile, enabling resilient detection in various lighting and clearance conditions.