Real-time ship recognition and georeferencing for the improvement of maritime situational awareness

Abstract

In an era where maritime infrastructures are paramount to societies, the need for advanced maritime situational awareness solutions has become increasingly important. Existing ship monitoring procedures, such as the Automatic Identification System (AIS), have limitations, suffer from delayed updates, and are vulnerable to cyberattacks. Other technologies, such as satellite imagery and radar, face challenges in real-time applications due to delays in acquiring and processing data. The use of optical camera systems and image processing can improve situational awareness, allowing real-time usage of maritime infrastructure footage. However, the number of video streams available poses a challenge for maritime operators, who could benefit from summarized spatial information of recognized ships, irrespective of their size and type, presented on a map in real-time. This motivates the development of automated ship recognition and georeferencing technologies. Moreover, the deployment of such camera systems, equipped with embedded devices, allows for local data processing on the edge to minimize network demand, reduce energy usage, decrease latency, cut costs, and enhance data protection. This thesis, integrating six of my publications, presents a comprehensive investigation into leveraging deep learning and computer vision to advance real-time ship recognition and georeferencing for the improvement of maritime situational awareness. I present a novel dataset for ship recognition and georeferencing, ShipSG, which facilitates the development and validation of recognition and georeferencing methodologies. The dataset contains 3,505 images and 11,625 ship masks with their corresponding class, geographic position, and length. Through a series of studies of state-of-the-art deep-learning-based object recognition algorithms, I introduce a custom real-time segmentation architecture, ScatYOLOv8+CBAM. This architecture was created and optimized for the NVIDIA Jetson AGX Xavier as an embedded system. ScatYOLOv8+CBAM incorporates the 2D scattering transform, a novel addition that enhances YOLOv8 in real-world applications such as ship segmentation. Additionally, the performance is further improved with the integration of attention mechanisms. The proposed architecture exceeds state-of-the-art methods by more than 5%, achieving a mean Average Precision (mAP) of 75.46%. The inference speed, once the customized architecture is deployed on the embedded system using TensorRT, is 25.3 ms per frame. Furthermore, I address the need for precision in recognizing small and distant ships and their real-time processing of full-resolution images on embedded systems with an enhanced slicing mechanism that performs batch inference and merges predictions, achieving mAP improvements ranging from 8% to 11%. The recognized ships are georeferenced using my proposed method, which automatically calculates the georeferencing pixel of the recognized ships and uses homographies to provide the geographic position of ships from single images without prior camera knowledge. In the quantitative analysis, the georeferencing method achieved a positioning error of 18 m ± 10 m for ranges inside the port basin (up to 400 m) and 44 m ± 27 m outside (from 400 m to 1200 m). The main findings reveal significant advancements in maritime situational awareness with the practical demonstration of the applicability of the methodologies in real-world scenarios, such as the detection of abnormal ship behavior, camera integrity assessment, and 3D reconstruction. The approach not only outperforms existing methods in terms of accuracy and processing speed but also provides a framework for seamlessly integrating recognized and georeferenced ships into real-time systems, enhancing operational effectiveness and decision-making for maritime authorities. The integration of these methodologies into embedded systems represents a pivotal advancement in the domain, offering a scalable and efficient solution for improving maritime situational awareness and response capabilities. This thesis contributes to the maritime computer vision field by establishing a benchmark for ship segmentation and georeferencing research, demonstrating the viability of deep-learning-based recognition and georeferencing methods for real-time maritime monitoring.