[STAGE MASTER] - Generative Zero-Shot Learning for Real-Time Object Tracking using 3D Point Cloud Sequences (LiDAR Datasets) in Collaborative Robot Environment

Scientific fields: Computer science, Artificial Intelligence, Computer Vision Keywords: Generative Deep Learning; Zero-Shot Learning; Transformers; Mixture-of-Experts; 3D Point Cloud Sequences; Multi-LiDAR Datasets; Real-Time Object Tracking; Human-Robot Collaboration (HRC); Human-System Interaction (HSI); Real Industrial Environment Research interest: Computer Vision; Machine Learning; Deep Learning Research work: Zero-Shot Deep Learning Models for Real-Time Object Tracking using 3D Point Cloud Sequences (LiDAR Datasets) in Collaborative Robot Environment In the emerging paradigm of Industry 5.0, more intelligent and adaptable collaborative robots (cobots) are replacing traditional robots [1]. Ensuring safe and efficient interaction between cobots and humans depends heavily on the cobots' ability to achieve a comprehensive semantic understanding of dynamic actions and to perform precise object detection and tracking in industrial environments. These capabilities are also critical for various robotics applications [1], including autonomous driving [2], [3], [4]. Despite the growing importance of semantic understanding in such applications, research on object detection [5], [6], real-time 3D object tracking [7], [8] in collaborative robot workspaces, real-time 3D object tracking models remains insufficient. However, 3D object tracking [7], [9] is gaining significant attention from both academia and industry as a fundamental task in AI-driven domains such as robotics [7], [8], autonomous vehicles [2], [10], and Extended Reality [11]. The integration of zero-shot learning (ZSL) [5], [12], [13] with real-time object tracking in collaborative robotics represents a significant leap forward in artificial intelligence and 3D computer vision. In this research, the generative approach leverages the power of zero-shot learning (ZSL) model for real-time object tracking leveraging 3D point cloud sequences (multi-LiDAR datasets [14]) in collaborative robotic environments. Zero-shot learning (ZSL) models can process and interpret complex 3D point cloud sequences and enable robots to detect and track previously unseen objects [5], [12], [13] without the need for task-specific retraining. By integrating multiple LiDAR datasets and advanced generative deep learning architectures [7], [8], [9], including transformer-based models [15]and mixture-of-experts models [16], [17], we will try to demonstrate a robust performance in tracking previously unseen objects without prior training examples. The proposed approach uniquely will combine spatial-temporal feature extraction from point cloud sequences with a scalable generative transformer [15]with mixture-of-experts architecture [16], [17], enabling efficient processing of high-dimensional LiDAR data streams. The main objective of the proposed internship is to provide a comprehensive literature review on three technologies: Zero-shot learning (ZSL), Real-time object detection/tracking and LiDAR sequences datasets, which have strong potential to promote and optimize operational efficiency and intelligent systems integration within Industry 5.0, facilitating advancements in automation, quality control, and safety measures in industrial environments. In this internship, student will propose Zero-shot learning (ZSL) models. Then, student will train these models on different 3D Point cloud datasets (COVERED-CollabOratiVE Robot Environment Dataset [14]and PandaSet [18]LiDAR dataset) for smart real-time object detection/tracking for Human-Robot Collaboration (HRC) in real industrial environment. Work plan: The working plan in general is divided in two phases: 1) In the first phase (about two-months), student will provide the state of-the-art (SOTA) of the Zero-shot learning models (machine/deep learning) applied for Real-Time Object Detection and Object Tracking using 3D Point Cloud LiDAR datasets. Then student will test SOTA models on COVERED 3D Point Cloud datasets [14]. 2) In the second phase (about four months), student would propose contributions to the following research directions: 1. Proposing new zero-shot learning model based on 3D Point Cloud Transformers and Mixture-of-Experts for Smart Real-Time Object Detection and Object Tracking 2. Studying the properties of such models (complexity, expressivity, frugality) 3. Application of the proposed zero-shot learning model on 3D Point Cloud as COVERED datasets [14] and Pandaset Lidar Data [18]for Smart Real-Time Object Detection and Object Tracking tasks to measure the accuracy and study the performance. Expected scientific production Different scientific productions, write an international peer-reviewed conference paper or an indexed journal paper are expected: 1. Journal publication relating to the literature review about Generative Zero-Shot learning for 3D Real Time Object Tracking using 3D Point Cloud datasets 2. Publication relating to our proposal of a new Generative Zero-Shot learning model for 3D Real Time Object Tracking, model performance and evaluation based on training/validation and testing on Human-Robot Collaboration datasets for 3D Point Cloud Datasets in real industrial environment. Introduction to the laboratory CESI LINEACT- Research Unit CESI LINEACT (Digital Innovation Laboratory for Companies and Learnings at the service of the territories competitiveness) is the CESI group laboratory whose activities are implemented on CESI campuses. Link to the laboratory website: https://lineact.cesi.fr/en/https://lineact.cesi.fr/en/research-unit/presentation-lineact/ CESI LINEACT (EA 7527), Digital Innovation Laboratory for Business and Learning at the service of the Competitiveness of Territories, anticipates and accompanies the technological mutations of the sectors and services related to industry and construction. CESI's historical proximity to companies is a determining factor for our research activities and has led us to focus our efforts on applied research close to companies and in partnership with them. A human-centered approach coupled with the use of technologies, as well as the territorial network and the links with training, have allowed us to build transversal research; it puts the human being, his needs and his uses, at the center of its problems and approaches the technological angle through these contributions. Its research is organized according to two interdisciplinary scientific themes and two application areas. Theme 1 "Learning and Innovation" is mainly concerned with Cognitive Sciences, Social Sciences and Management Sciences, Training Sciences and Techniques and Innovation Sciences. The main…

Location: Nice, FR

Posted Date: 2/3/2025