MOPS - Multi-modal Open-set Perception for Safer Autonomous Trucks

Purpose and goal

Over the life-time of an autonomous vehicle, it will probably encounter long-tail situations in its Operational Design Domain (ODD) that have not been tested in V&V, but it must still be able to handle them. Foundation models show generalization and reasoning capabilities, which could solve the long-tail problem. The purpose of this project is to study and propose sensor data processing methods that are rich and compressed enough to enable the use of foundation models in autonomous vehicles.

Expected effects and result

Foundation Models demonstrate generalization and reasoning properties needed for an autonomous vehicle to handle unforeseen situations in a safe and reasonable way. The research conducted in this project aims to enable foundation models in autonomous vehicles which will provide safer vehicles that can handle larger Operational Domain Designs (ODDs), e.g., autonomous highway driving in different seasons and weather conditions.

Planned approach and implementation

The project is divided into four work packages. Two work packages concern methods for multimodal sensor data processing, one work package concerns evaluation of these methods in different foundation models in applications for autonomous vehicles and the fourth is project coordination. The methods will first study spatial reasoning and then take in the time aspect and handling of multimodal sensor data over both time and space. The evaluation will be done in post-perception, e.g. planning.