Risk assessment in vision-language-action models for autonomous driving

Purpose and goal

This project will test different strategies for uncertainty extraction from Vision-Language-Action models when used in Autonomous Driving (AD). AD systems are normally end-to-end, with makes uncertainty extraction challenging. In AD, uncertainty means e.g. when the system encounters unusual objects or behaviours, bad weather or blockages. We will also identify the possible use cases of uncertainty in automated driving through the use of structured interviews with experts in the field.

Expected effects and result

Expected results are: an implementation of a risk-monitor system for uncertainty extraction in VLAs. A scientific paper that describes the system, and experiments that verify its behaviour. A report that summarizes our structured interviews about use cases for uncertainty in AD.

Planned approach and implementation

We will test different strategies for extraction of uncertainty from VLAs. We will start by adapting our previously developed RiskMonitor system for end-to-end AD to the VLA setting, by integrating it in an existing system, e.g. SimLingo. We will also identify the possible use cases of uncertainty in automated driving through the use of structured interviews with experts in the field.