Metrologically based methods and tools to objectively measure, compare, and ensure the performance of AI systems

Purpose and goal

The aim of the planning project is to develop and strengthen an international project application to EURAMET´s call for proposals in fundamental metrology, with a focus on metrologically based methods for AI evaluation. The unique funding of the project is to transfer methods from Item Response Theory (IRT) to evaluation of AI models. This will utilize IRT as a method to measure the latent capabilities of AI models as well as the tasks that AI is faced with.

Expected effects and result

The main project´s objective is to develop metrologically based methods for comparison and to try to achieve an objective and traceable measurement of AI performance, incl quantified uncertainty and decision risk. This is in line with the call´s aim to strengthen Europe´s metrological capacity, support regulation and standardization, and enable reliable innovation. The project contributes to the expected effects by creating common methods that can be used in conformity assessment and standards.

Planned approach and implementation

The work is divided into two work packages: 1. Literature and method review of existing IRT applications in machine learning and AI benchmarking, identification of relevant IRT models and a conceptual link to metrological principles such as traceability and measurement uncertainty. Investigate how Item Response Theory (IRT) can be used as a metrologically based framework to evaluate and compare the performance of AI models. 2. Coordination and writing. Partnering meetings with European partners.