SwedishGlue
| Reference number | |
| Coordinator | RISE Research Institutes of Sweden AB - RISE SICS |
| Funding from Vinnova | SEK 1 999 916 |
| Project duration | September 2020 - July 2021 |
| Status | Completed |
Purpose and goal
The goal of SuperLim has been to produce test data to evaluate the quality of and identify inequalities in Swedish language models, as well as enable comparisons between them. The project has exceeded expectations and have delivered 13 test sets to investigate different aspects of language comprehension. These form the basis of, and an important step towards, a Swedish framework for the development and comparison of language models.
Expected results and effects
Multiple strategies have been used to develop the resources included in the collection. Some existing resources have been reformatted and reused. Some resources included in (Super) GLUE have been machine translated and then manually corrected. Some resources are brand new and were created specifically for SuperLim. We received some resources as voluntary funding from external parties, as the project has become widespread nationally and mobilized a great deal of interest in the private and public sectors.
Planned approach and implementation
It is highly desirable to further develop the result, first and foremost, to provide training amounts for the test amounts that lack it, to establish a baseline (a reference value with which the models´ performance can be compared) and to create a leaderboard to make the models´ relative performance visible.a